Note: This is test shard 5 of 8.
[==========] Running 3 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 2 tests from DeleteTableITest
[ RUN ] DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy
2026-09-28T01:57:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-28T01:57:18Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260928 01:57:18.467976 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.83.62:34493
--webserver_interface=127.9.83.62
--webserver_port=0
--builtin_ntp_servers=127.9.83.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.83.62:34493
--allow_unsafe_replication_factor=true
--catalog_manager_evict_excess_replicas=false
--master_add_server_when_underreplicated=false with env {}
W20260928 01:57:18.567688 9556 flags.cc:432] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20260928 01:57:18.567929 9556 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:18.567956 9556 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:18.567991 9556 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:18.571425 9556 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 01:57:18.571498 9556 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:18.571524 9556 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 01:57:18.571542 9556 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260928 01:57:18.575661 9556 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
--allow_unsafe_replication_factor=true
--catalog_manager_evict_excess_replicas=false
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.9.83.62:34493
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.83.62:34493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.83.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.9556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:18.576742 9556 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:18.577818 9556 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:18.583125 9562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:18.583098 9561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:18.583216 9556 server_base.cc:1061] running on GCE node
W20260928 01:57:18.583093 9564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:18.583792 9556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:18.584704 9556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:18.585868 9556 hybrid_clock.cc:648] HybridClock initialized: now 1790560638585857 us; error 27 us; skew 500 ppm
I20260928 01:57:18.587690 9556 webserver.cc:533] Webserver started at http://127.9.83.62:36453/ using document root <none> and password file <none>
I20260928 01:57:18.588222 9556 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:18.588272 9556 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:18.588434 9556 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 01:57:18.590291 9556 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/instance:
uuid: "a9dce204f3a642aa84955400c67e32fd"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:18.590736 9556 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal/instance:
uuid: "a9dce204f3a642aa84955400c67e32fd"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:18.594081 9556 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 01:57:18.596210 9570 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:18.597143 9556 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 01:57:18.597234 9556 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
uuid: "a9dce204f3a642aa84955400c67e32fd"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:18.597306 9556 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:18.607379 9556 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:18.607916 9556 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:18.608052 9556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:18.614511 9556 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.62:34493
I20260928 01:57:18.614526 9622 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.62:34493 every 8 connection(s)
I20260928 01:57:18.615511 9556 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
I20260928 01:57:18.618244 9623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:18.623102 9623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: Bootstrap starting.
I20260928 01:57:18.623431 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 9556
I20260928 01:57:18.623564 9548 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal/instance
I20260928 01:57:18.625365 9623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: Neither blocks nor log segments found. Creating new log.
I20260928 01:57:18.626196 9623 log.cc:826] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:18.628304 9623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: No bootstrap required, opened a new log
I20260928 01:57:18.631611 9623 raft_consensus.cc:359] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } }
I20260928 01:57:18.631830 9623 raft_consensus.cc:385] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 01:57:18.631896 9623 raft_consensus.cc:740] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dce204f3a642aa84955400c67e32fd, State: Initialized, Role: FOLLOWER
I20260928 01:57:18.632323 9623 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } }
I20260928 01:57:18.632452 9623 raft_consensus.cc:399] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 01:57:18.632510 9623 raft_consensus.cc:493] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 01:57:18.632596 9623 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 01:57:18.633482 9623 raft_consensus.cc:515] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } }
I20260928 01:57:18.633846 9623 leader_election.cc:304] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9dce204f3a642aa84955400c67e32fd; no voters:
I20260928 01:57:18.634187 9623 leader_election.cc:290] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 01:57:18.634286 9628 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 01:57:18.634475 9628 raft_consensus.cc:697] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 1 LEADER]: Becoming Leader. State: Replica: a9dce204f3a642aa84955400c67e32fd, State: Running, Role: LEADER
I20260928 01:57:18.634820 9628 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } }
I20260928 01:57:18.635443 9623 sys_catalog.cc:565] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 01:57:18.636270 9629 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9dce204f3a642aa84955400c67e32fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } } }
I20260928 01:57:18.636382 9629 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: This master's current role is: LEADER
I20260928 01:57:18.636722 9630 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9dce204f3a642aa84955400c67e32fd. Latest consensus state: current_term: 1 leader_uuid: "a9dce204f3a642aa84955400c67e32fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } } }
I20260928 01:57:18.637310 9630 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: This master's current role is: LEADER
I20260928 01:57:18.637614 9636 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260928 01:57:18.640640 9636 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260928 01:57:18.645721 9636 catalog_manager.cc:1396] Generated new cluster ID: 266fe30b665b49798234e98038a7ea12
I20260928 01:57:18.645793 9636 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260928 01:57:18.668990 9636 catalog_manager.cc:1419] Generated new certificate authority record
I20260928 01:57:18.669878 9636 catalog_manager.cc:1553] Loading token signing keys...
I20260928 01:57:18.676440 9636 catalog_manager.cc:6219] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: Generated new TSK 0
I20260928 01:57:18.677162 9636 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260928 01:57:18.687367 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.1:0
--local_ip_for_outbound_sockets=127.9.83.1
--webserver_interface=127.9.83.1
--webserver_port=0
--tserver_master_addrs=127.9.83.62:34493
--builtin_ntp_servers=127.9.83.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=0
--maintenance_manager_polling_interval_ms=100 with env {}
W20260928 01:57:18.787250 9647 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:18.787487 9647 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:18.787539 9647 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:18.791476 9647 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:18.791641 9647 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260928 01:57:18.791724 9647 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.1
I20260928 01:57:18.797271 9647 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.83.1
--webserver_port=0
--flush_threshold_mb=0
--tserver_master_addrs=127.9.83.62:34493
--never_fsync=true
--heap_profile_path=/tmp/kudu.9647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:18.798467 9647 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:18.799746 9647 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:18.805910 9655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:18.805877 9653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:18.805882 9652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:18.806245 9647 server_base.cc:1061] running on GCE node
I20260928 01:57:18.806586 9647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:18.807152 9647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:18.808295 9647 hybrid_clock.cc:648] HybridClock initialized: now 1790560638808280 us; error 36 us; skew 500 ppm
I20260928 01:57:18.809852 9647 webserver.cc:533] Webserver started at http://127.9.83.1:46379/ using document root <none> and password file <none>
I20260928 01:57:18.810387 9647 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:18.810441 9647 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:18.810628 9647 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 01:57:18.812247 9647 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/instance:
uuid: "d3f0259187c841a98efbc8ed0eb8021a"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:18.812680 9647 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal/instance:
uuid: "d3f0259187c841a98efbc8ed0eb8021a"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:18.815809 9647 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 01:57:18.817785 9661 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:18.818797 9647 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 01:57:18.818921 9647 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
uuid: "d3f0259187c841a98efbc8ed0eb8021a"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:18.819003 9647 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:18.845921 9647 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:18.846616 9647 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:18.846823 9647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:18.847396 9647 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:18.848377 9647 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 01:57:18.848424 9647 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:18.848474 9647 ts_tablet_manager.cc:616] Registered 0 tablets
I20260928 01:57:18.848502 9647 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:18.856906 9647 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.1:41183
I20260928 01:57:18.856926 9774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.1:41183 every 8 connection(s)
I20260928 01:57:18.857810 9647 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
I20260928 01:57:18.862709 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 9647
I20260928 01:57:18.862834 9548 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal/instance
I20260928 01:57:18.865123 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.2:0
--local_ip_for_outbound_sockets=127.9.83.2
--webserver_interface=127.9.83.2
--webserver_port=0
--tserver_master_addrs=127.9.83.62:34493
--builtin_ntp_servers=127.9.83.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=0
--maintenance_manager_polling_interval_ms=100 with env {}
I20260928 01:57:18.866297 9775 heartbeater.cc:344] Connected to a master server at 127.9.83.62:34493
I20260928 01:57:18.866529 9775 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:18.867076 9775 heartbeater.cc:507] Master 127.9.83.62:34493 requested a full tablet report, sending...
I20260928 01:57:18.868604 9587 ts_manager.cc:194] Registered new tserver with Master: d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183)
I20260928 01:57:18.870185 9587 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.1:53383
W20260928 01:57:18.965777 9778 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:18.966006 9778 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:18.966051 9778 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:18.969472 9778 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:18.969550 9778 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260928 01:57:18.969597 9778 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.2
I20260928 01:57:18.973526 9778 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.83.2
--webserver_port=0
--flush_threshold_mb=0
--tserver_master_addrs=127.9.83.62:34493
--never_fsync=true
--heap_profile_path=/tmp/kudu.9778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:18.974565 9778 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:18.975713 9778 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:18.982146 9784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:18.982126 9783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:18.982124 9786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:18.982240 9778 server_base.cc:1061] running on GCE node
I20260928 01:57:18.982873 9778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:18.983425 9778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:18.984591 9778 hybrid_clock.cc:648] HybridClock initialized: now 1790560638984575 us; error 32 us; skew 500 ppm
I20260928 01:57:18.986301 9778 webserver.cc:533] Webserver started at http://127.9.83.2:37769/ using document root <none> and password file <none>
I20260928 01:57:18.986899 9778 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:18.986959 9778 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:18.987144 9778 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 01:57:18.988729 9778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/instance:
uuid: "00af439d64e84c9db541f5414ac1d2e4"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:18.989172 9778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/wal/instance:
uuid: "00af439d64e84c9db541f5414ac1d2e4"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:18.992961 9778 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 01:57:18.995143 9792 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:18.996114 9778 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 01:57:18.996217 9778 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/wal
uuid: "00af439d64e84c9db541f5414ac1d2e4"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:18.996301 9778 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:19.011008 9778 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:19.011641 9778 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:19.011801 9778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:19.012327 9778 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:19.013257 9778 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 01:57:19.013304 9778 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:19.013355 9778 ts_tablet_manager.cc:616] Registered 0 tablets
I20260928 01:57:19.013382 9778 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:19.021790 9778 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.2:44831
I20260928 01:57:19.021821 9905 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.2:44831 every 8 connection(s)
I20260928 01:57:19.022754 9778 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb
I20260928 01:57:19.030323 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 9778
I20260928 01:57:19.030417 9548 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/wal/instance
I20260928 01:57:19.031000 9906 heartbeater.cc:344] Connected to a master server at 127.9.83.62:34493
I20260928 01:57:19.031234 9906 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:19.031739 9906 heartbeater.cc:507] Master 127.9.83.62:34493 requested a full tablet report, sending...
I20260928 01:57:19.032510 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.3:0
--local_ip_for_outbound_sockets=127.9.83.3
--webserver_interface=127.9.83.3
--webserver_port=0
--tserver_master_addrs=127.9.83.62:34493
--builtin_ntp_servers=127.9.83.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=0
--maintenance_manager_polling_interval_ms=100 with env {}
I20260928 01:57:19.032724 9587 ts_manager.cc:194] Registered new tserver with Master: 00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831)
I20260928 01:57:19.033267 9587 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.2:53453
W20260928 01:57:19.136013 9909 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:19.136255 9909 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:19.136307 9909 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:19.140120 9909 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:19.140205 9909 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260928 01:57:19.140261 9909 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.3
I20260928 01:57:19.144345 9909 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.83.3
--webserver_port=0
--flush_threshold_mb=0
--tserver_master_addrs=127.9.83.62:34493
--never_fsync=true
--heap_profile_path=/tmp/kudu.9909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:19.145438 9909 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:19.146565 9909 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:19.152915 9915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:19.153038 9909 server_base.cc:1061] running on GCE node
W20260928 01:57:19.152930 9917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:19.152976 9914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:19.153527 9909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:19.154060 9909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:19.155212 9909 hybrid_clock.cc:648] HybridClock initialized: now 1790560639155197 us; error 27 us; skew 500 ppm
I20260928 01:57:19.157022 9909 webserver.cc:533] Webserver started at http://127.9.83.3:44237/ using document root <none> and password file <none>
I20260928 01:57:19.157588 9909 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:19.157644 9909 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:19.157830 9909 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 01:57:19.159482 9909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/instance:
uuid: "71a7ac59363f4dd1bc6b7e38506bc135"
format_stamp: "Formatted at 2026-09-28 01:57:19 on dist-test-slave-lkt6"
I20260928 01:57:19.159943 9909 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/wal/instance:
uuid: "71a7ac59363f4dd1bc6b7e38506bc135"
format_stamp: "Formatted at 2026-09-28 01:57:19 on dist-test-slave-lkt6"
I20260928 01:57:19.163348 9909 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 01:57:19.165608 9923 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:19.166848 9909 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 01:57:19.166963 9909 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/wal
uuid: "71a7ac59363f4dd1bc6b7e38506bc135"
format_stamp: "Formatted at 2026-09-28 01:57:19 on dist-test-slave-lkt6"
I20260928 01:57:19.167052 9909 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:19.186980 9909 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:19.187647 9909 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:19.187816 9909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:19.188364 9909 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:19.189443 9909 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 01:57:19.189491 9909 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:19.189541 9909 ts_tablet_manager.cc:616] Registered 0 tablets
I20260928 01:57:19.189567 9909 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:19.198513 9909 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.3:41171
I20260928 01:57:19.198537 10036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.3:41171 every 8 connection(s)
I20260928 01:57:19.199533 9909 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb
I20260928 01:57:19.207801 10037 heartbeater.cc:344] Connected to a master server at 127.9.83.62:34493
I20260928 01:57:19.208019 10037 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:19.208241 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 9909
I20260928 01:57:19.208335 9548 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/wal/instance
I20260928 01:57:19.208636 10037 heartbeater.cc:507] Master 127.9.83.62:34493 requested a full tablet report, sending...
I20260928 01:57:19.209640 9587 ts_manager.cc:194] Registered new tserver with Master: 71a7ac59363f4dd1bc6b7e38506bc135 (127.9.83.3:41171)
I20260928 01:57:19.210275 9587 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.3:53129
I20260928 01:57:19.210899 9548 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260928 01:57:19.220075 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 9556
I20260928 01:57:19.225937 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 9778
I20260928 01:57:19.232580 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 9909
I20260928 01:57:19.237802 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.83.62:34493
--webserver_interface=127.9.83.62
--webserver_port=36453
--builtin_ntp_servers=127.9.83.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.83.62:34493
--allow_unsafe_replication_factor=true
--catalog_manager_evict_excess_replicas=false
--master_add_server_when_underreplicated=false with env {}
W20260928 01:57:19.339218 10045 flags.cc:432] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20260928 01:57:19.339454 10045 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:19.339483 10045 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:19.339540 10045 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:19.342931 10045 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 01:57:19.342999 10045 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:19.343019 10045 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 01:57:19.343036 10045 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260928 01:57:19.347093 10045 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
--allow_unsafe_replication_factor=true
--catalog_manager_evict_excess_replicas=false
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.9.83.62:34493
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.83.62:34493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.83.62
--webserver_port=36453
--never_fsync=true
--heap_profile_path=/tmp/kudu.10045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:19.348241 10045 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:19.349388 10045 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:19.355073 10053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:19.355059 10051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:19.355059 10050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:19.355396 10045 server_base.cc:1061] running on GCE node
I20260928 01:57:19.355907 10045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:19.356812 10045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:19.357960 10045 hybrid_clock.cc:648] HybridClock initialized: now 1790560639357952 us; error 26 us; skew 500 ppm
I20260928 01:57:19.359723 10045 webserver.cc:533] Webserver started at http://127.9.83.62:36453/ using document root <none> and password file <none>
I20260928 01:57:19.360275 10045 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:19.360352 10045 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:19.364142 10045 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 01:57:19.366151 10059 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:19.367233 10045 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 01:57:19.367343 10045 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
uuid: "a9dce204f3a642aa84955400c67e32fd"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:19.367736 10045 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:19.376361 10045 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:19.377002 10045 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:19.377171 10045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:19.383949 10111 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.62:34493 every 8 connection(s)
I20260928 01:57:19.383949 10045 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.62:34493
I20260928 01:57:19.385062 10045 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
I20260928 01:57:19.388929 10112 sys_catalog.cc:263] Verifying existing consensus state
I20260928 01:57:19.390498 10112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: Bootstrap starting.
I20260928 01:57:19.393167 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 10045
I20260928 01:57:19.405102 10112 log.cc:826] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:19.411222 10112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 01:57:19.411608 10112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: Bootstrap complete.
I20260928 01:57:19.415277 10112 raft_consensus.cc:359] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } }
I20260928 01:57:19.415668 10112 raft_consensus.cc:740] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dce204f3a642aa84955400c67e32fd, State: Initialized, Role: FOLLOWER
I20260928 01:57:19.416087 10112 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } }
I20260928 01:57:19.416201 10112 raft_consensus.cc:399] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 01:57:19.416263 10112 raft_consensus.cc:493] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 01:57:19.416352 10112 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 1 FOLLOWER]: Advancing to term 2
I20260928 01:57:19.417896 10112 raft_consensus.cc:515] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } }
I20260928 01:57:19.418216 10112 leader_election.cc:304] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9dce204f3a642aa84955400c67e32fd; no voters:
I20260928 01:57:19.418494 10112 leader_election.cc:290] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 01:57:19.418614 10117 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 2 FOLLOWER]: Leader election won for term 2
I20260928 01:57:19.418833 10117 raft_consensus.cc:697] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 2 LEADER]: Becoming Leader. State: Replica: a9dce204f3a642aa84955400c67e32fd, State: Running, Role: LEADER
I20260928 01:57:19.419158 10117 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } }
I20260928 01:57:19.419622 10112 sys_catalog.cc:565] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 01:57:19.420509 10118 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a9dce204f3a642aa84955400c67e32fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } } }
I20260928 01:57:19.420488 10119 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9dce204f3a642aa84955400c67e32fd. Latest consensus state: current_term: 2 leader_uuid: "a9dce204f3a642aa84955400c67e32fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } } }
I20260928 01:57:19.420642 10119 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: This master's current role is: LEADER
I20260928 01:57:19.420642 10118 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: This master's current role is: LEADER
I20260928 01:57:19.421277 10126 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260928 01:57:19.424185 10126 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260928 01:57:19.424978 10126 catalog_manager.cc:1308] Loaded cluster ID: 266fe30b665b49798234e98038a7ea12
I20260928 01:57:19.425035 10126 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260928 01:57:19.427049 10126 catalog_manager.cc:1553] Loading token signing keys...
I20260928 01:57:19.427649 10126 catalog_manager.cc:6230] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: Loaded TSK: 0
I20260928 01:57:19.428426 10126 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260928 01:57:19.872185 9775 heartbeater.cc:499] Master 127.9.83.62:34493 was elected leader, sending a full tablet report...
I20260928 01:57:19.876639 10076 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" instance_seqno: 1790560638854638) as {username='slave'} at 127.9.83.1:58637; Asking this server to re-register.
I20260928 01:57:19.877167 9775 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:19.877329 9775 heartbeater.cc:507] Master 127.9.83.62:34493 requested a full tablet report, sending...
I20260928 01:57:19.878294 10076 ts_manager.cc:194] Registered new tserver with Master: d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183)
I20260928 01:57:19.879485 9548 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260928 01:57:19.890859 10075 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44680:
name: "other-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 01:57:19.906370 9709 tablet_service.cc:1511] Processing CreateTablet for tablet f7281e2e7d0b4d60a7b5242c208c1d34 (DEFAULT_TABLE table=other-table [id=93b5daf20e40454cb9722e2bf20c1841]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 01:57:19.907269 9709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7281e2e7d0b4d60a7b5242c208c1d34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:19.911985 10144 tablet_bootstrap.cc:492] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap starting.
I20260928 01:57:19.913887 10144 tablet_bootstrap.cc:654] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Neither blocks nor log segments found. Creating new log.
I20260928 01:57:19.914498 10144 log.cc:826] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:19.916019 10144 tablet_bootstrap.cc:492] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: No bootstrap required, opened a new log
I20260928 01:57:19.916177 10144 ts_tablet_manager.cc:1403] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260928 01:57:19.918962 10144 raft_consensus.cc:359] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:19.919134 10144 raft_consensus.cc:385] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 01:57:19.919176 10144 raft_consensus.cc:740] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f0259187c841a98efbc8ed0eb8021a, State: Initialized, Role: FOLLOWER
I20260928 01:57:19.919510 10144 consensus_queue.cc:260] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:19.919608 10144 raft_consensus.cc:399] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 01:57:19.919648 10144 raft_consensus.cc:493] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 01:57:19.919708 10144 raft_consensus.cc:3060] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 0 FOLLOWER]: Advancing to term 1
I20260928 01:57:19.920401 10144 raft_consensus.cc:515] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:19.920701 10144 leader_election.cc:304] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3f0259187c841a98efbc8ed0eb8021a; no voters:
I20260928 01:57:19.920994 10144 leader_election.cc:290] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 01:57:19.921126 10146 raft_consensus.cc:2804] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 01:57:19.921336 10146 raft_consensus.cc:697] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 1 LEADER]: Becoming Leader. State: Replica: d3f0259187c841a98efbc8ed0eb8021a, State: Running, Role: LEADER
I20260928 01:57:19.921663 10146 consensus_queue.cc:237] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:19.922159 10144 ts_tablet_manager.cc:1434] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260928 01:57:19.925390 10075 catalog_manager.cc:5838] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a reported cstate change: term changed from 0 to 1, leader changed from <none> to d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1). New cstate: current_term: 1 leader_uuid: "d3f0259187c841a98efbc8ed0eb8021a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } health_report { overall_health: HEALTHY } } }
I20260928 01:57:19.961238 9776 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling FlushMRSOp(f7281e2e7d0b4d60a7b5242c208c1d34): perf score=1.000000
I20260928 01:57:19.984797 9666 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: FlushMRSOp(f7281e2e7d0b4d60a7b5242c208c1d34) complete. Timing: real 0.023s user 0.021s sys 0.000s Metrics: {"bytes_written":8058,"cfile_init":1,"compiler_manager_pool.queue_time_us":206,"dirs.queue_time_us":228,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":2795,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":553,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":150,"thread_start_us":239,"threads_started":2}
I20260928 01:57:19.985733 9776 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling FlushMRSOp(f7281e2e7d0b4d60a7b5242c208c1d34): perf score=1.000000
I20260928 01:57:19.997308 9666 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: FlushMRSOp(f7281e2e7d0b4d60a7b5242c208c1d34) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":10365,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":2194,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":497,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":250}
I20260928 01:57:19.998152 9776 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling FlushMRSOp(f7281e2e7d0b4d60a7b5242c208c1d34): perf score=1.000000
I20260928 01:57:20.005318 9666 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: FlushMRSOp(f7281e2e7d0b4d60a7b5242c208c1d34) complete. Timing: real 0.007s user 0.005s sys 0.000s Metrics: {"bytes_written":6945,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1770,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":390,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":100}
I20260928 01:57:20.006001 9776 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling UndoDeltaBlockGCOp(f7281e2e7d0b4d60a7b5242c208c1d34): 3600 bytes on disk
I20260928 01:57:20.007206 9666 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: UndoDeltaBlockGCOp(f7281e2e7d0b4d60a7b5242c208c1d34) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":155,"lbm_reads_lt_1ms":12}
I20260928 01:57:20.007704 9776 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling FlushMRSOp(f7281e2e7d0b4d60a7b5242c208c1d34): perf score=1.000000
I20260928 01:57:20.011088 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.2:44831
--local_ip_for_outbound_sockets=127.9.83.2
--tserver_master_addrs=127.9.83.62:34493
--webserver_port=37769
--webserver_interface=127.9.83.2
--builtin_ntp_servers=127.9.83.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=0
--maintenance_manager_polling_interval_ms=100 with env {}
I20260928 01:57:20.014304 9666 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: FlushMRSOp(f7281e2e7d0b4d60a7b5242c208c1d34) complete. Timing: real 0.006s user 0.006s sys 0.000s Metrics: {"bytes_written":5818,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1706,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":489,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":50}
I20260928 01:57:20.014936 9776 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling UndoDeltaBlockGCOp(f7281e2e7d0b4d60a7b5242c208c1d34): 450 bytes on disk
I20260928 01:57:20.015281 9666 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: UndoDeltaBlockGCOp(f7281e2e7d0b4d60a7b5242c208c1d34) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4}
I20260928 01:57:20.015689 9776 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling FlushMRSOp(f7281e2e7d0b4d60a7b5242c208c1d34): perf score=1.000000
I20260928 01:57:20.020861 9666 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: FlushMRSOp(f7281e2e7d0b4d60a7b5242c208c1d34) complete. Timing: real 0.005s user 0.004s sys 0.000s Metrics: {"bytes_written":5817,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1276,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":362,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":50}
I20260928 01:57:20.021476 9776 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling CompactRowSetsOp(f7281e2e7d0b4d60a7b5242c208c1d34): perf score=0.041989
I20260928 01:57:20.042892 9666 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: CompactRowSetsOp(f7281e2e7d0b4d60a7b5242c208c1d34) complete. Timing: real 0.021s user 0.021s sys 0.000s Metrics: {"bytes_written":18191,"cfile_cache_hit":25,"cfile_cache_hit_bytes":11481,"cfile_cache_miss":65,"cfile_cache_miss_bytes":26473,"cfile_init":12,"delta_iterators_relevant":10,"dirs.queue_time_us":2343,"dirs.run_cpu_time_us":376,"dirs.run_wall_time_us":2326,"drs_written":1,"lbm_read_time_us":949,"lbm_reads_lt_1ms":113,"lbm_write_time_us":459,"lbm_writes_lt_1ms":23,"num_input_rowsets":5,"peak_mem_usage":163840,"rows_written":600,"thread_start_us":824,"threads_started":7}
I20260928 01:57:20.043579 9776 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling UndoDeltaBlockGCOp(f7281e2e7d0b4d60a7b5242c208c1d34): 4615 bytes on disk
I20260928 01:57:20.044021 9666 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: UndoDeltaBlockGCOp(f7281e2e7d0b4d60a7b5242c208c1d34) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20260928 01:57:20.140436 10156 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:20.140669 10156 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:20.140734 10156 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:20.144200 10156 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:20.144301 10156 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260928 01:57:20.144368 10156 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.2
I20260928 01:57:20.148424 10156 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.2:44831
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.83.2
--webserver_port=37769
--flush_threshold_mb=0
--tserver_master_addrs=127.9.83.62:34493
--never_fsync=true
--heap_profile_path=/tmp/kudu.10156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:20.149535 10156 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:20.150663 10156 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:20.157426 10168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:20.157426 10171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:20.157429 10169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:20.157794 10156 server_base.cc:1061] running on GCE node
I20260928 01:57:20.158183 10156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:20.158737 10156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:20.159943 10156 hybrid_clock.cc:648] HybridClock initialized: now 1790560640159920 us; error 38 us; skew 500 ppm
I20260928 01:57:20.161831 10156 webserver.cc:533] Webserver started at http://127.9.83.2:37769/ using document root <none> and password file <none>
I20260928 01:57:20.162410 10156 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:20.162483 10156 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:20.165922 10156 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 01:57:20.167739 10177 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:20.168718 10156 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 01:57:20.168802 10156 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/wal
uuid: "00af439d64e84c9db541f5414ac1d2e4"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:20.169196 10156 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:20.210620 10156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:20.211318 10156 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:20.211485 10156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:20.212051 10156 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:20.213021 10156 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 01:57:20.213068 10156 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:20.213105 10156 ts_tablet_manager.cc:616] Registered 0 tablets
I20260928 01:57:20.213119 10156 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:20.221562 10156 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.2:44831
I20260928 01:57:20.221616 10290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.2:44831 every 8 connection(s)
I20260928 01:57:20.222586 10156 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb
I20260928 01:57:20.228073 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 10156
I20260928 01:57:20.228478 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.3:41171
--local_ip_for_outbound_sockets=127.9.83.3
--tserver_master_addrs=127.9.83.62:34493
--webserver_port=44237
--webserver_interface=127.9.83.3
--builtin_ntp_servers=127.9.83.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=0
--maintenance_manager_polling_interval_ms=100 with env {}
I20260928 01:57:20.231355 10291 heartbeater.cc:344] Connected to a master server at 127.9.83.62:34493
I20260928 01:57:20.231593 10291 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:20.232095 10291 heartbeater.cc:507] Master 127.9.83.62:34493 requested a full tablet report, sending...
I20260928 01:57:20.233125 10070 ts_manager.cc:194] Registered new tserver with Master: 00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831)
I20260928 01:57:20.233887 10070 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.2:53537
W20260928 01:57:20.329840 10294 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:20.330060 10294 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:20.330125 10294 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:20.333483 10294 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:20.333578 10294 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260928 01:57:20.333643 10294 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.3
I20260928 01:57:20.337484 10294 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.3:41171
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.83.3
--webserver_port=44237
--flush_threshold_mb=0
--tserver_master_addrs=127.9.83.62:34493
--never_fsync=true
--heap_profile_path=/tmp/kudu.10294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:20.338559 10294 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:20.339718 10294 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:20.346179 10302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:20.346179 10300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:20.346174 10299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:20.346589 10294 server_base.cc:1061] running on GCE node
I20260928 01:57:20.346998 10294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:20.347594 10294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:20.348759 10294 hybrid_clock.cc:648] HybridClock initialized: now 1790560640348743 us; error 32 us; skew 500 ppm
I20260928 01:57:20.350602 10294 webserver.cc:533] Webserver started at http://127.9.83.3:44237/ using document root <none> and password file <none>
I20260928 01:57:20.351244 10294 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:20.351322 10294 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:20.354843 10294 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260928 01:57:20.356897 10308 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:20.357970 10294 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 01:57:20.358091 10294 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/wal
uuid: "71a7ac59363f4dd1bc6b7e38506bc135"
format_stamp: "Formatted at 2026-09-28 01:57:19 on dist-test-slave-lkt6"
I20260928 01:57:20.358510 10294 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:20.378327 10294 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:20.379027 10294 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:20.379197 10294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:20.379791 10294 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:20.380813 10294 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 01:57:20.380884 10294 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:20.380941 10294 ts_tablet_manager.cc:616] Registered 0 tablets
I20260928 01:57:20.380975 10294 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:20.390700 10294 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.3:41171
I20260928 01:57:20.390841 10421 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.3:41171 every 8 connection(s)
I20260928 01:57:20.395114 10294 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb
I20260928 01:57:20.404179 10422 heartbeater.cc:344] Connected to a master server at 127.9.83.62:34493
I20260928 01:57:20.404440 10422 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:20.404546 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 10294
I20260928 01:57:20.404709 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 9647
I20260928 01:57:20.405018 10422 heartbeater.cc:507] Master 127.9.83.62:34493 requested a full tablet report, sending...
I20260928 01:57:20.406153 10070 ts_manager.cc:194] Registered new tserver with Master: 71a7ac59363f4dd1bc6b7e38506bc135 (127.9.83.3:41171)
I20260928 01:57:20.406893 10070 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.3:41251
I20260928 01:57:20.411885 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 10045
I20260928 01:57:20.417832 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.83.62:34493
--webserver_interface=127.9.83.62
--webserver_port=36453
--builtin_ntp_servers=127.9.83.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.83.62:34493
--allow_unsafe_replication_factor=true
--catalog_manager_evict_excess_replicas=false
--master_add_server_when_underreplicated=false with env {}
W20260928 01:57:20.519668 10425 flags.cc:432] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20260928 01:57:20.519896 10425 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:20.519927 10425 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:20.519986 10425 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:20.523389 10425 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 01:57:20.523450 10425 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:20.523469 10425 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 01:57:20.523486 10425 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260928 01:57:20.527478 10425 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
--allow_unsafe_replication_factor=true
--catalog_manager_evict_excess_replicas=false
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.9.83.62:34493
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.83.62:34493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.83.62
--webserver_port=36453
--never_fsync=true
--heap_profile_path=/tmp/kudu.10425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:20.528548 10425 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:20.529685 10425 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:20.534945 10433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:20.534940 10431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:20.534940 10430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:20.535187 10425 server_base.cc:1061] running on GCE node
I20260928 01:57:20.535681 10425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:20.536540 10425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:20.537686 10425 hybrid_clock.cc:648] HybridClock initialized: now 1790560640537671 us; error 32 us; skew 500 ppm
I20260928 01:57:20.539510 10425 webserver.cc:533] Webserver started at http://127.9.83.62:36453/ using document root <none> and password file <none>
I20260928 01:57:20.540047 10425 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:20.540102 10425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:20.543303 10425 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 01:57:20.545050 10439 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:20.545972 10425 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 01:57:20.546073 10425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
uuid: "a9dce204f3a642aa84955400c67e32fd"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:20.546408 10425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:20.558854 10425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:20.559541 10425 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:20.559682 10425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:20.566501 10491 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.62:34493 every 8 connection(s)
I20260928 01:57:20.566510 10425 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.62:34493
I20260928 01:57:20.567653 10425 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
I20260928 01:57:20.571537 10492 sys_catalog.cc:263] Verifying existing consensus state
I20260928 01:57:20.572801 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 10425
I20260928 01:57:20.573114 10492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: Bootstrap starting.
I20260928 01:57:20.587900 10492 log.cc:826] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:20.595676 10492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 01:57:20.596095 10492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: Bootstrap complete.
I20260928 01:57:20.599123 10492 raft_consensus.cc:359] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } }
I20260928 01:57:20.599457 10492 raft_consensus.cc:740] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dce204f3a642aa84955400c67e32fd, State: Initialized, Role: FOLLOWER
I20260928 01:57:20.599860 10492 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } }
I20260928 01:57:20.599969 10492 raft_consensus.cc:399] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 01:57:20.600029 10492 raft_consensus.cc:493] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 01:57:20.600128 10492 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 2 FOLLOWER]: Advancing to term 3
I20260928 01:57:20.601625 10492 raft_consensus.cc:515] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } }
I20260928 01:57:20.601927 10492 leader_election.cc:304] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9dce204f3a642aa84955400c67e32fd; no voters:
I20260928 01:57:20.602198 10492 leader_election.cc:290] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 01:57:20.602308 10497 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 3 FOLLOWER]: Leader election won for term 3
I20260928 01:57:20.602494 10497 raft_consensus.cc:697] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [term 3 LEADER]: Becoming Leader. State: Replica: a9dce204f3a642aa84955400c67e32fd, State: Running, Role: LEADER
I20260928 01:57:20.602783 10497 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } }
I20260928 01:57:20.603262 10492 sys_catalog.cc:565] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 01:57:20.604090 10499 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9dce204f3a642aa84955400c67e32fd. Latest consensus state: current_term: 3 leader_uuid: "a9dce204f3a642aa84955400c67e32fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } } }
I20260928 01:57:20.604225 10499 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: This master's current role is: LEADER
I20260928 01:57:20.604087 10498 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a9dce204f3a642aa84955400c67e32fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dce204f3a642aa84955400c67e32fd" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 34493 } } }
I20260928 01:57:20.604735 10498 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd [sys.catalog]: This master's current role is: LEADER
I20260928 01:57:20.607939 10513 catalog_manager.cc:1308] Loaded cluster ID: 266fe30b665b49798234e98038a7ea12
I20260928 01:57:20.607997 10513 catalog_manager.cc:1601] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: loading cluster ID for follower catalog manager: success
I20260928 01:57:20.609692 10513 catalog_manager.cc:1623] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: acquiring CA information for follower catalog manager: success
I20260928 01:57:20.610726 10513 catalog_manager.cc:1651] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 01:57:20.610873 10506 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260928 01:57:20.612079 10506 catalog_manager.cc:692] Loaded metadata for table other-table [id=93b5daf20e40454cb9722e2bf20c1841]
I20260928 01:57:20.613013 10506 tablet_loader.cc:96] loaded metadata for tablet f7281e2e7d0b4d60a7b5242c208c1d34 (table other-table [id=93b5daf20e40454cb9722e2bf20c1841])
I20260928 01:57:20.613144 10506 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260928 01:57:20.613548 10506 catalog_manager.cc:1308] Loaded cluster ID: 266fe30b665b49798234e98038a7ea12
I20260928 01:57:20.613593 10506 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260928 01:57:20.614210 10506 catalog_manager.cc:1553] Loading token signing keys...
I20260928 01:57:20.614622 10506 catalog_manager.cc:6230] T 00000000000000000000000000000000 P a9dce204f3a642aa84955400c67e32fd: Loaded TSK: 0
I20260928 01:57:20.615265 10506 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260928 01:57:21.235914 10291 heartbeater.cc:499] Master 127.9.83.62:34493 was elected leader, sending a full tablet report...
I20260928 01:57:21.240731 10456 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" instance_seqno: 1790560640219290) as {username='slave'} at 127.9.83.2:51747; Asking this server to re-register.
I20260928 01:57:21.241283 10291 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:21.241464 10291 heartbeater.cc:507] Master 127.9.83.62:34493 requested a full tablet report, sending...
I20260928 01:57:21.242374 10455 ts_manager.cc:194] Registered new tserver with Master: 00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831)
I20260928 01:57:21.408936 10422 heartbeater.cc:499] Master 127.9.83.62:34493 was elected leader, sending a full tablet report...
I20260928 01:57:21.412822 10455 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" instance_seqno: 1790560640387938) as {username='slave'} at 127.9.83.3:43149; Asking this server to re-register.
I20260928 01:57:21.413223 10422 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:21.413375 10422 heartbeater.cc:507] Master 127.9.83.62:34493 requested a full tablet report, sending...
I20260928 01:57:21.413918 10455 ts_manager.cc:194] Registered new tserver with Master: 71a7ac59363f4dd1bc6b7e38506bc135 (127.9.83.3:41171)
I20260928 01:57:21.420521 9548 external_mini_cluster.cc:955] 2 TS(s) registered with all masters
I20260928 01:57:21.430444 10455 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44700:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 01:57:21.431792 10455 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20260928 01:57:21.446355 10225 tablet_service.cc:1511] Processing CreateTablet for tablet 6bcce401df724b83973071baa1e5828b (DEFAULT_TABLE table=test-workload [id=91ee3d2cfd8540cf8d3fee1acc68ce1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 01:57:21.446359 10356 tablet_service.cc:1511] Processing CreateTablet for tablet 6bcce401df724b83973071baa1e5828b (DEFAULT_TABLE table=test-workload [id=91ee3d2cfd8540cf8d3fee1acc68ce1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 01:57:21.447232 10225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bcce401df724b83973071baa1e5828b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:21.447232 10356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bcce401df724b83973071baa1e5828b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:21.453140 10530 tablet_bootstrap.cc:492] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4: Bootstrap starting.
I20260928 01:57:21.453676 10531 tablet_bootstrap.cc:492] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135: Bootstrap starting.
I20260928 01:57:21.455633 10531 tablet_bootstrap.cc:654] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135: Neither blocks nor log segments found. Creating new log.
I20260928 01:57:21.456454 10531 log.cc:826] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:21.456751 10530 tablet_bootstrap.cc:654] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4: Neither blocks nor log segments found. Creating new log.
I20260928 01:57:21.457490 10530 log.cc:826] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:21.458427 10531 tablet_bootstrap.cc:492] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135: No bootstrap required, opened a new log
I20260928 01:57:21.458627 10531 ts_tablet_manager.cc:1403] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 01:57:21.459321 10530 tablet_bootstrap.cc:492] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4: No bootstrap required, opened a new log
I20260928 01:57:21.459517 10530 ts_tablet_manager.cc:1403] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20260928 01:57:21.462342 10531 raft_consensus.cc:359] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } }
I20260928 01:57:21.462656 10531 raft_consensus.cc:385] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 01:57:21.462724 10531 raft_consensus.cc:740] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71a7ac59363f4dd1bc6b7e38506bc135, State: Initialized, Role: FOLLOWER
I20260928 01:57:21.463014 10530 raft_consensus.cc:359] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } }
I20260928 01:57:21.463186 10531 consensus_queue.cc:260] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } }
I20260928 01:57:21.463248 10530 raft_consensus.cc:385] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 01:57:21.463290 10530 raft_consensus.cc:740] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00af439d64e84c9db541f5414ac1d2e4, State: Initialized, Role: FOLLOWER
I20260928 01:57:21.463672 10530 consensus_queue.cc:260] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } }
I20260928 01:57:21.464087 10531 ts_tablet_manager.cc:1434] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260928 01:57:21.464437 10530 ts_tablet_manager.cc:1434] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
W20260928 01:57:21.497903 10423 tablet.cc:2433] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 01:57:21.521343 10535 raft_consensus.cc:493] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 01:57:21.521657 10535 raft_consensus.cc:515] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } }
I20260928 01:57:21.522728 10535 leader_election.cc:290] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71a7ac59363f4dd1bc6b7e38506bc135 (127.9.83.3:41171)
W20260928 01:57:21.525614 10292 tablet.cc:2433] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 01:57:21.526224 10376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6bcce401df724b83973071baa1e5828b" candidate_uuid: "00af439d64e84c9db541f5414ac1d2e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" is_pre_election: true
I20260928 01:57:21.526489 10376 raft_consensus.cc:2468] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00af439d64e84c9db541f5414ac1d2e4 in term 0.
I20260928 01:57:21.527029 10178 leader_election.cc:304] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 00af439d64e84c9db541f5414ac1d2e4, 71a7ac59363f4dd1bc6b7e38506bc135; no voters:
I20260928 01:57:21.527257 10535 raft_consensus.cc:2804] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 01:57:21.527334 10535 raft_consensus.cc:493] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 01:57:21.527384 10535 raft_consensus.cc:3060] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 01:57:21.528250 10535 raft_consensus.cc:515] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } }
I20260928 01:57:21.528563 10535 leader_election.cc:290] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [CANDIDATE]: Term 1 election: Requested vote from peers 71a7ac59363f4dd1bc6b7e38506bc135 (127.9.83.3:41171)
I20260928 01:57:21.529011 10376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6bcce401df724b83973071baa1e5828b" candidate_uuid: "00af439d64e84c9db541f5414ac1d2e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71a7ac59363f4dd1bc6b7e38506bc135"
I20260928 01:57:21.529142 10376 raft_consensus.cc:3060] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135 [term 0 FOLLOWER]: Advancing to term 1
I20260928 01:57:21.529960 10376 raft_consensus.cc:2468] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00af439d64e84c9db541f5414ac1d2e4 in term 1.
I20260928 01:57:21.530273 10178 leader_election.cc:304] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 00af439d64e84c9db541f5414ac1d2e4, 71a7ac59363f4dd1bc6b7e38506bc135; no voters:
I20260928 01:57:21.530433 10535 raft_consensus.cc:2804] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 01:57:21.530833 10535 raft_consensus.cc:697] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [term 1 LEADER]: Becoming Leader. State: Replica: 00af439d64e84c9db541f5414ac1d2e4, State: Running, Role: LEADER
I20260928 01:57:21.531159 10535 consensus_queue.cc:237] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } }
I20260928 01:57:21.534165 10456 catalog_manager.cc:5838] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2). New cstate: current_term: 1 leader_uuid: "00af439d64e84c9db541f5414ac1d2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260928 01:57:21.556994 10376 raft_consensus.cc:1275] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135 [term 1 FOLLOWER]: Refusing update from remote peer 00af439d64e84c9db541f5414ac1d2e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 01:57:21.557758 10535 consensus_queue.cc:1048] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 01:57:21.598893 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.622370 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.023s user 0.021s sys 0.000s Metrics: {"bytes_written":10353,"cfile_init":1,"compiler_manager_pool.queue_time_us":224,"dirs.queue_time_us":224,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":819,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":568,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":250,"thread_start_us":262,"threads_started":2}
I20260928 01:57:21.623200 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.626559 10292 maintenance_manager.cc:419] P 00af439d64e84c9db541f5414ac1d2e4: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.633821 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.010s user 0.008s sys 0.000s Metrics: {"bytes_written":8067,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1115,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1786,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":150}
I20260928 01:57:21.634588 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 2834 bytes on disk
I20260928 01:57:21.635314 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":105,"lbm_reads_lt_1ms":8}
I20260928 01:57:21.635831 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.640990 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.005s user 0.004s sys 0.000s Metrics: {"bytes_written":5814,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":1107,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":371,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":50}
I20260928 01:57:21.641510 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 450 bytes on disk
I20260928 01:57:21.641813 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
I20260928 01:57:21.642174 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling CompactRowSetsOp(6bcce401df724b83973071baa1e5828b): perf score=0.023993
I20260928 01:57:21.668097 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: CompactRowSetsOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.026s user 0.017s sys 0.003s Metrics: {"bytes_written":14861,"cfile_cache_hit":15,"cfile_cache_hit_bytes":8532,"cfile_cache_miss":39,"cfile_cache_miss_bytes":19466,"cfile_init":7,"delta_iterators_relevant":6,"dirs.queue_time_us":2724,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":5206,"drs_written":1,"lbm_read_time_us":1918,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":66,"lbm_write_time_us":432,"lbm_writes_lt_1ms":23,"mutex_wait_us":22,"num_input_rowsets":3,"peak_mem_usage":98304,"rows_written":450,"spinlock_wait_cycles":8064,"thread_start_us":368,"threads_started":6}
I20260928 01:57:21.668625 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.679549 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.011s user 0.007s sys 0.001s Metrics: {"bytes_written":8077,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":938,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2526,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":150}
I20260928 01:57:21.680022 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.683836 10182 maintenance_manager.cc:643] P 00af439d64e84c9db541f5414ac1d2e4: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.057s user 0.019s sys 0.008s Metrics: {"bytes_written":13690,"cfile_init":1,"compiler_manager_pool.queue_time_us":20744,"dirs.queue_time_us":14836,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":471,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":523,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":400,"thread_start_us":255,"threads_started":2}
I20260928 01:57:21.684852 10292 maintenance_manager.cc:419] P 00af439d64e84c9db541f5414ac1d2e4: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.688738 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.009s user 0.005s sys 0.000s Metrics: {"bytes_written":5820,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":902,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":465,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":50}
I20260928 01:57:21.689286 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 4943 bytes on disk
I20260928 01:57:21.689841 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":146,"lbm_reads_lt_1ms":12}
I20260928 01:57:21.690266 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.697628 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.007s user 0.005s sys 0.000s Metrics: {"bytes_written":5814,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":943,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":490,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":50}
I20260928 01:57:21.698215 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 450 bytes on disk
I20260928 01:57:21.698720 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20260928 01:57:21.699419 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.708652 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.009s user 0.005s sys 0.000s Metrics: {"bytes_written":5823,"cfile_init":1,"dirs.queue_time_us":30,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":1030,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":480,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":50}
I20260928 01:57:21.709298 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.717386 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.008s user 0.005s sys 0.000s Metrics: {"bytes_written":5820,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":876,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":449,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":50}
I20260928 01:57:21.718127 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 900 bytes on disk
I20260928 01:57:21.718673 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":111,"lbm_reads_lt_1ms":8}
I20260928 01:57:21.719277 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.720927 10182 maintenance_manager.cc:643] P 00af439d64e84c9db541f5414ac1d2e4: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.036s user 0.010s sys 0.002s Metrics: {"bytes_written":11500,"cfile_init":1,"dirs.queue_time_us":3401,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":1021,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6856,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"peak_mem_usage":0,"rows_written":300}
I20260928 01:57:21.725097 10292 maintenance_manager.cc:419] P 00af439d64e84c9db541f5414ac1d2e4: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 5145 bytes on disk
I20260928 01:57:21.726058 10182 maintenance_manager.cc:643] P 00af439d64e84c9db541f5414ac1d2e4: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":115,"lbm_reads_lt_1ms":8}
I20260928 01:57:21.726514 10292 maintenance_manager.cc:419] P 00af439d64e84c9db541f5414ac1d2e4: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.736655 10182 maintenance_manager.cc:643] P 00af439d64e84c9db541f5414ac1d2e4: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.010s user 0.008s sys 0.000s Metrics: {"bytes_written":8079,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":1050,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":491,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":150}
I20260928 01:57:21.737422 10292 maintenance_manager.cc:419] P 00af439d64e84c9db541f5414ac1d2e4: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.738586 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.019s user 0.003s sys 0.003s Metrics: {"bytes_written":5821,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":965,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":469,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":50}
I20260928 01:57:21.739935 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.745267 10182 maintenance_manager.cc:643] P 00af439d64e84c9db541f5414ac1d2e4: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.008s user 0.005s sys 0.001s Metrics: {"bytes_written":5823,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":2391,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":553,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":50}
I20260928 01:57:21.746676 10292 maintenance_manager.cc:419] P 00af439d64e84c9db541f5414ac1d2e4: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.747572 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.007s user 0.004s sys 0.002s Metrics: {"bytes_written":5823,"cfile_init":1,"dirs.queue_time_us":146,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1162,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":502,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":50}
I20260928 01:57:21.748348 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 900 bytes on disk
I20260928 01:57:21.748898 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":115,"lbm_reads_lt_1ms":8}
I20260928 01:57:21.749580 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.759155 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.1:41183
--local_ip_for_outbound_sockets=127.9.83.1
--tserver_master_addrs=127.9.83.62:34493
--webserver_port=46379
--webserver_interface=127.9.83.1
--builtin_ntp_servers=127.9.83.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=0
--maintenance_manager_polling_interval_ms=100
--fault_crash_after_tc_files_fetched=1.0 with env {}
I20260928 01:57:21.760880 10182 maintenance_manager.cc:643] P 00af439d64e84c9db541f5414ac1d2e4: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.010s user 0.005s sys 0.000s Metrics: {"bytes_written":5818,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":1150,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":460,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":50}
I20260928 01:57:21.761449 10292 maintenance_manager.cc:419] P 00af439d64e84c9db541f5414ac1d2e4: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 1989 bytes on disk
I20260928 01:57:21.762070 10182 maintenance_manager.cc:643] P 00af439d64e84c9db541f5414ac1d2e4: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":158,"lbm_reads_lt_1ms":12}
I20260928 01:57:21.762557 10292 maintenance_manager.cc:419] P 00af439d64e84c9db541f5414ac1d2e4: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.765017 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.015s user 0.005s sys 0.000s Metrics: {"bytes_written":5818,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":2881,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":593,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":50}
I20260928 01:57:21.765847 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 450 bytes on disk
I20260928 01:57:21.766265 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20260928 01:57:21.766988 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling FlushMRSOp(6bcce401df724b83973071baa1e5828b): perf score=1.000000
I20260928 01:57:21.769321 10182 maintenance_manager.cc:643] P 00af439d64e84c9db541f5414ac1d2e4: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.007s user 0.005s sys 0.000s Metrics: {"bytes_written":5813,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":983,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":503,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":50}
I20260928 01:57:21.769901 10292 maintenance_manager.cc:419] P 00af439d64e84c9db541f5414ac1d2e4: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 450 bytes on disk
I20260928 01:57:21.770280 10182 maintenance_manager.cc:643] P 00af439d64e84c9db541f5414ac1d2e4: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20260928 01:57:21.770761 10292 maintenance_manager.cc:419] P 00af439d64e84c9db541f5414ac1d2e4: Scheduling CompactRowSetsOp(6bcce401df724b83973071baa1e5828b): perf score=0.050984
I20260928 01:57:21.782929 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: FlushMRSOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.016s user 0.006s sys 0.000s Metrics: {"bytes_written":5813,"cfile_init":1,"dirs.queue_time_us":1790,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1036,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1051,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":50}
I20260928 01:57:21.783917 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 450 bytes on disk
I20260928 01:57:21.784476 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20260928 01:57:21.785233 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling CompactRowSetsOp(6bcce401df724b83973071baa1e5828b): perf score=0.086979
I20260928 01:57:21.807642 10182 maintenance_manager.cc:643] P 00af439d64e84c9db541f5414ac1d2e4: CompactRowSetsOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.037s user 0.036s sys 0.000s Metrics: {"bytes_written":27230,"cfile_cache_hit":30,"cfile_cache_hit_bytes":18888,"cfile_cache_miss":78,"cfile_cache_miss_bytes":43654,"cfile_init":13,"delta_iterators_relevant":12,"dirs.queue_time_us":104,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1155,"drs_written":1,"lbm_read_time_us":1595,"lbm_reads_lt_1ms":130,"lbm_write_time_us":561,"lbm_writes_lt_1ms":23,"mutex_wait_us":29,"num_input_rowsets":6,"peak_mem_usage":196608,"rows_written":1000,"spinlock_wait_cycles":1792,"thread_start_us":420,"threads_started":7}
I20260928 01:57:21.808274 10292 maintenance_manager.cc:419] P 00af439d64e84c9db541f5414ac1d2e4: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 7929 bytes on disk
I20260928 01:57:21.808733 10182 maintenance_manager.cc:643] P 00af439d64e84c9db541f5414ac1d2e4: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20260928 01:57:21.836076 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: CompactRowSetsOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.051s user 0.038s sys 0.000s Metrics: {"bytes_written":27230,"cfile_cache_hit":50,"cfile_cache_hit_bytes":19317,"cfile_cache_miss":130,"cfile_cache_miss_bytes":45781,"cfile_init":21,"delta_iterators_relevant":20,"dirs.queue_time_us":518,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1144,"drs_written":1,"lbm_read_time_us":2539,"lbm_reads_lt_1ms":214,"lbm_write_time_us":692,"lbm_writes_lt_1ms":23,"mutex_wait_us":1,"num_input_rowsets":10,"peak_mem_usage":327680,"rows_written":1000,"thread_start_us":131,"threads_started":1}
I20260928 01:57:21.836624 10423 maintenance_manager.cc:419] P 71a7ac59363f4dd1bc6b7e38506bc135: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 7929 bytes on disk
I20260928 01:57:21.837162 10313 maintenance_manager.cc:643] P 71a7ac59363f4dd1bc6b7e38506bc135: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20260928 01:57:21.916702 10555 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:21.916917 10555 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:21.916982 10555 flags.cc:432] Enabled unsafe flag: --fault_crash_after_tc_files_fetched=1
W20260928 01:57:21.917009 10555 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:21.920500 10555 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:21.920598 10555 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260928 01:57:21.920680 10555 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.1
I20260928 01:57:21.924706 10555 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.1:41183
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.83.1
--webserver_port=46379
--flush_threshold_mb=0
--tserver_master_addrs=127.9.83.62:34493
--fault_crash_after_tc_files_fetched=1
--never_fsync=true
--heap_profile_path=/tmp/kudu.10555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:21.925756 10555 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:21.926901 10555 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:21.933024 10568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:21.932992 10569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:21.933070 10555 server_base.cc:1061] running on GCE node
W20260928 01:57:21.933000 10571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:21.933631 10555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:21.934118 10555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:21.935258 10555 hybrid_clock.cc:648] HybridClock initialized: now 1790560641935243 us; error 32 us; skew 500 ppm
I20260928 01:57:21.936981 10555 webserver.cc:533] Webserver started at http://127.9.83.1:46379/ using document root <none> and password file <none>
I20260928 01:57:21.937500 10555 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:21.937564 10555 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:21.940811 10555 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 01:57:21.945489 10582 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:21.946417 10555 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260928 01:57:21.946519 10555 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
uuid: "d3f0259187c841a98efbc8ed0eb8021a"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:21.946959 10555 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/data
Total live blocks: 5
Total live bytes: 22806
Total live bytes (after alignment): 36864
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:21.971252 10555 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:21.971838 10555 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:21.971982 10555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:21.972493 10555 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:21.973661 10589 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 01:57:21.977741 10555 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 01:57:21.977802 10555 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 01:57:21.977856 10555 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20260928 01:57:21.979313 10555 ts_tablet_manager.cc:616] Registered 1 tablets
I20260928 01:57:21.979357 10555 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260928 01:57:21.979692 10589 tablet_bootstrap.cc:492] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap starting.
I20260928 01:57:21.988380 10555 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.1:41183
I20260928 01:57:21.988401 10696 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.1:41183 every 8 connection(s)
I20260928 01:57:21.989362 10555 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
I20260928 01:57:21.997557 10697 heartbeater.cc:344] Connected to a master server at 127.9.83.62:34493
I20260928 01:57:21.997807 10697 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:21.997807 10589 log.cc:826] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:21.997918 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 10555
I20260928 01:57:21.998411 10697 heartbeater.cc:507] Master 127.9.83.62:34493 requested a full tablet report, sending...
I20260928 01:57:22.000497 10450 ts_manager.cc:194] Registered new tserver with Master: d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183)
I20260928 01:57:22.001901 10450 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.1:41045
I20260928 01:57:22.005080 10245 consensus_queue.cc:237] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 1.21, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } } peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } attrs { } }
I20260928 01:57:22.006448 10376 raft_consensus.cc:1275] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135 [term 1 FOLLOWER]: Refusing update from remote peer 00af439d64e84c9db541f5414ac1d2e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 21. Preceding OpId from leader: term: 1 index: 22. (index mismatch)
I20260928 01:57:22.006978 10535 consensus_queue.cc:1048] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260928 01:57:22.008826 10536 raft_consensus.cc:2955] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [term 1 LEADER]: Committing config change with OpId 1.22: config changed from index -1 to 22, VOTER d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } } peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } attrs { } } }
I20260928 01:57:22.009377 10589 tablet_bootstrap.cc:492] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=12} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 01:57:22.009279 10376 raft_consensus.cc:2955] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135 [term 1 FOLLOWER]: Committing config change with OpId 1.22: config changed from index -1 to 22, VOTER d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } } peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } attrs { } } }
I20260928 01:57:22.009754 10589 tablet_bootstrap.cc:492] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap complete.
I20260928 01:57:22.010887 10589 ts_tablet_manager.cc:1403] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.009s
I20260928 01:57:22.012146 10454 catalog_manager.cc:5838] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 reported cstate change: config changed from index -1 to 22, VOTER d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1) added. New cstate: current_term: 1 leader_uuid: "00af439d64e84c9db541f5414ac1d2e4" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } attrs { } health_report { overall_health: UNKNOWN } } }
W20260928 01:57:22.013546 10181 consensus_peers.cc:597] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 -> Peer d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183): Couldn't send request to peer d3f0259187c841a98efbc8ed0eb8021a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6bcce401df724b83973071baa1e5828b. This is attempt 1: this message will repeat every 5th retry.
I20260928 01:57:22.013326 10589 raft_consensus.cc:359] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:22.013715 10589 raft_consensus.cc:740] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f0259187c841a98efbc8ed0eb8021a, State: Initialized, Role: FOLLOWER
I20260928 01:57:22.014222 10589 consensus_queue.cc:260] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:22.014372 10589 raft_consensus.cc:399] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 01:57:22.014439 10589 raft_consensus.cc:493] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 01:57:22.014534 10589 raft_consensus.cc:3060] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Advancing to term 2
I20260928 01:57:22.016090 10589 raft_consensus.cc:515] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:22.016412 10589 leader_election.cc:304] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3f0259187c841a98efbc8ed0eb8021a; no voters:
I20260928 01:57:22.016714 10589 leader_election.cc:290] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 01:57:22.016839 10704 raft_consensus.cc:2804] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 2 FOLLOWER]: Leader election won for term 2
I20260928 01:57:22.017051 10704 raft_consensus.cc:697] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 2 LEADER]: Becoming Leader. State: Replica: d3f0259187c841a98efbc8ed0eb8021a, State: Running, Role: LEADER
I20260928 01:57:22.017274 10697 heartbeater.cc:499] Master 127.9.83.62:34493 was elected leader, sending a full tablet report...
I20260928 01:57:22.017326 10704 consensus_queue.cc:237] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:22.017583 10589 ts_tablet_manager.cc:1434] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20260928 01:57:22.019454 10450 catalog_manager.cc:5838] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "d3f0259187c841a98efbc8ed0eb8021a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } health_report { overall_health: HEALTHY } } }
I20260928 01:57:22.091138 10698 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling UndoDeltaBlockGCOp(f7281e2e7d0b4d60a7b5242c208c1d34): 4615 bytes on disk
I20260928 01:57:22.092196 10587 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: UndoDeltaBlockGCOp(f7281e2e7d0b4d60a7b5242c208c1d34) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20260928 01:57:22.621223 10709 ts_tablet_manager.cc:933] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Initiating tablet copy from peer 00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831)
I20260928 01:57:22.621711 10709 tablet_copy_client.cc:323] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Beginning tablet copy session from remote peer at address 127.9.83.2:44831
I20260928 01:57:22.625319 10265 tablet_copy_service.cc:140] P 00af439d64e84c9db541f5414ac1d2e4: Received BeginTabletCopySession request for tablet 6bcce401df724b83973071baa1e5828b from peer d3f0259187c841a98efbc8ed0eb8021a ({username='slave'} at 127.9.83.1:34231)
I20260928 01:57:22.625450 10265 tablet_copy_service.cc:161] P 00af439d64e84c9db541f5414ac1d2e4: Beginning new tablet copy session on tablet 6bcce401df724b83973071baa1e5828b from peer d3f0259187c841a98efbc8ed0eb8021a at {username='slave'} at 127.9.83.1:34231: session id = d3f0259187c841a98efbc8ed0eb8021a-6bcce401df724b83973071baa1e5828b
I20260928 01:57:22.626823 10265 tablet_copy_source_session.cc:215] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4: Tablet Copy: opened 5 blocks and 1 log segments
I20260928 01:57:22.628504 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bcce401df724b83973071baa1e5828b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:22.631112 10709 tablet_copy_client.cc:806] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Starting download of 5 data blocks...
I20260928 01:57:22.636461 10709 tablet_copy_client.cc:670] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Starting download of 1 WAL segments...
W20260928 01:57:22.637609 10709 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260928 01:57:22.637656 10709 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260928 01:57:22.637679 10709 fault_injection.cc:59] injecting fault for kudu (pid 10555): FLAGS_fault_crash_after_tc_files_fetched (process will exit)
I20260928 01:57:22.672865 9548 delete_table-itest.cc:578] Test progress: crashed on first attempt to copy
I20260928 01:57:22.673998 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.1:41183
--local_ip_for_outbound_sockets=127.9.83.1
--tserver_master_addrs=127.9.83.62:34493
--webserver_port=46379
--webserver_interface=127.9.83.1
--builtin_ntp_servers=127.9.83.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=0
--maintenance_manager_polling_interval_ms=100
--fault_crash_after_tc_files_fetched=1.0 with env {}
W20260928 01:57:22.776625 10713 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:22.776847 10713 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:22.776897 10713 flags.cc:432] Enabled unsafe flag: --fault_crash_after_tc_files_fetched=1
W20260928 01:57:22.776929 10713 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:22.780553 10713 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:22.780647 10713 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260928 01:57:22.780725 10713 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.1
I20260928 01:57:22.784999 10713 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.1:41183
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.83.1
--webserver_port=46379
--flush_threshold_mb=0
--tserver_master_addrs=127.9.83.62:34493
--fault_crash_after_tc_files_fetched=1
--never_fsync=true
--heap_profile_path=/tmp/kudu.10713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:22.786072 10713 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:22.787206 10713 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:22.793563 10721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:22.793612 10719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:22.793604 10718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:22.793692 10713 server_base.cc:1061] running on GCE node
I20260928 01:57:22.794214 10713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:22.794725 10713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:22.795902 10713 hybrid_clock.cc:648] HybridClock initialized: now 1790560642795880 us; error 30 us; skew 500 ppm
I20260928 01:57:22.797597 10713 webserver.cc:533] Webserver started at http://127.9.83.1:46379/ using document root <none> and password file <none>
I20260928 01:57:22.798130 10713 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:22.798188 10713 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:22.801524 10713 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 01:57:22.806566 10732 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:22.807574 10713 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260928 01:57:22.807687 10713 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
uuid: "d3f0259187c841a98efbc8ed0eb8021a"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:22.808107 10713 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/data
Total live blocks: 5
Total live bytes: 22806
Total live bytes (after alignment): 36864
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:22.842128 10713 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:22.842728 10713 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:22.842916 10713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:22.843437 10713 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:22.844581 10739 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
W20260928 01:57:22.849022 10739 ts_tablet_manager.cc:1863] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_COPYING
I20260928 01:57:22.850003 10739 ts_tablet_manager.cc:1916] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260928 01:57:22.853933 10739 ts_tablet_manager.cc:1929] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.0
I20260928 01:57:22.854046 10739 log.cc:1199] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Deleting WAL directory at /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal/wals/6bcce401df724b83973071baa1e5828b
I20260928 01:57:22.855036 10713 ts_tablet_manager.cc:585] Loaded tablet metadata (2 total tablets, 1 live tablets)
I20260928 01:57:22.855108 10713 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.011s user 0.000s sys 0.000s
I20260928 01:57:22.855165 10713 ts_tablet_manager.cc:600] Registering tablets (0/2 complete)
I20260928 01:57:22.855810 10713 ts_tablet_manager.cc:616] Registered 1 tablets
I20260928 01:57:22.855849 10713 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 01:57:22.856086 10739 tablet_bootstrap.cc:492] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap starting.
I20260928 01:57:22.865504 10713 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.1:41183
I20260928 01:57:22.865522 10846 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.1:41183 every 8 connection(s)
I20260928 01:57:22.866663 10713 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
I20260928 01:57:22.870255 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 10713
I20260928 01:57:22.874130 10739 log.cc:826] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:22.876166 10847 heartbeater.cc:344] Connected to a master server at 127.9.83.62:34493
I20260928 01:57:22.876407 10847 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:22.876986 10847 heartbeater.cc:507] Master 127.9.83.62:34493 requested a full tablet report, sending...
I20260928 01:57:22.878923 10450 ts_manager.cc:194] Re-registered known tserver with Master: d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183)
I20260928 01:57:22.879812 10450 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.1:37731
I20260928 01:57:22.884269 10739 tablet_bootstrap.cc:492] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=12} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 01:57:22.884584 10739 tablet_bootstrap.cc:492] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap complete.
I20260928 01:57:22.885404 10739 ts_tablet_manager.cc:1403] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.004s
I20260928 01:57:22.887363 10739 raft_consensus.cc:359] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:22.887669 10739 raft_consensus.cc:740] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f0259187c841a98efbc8ed0eb8021a, State: Initialized, Role: FOLLOWER
I20260928 01:57:22.888067 10739 consensus_queue.cc:260] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:22.888166 10739 raft_consensus.cc:399] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 01:57:22.888226 10739 raft_consensus.cc:493] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 01:57:22.888302 10739 raft_consensus.cc:3060] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 2 FOLLOWER]: Advancing to term 3
I20260928 01:57:22.889084 10739 raft_consensus.cc:515] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:22.889374 10739 leader_election.cc:304] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3f0259187c841a98efbc8ed0eb8021a; no voters:
I20260928 01:57:22.889652 10739 leader_election.cc:290] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 01:57:22.889832 10852 raft_consensus.cc:2804] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 3 FOLLOWER]: Leader election won for term 3
I20260928 01:57:22.890058 10852 raft_consensus.cc:697] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 3 LEADER]: Becoming Leader. State: Replica: d3f0259187c841a98efbc8ed0eb8021a, State: Running, Role: LEADER
I20260928 01:57:22.890375 10852 consensus_queue.cc:237] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:22.890700 10739 ts_tablet_manager.cc:1434] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260928 01:57:22.891186 10847 heartbeater.cc:499] Master 127.9.83.62:34493 was elected leader, sending a full tablet report...
I20260928 01:57:22.892586 10450 catalog_manager.cc:5838] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "d3f0259187c841a98efbc8ed0eb8021a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } health_report { overall_health: HEALTHY } } }
I20260928 01:57:22.968291 10848 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling UndoDeltaBlockGCOp(f7281e2e7d0b4d60a7b5242c208c1d34): 4615 bytes on disk
I20260928 01:57:22.969420 10737 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: UndoDeltaBlockGCOp(f7281e2e7d0b4d60a7b5242c208c1d34) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20260928 01:57:23.564287 10864 tablet_replica.cc:333] stopping tablet replica
I20260928 01:57:23.564441 10864 raft_consensus.cc:2243] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 01:57:23.564517 10864 raft_consensus.cc:2272] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 01:57:23.564649 10864 ts_tablet_manager.cc:933] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Initiating tablet copy from peer 00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831)
I20260928 01:57:23.565181 10864 tablet_copy_client.cc:323] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Beginning tablet copy session from remote peer at address 127.9.83.2:44831
I20260928 01:57:23.568813 10265 tablet_copy_service.cc:140] P 00af439d64e84c9db541f5414ac1d2e4: Received BeginTabletCopySession request for tablet 6bcce401df724b83973071baa1e5828b from peer d3f0259187c841a98efbc8ed0eb8021a ({username='slave'} at 127.9.83.1:55781)
I20260928 01:57:23.568919 10265 tablet_copy_service.cc:171] P 00af439d64e84c9db541f5414ac1d2e4: Re-sending initialization info for existing tablet copy session on tablet 6bcce401df724b83973071baa1e5828b from peer d3f0259187c841a98efbc8ed0eb8021a at {username='slave'} at 127.9.83.1:55781: session_id = d3f0259187c841a98efbc8ed0eb8021a-6bcce401df724b83973071baa1e5828b
I20260928 01:57:23.570231 10864 ts_tablet_manager.cc:1916] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260928 01:57:23.572358 10864 ts_tablet_manager.cc:1929] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.0
I20260928 01:57:23.572731 10864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bcce401df724b83973071baa1e5828b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:23.574085 10864 tablet_copy_client.cc:806] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Starting download of 5 data blocks...
I20260928 01:57:23.579331 10864 tablet_copy_client.cc:670] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Starting download of 1 WAL segments...
W20260928 01:57:23.580287 10864 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260928 01:57:23.580325 10864 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260928 01:57:23.580348 10864 fault_injection.cc:59] injecting fault for kudu (pid 10713): FLAGS_fault_crash_after_tc_files_fetched (process will exit)
I20260928 01:57:23.652091 9548 delete_table-itest.cc:588] Test progress: crashed on second attempt to copy
I20260928 01:57:23.652516 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.1:41183
--local_ip_for_outbound_sockets=127.9.83.1
--tserver_master_addrs=127.9.83.62:34493
--webserver_port=46379
--webserver_interface=127.9.83.1
--builtin_ntp_servers=127.9.83.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=0
--maintenance_manager_polling_interval_ms=100 with env {}
W20260928 01:57:23.753661 10868 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:23.753909 10868 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:23.753978 10868 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:23.757562 10868 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:23.757666 10868 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260928 01:57:23.757735 10868 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.1
I20260928 01:57:23.761920 10868 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.1:41183
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.83.1
--webserver_port=46379
--flush_threshold_mb=0
--tserver_master_addrs=127.9.83.62:34493
--never_fsync=true
--heap_profile_path=/tmp/kudu.10868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:23.763108 10868 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:23.764254 10868 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:23.770828 10874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:23.770833 10873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:23.770929 10868 server_base.cc:1061] running on GCE node
W20260928 01:57:23.770828 10876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:23.771420 10868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:23.771970 10868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:23.773134 10868 hybrid_clock.cc:648] HybridClock initialized: now 1790560643773125 us; error 28 us; skew 500 ppm
I20260928 01:57:23.774907 10868 webserver.cc:533] Webserver started at http://127.9.83.1:46379/ using document root <none> and password file <none>
I20260928 01:57:23.775446 10868 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:23.775502 10868 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:23.778744 10868 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 01:57:23.783717 10887 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:23.784636 10868 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260928 01:57:23.784740 10868 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
uuid: "d3f0259187c841a98efbc8ed0eb8021a"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:23.785137 10868 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/data
Total live blocks: 5
Total live bytes: 22806
Total live bytes (after alignment): 36864
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:23.793762 10868 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:23.794405 10868 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:23.794569 10868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:23.795159 10868 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:23.796322 10894 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
W20260928 01:57:23.801987 10894 ts_tablet_manager.cc:1863] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_COPYING
I20260928 01:57:23.803582 10894 ts_tablet_manager.cc:1916] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260928 01:57:23.808197 10894 ts_tablet_manager.cc:1929] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.0
I20260928 01:57:23.808305 10894 log.cc:1199] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Deleting WAL directory at /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal/wals/6bcce401df724b83973071baa1e5828b
I20260928 01:57:23.809274 10868 ts_tablet_manager.cc:585] Loaded tablet metadata (2 total tablets, 1 live tablets)
I20260928 01:57:23.809362 10868 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.013s user 0.000s sys 0.000s
I20260928 01:57:23.809435 10868 ts_tablet_manager.cc:600] Registering tablets (0/2 complete)
I20260928 01:57:23.810194 10868 ts_tablet_manager.cc:616] Registered 1 tablets
I20260928 01:57:23.810237 10868 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 01:57:23.810516 10894 tablet_bootstrap.cc:492] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap starting.
I20260928 01:57:23.820462 10868 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.1:41183
I20260928 01:57:23.820529 11001 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.1:41183 every 8 connection(s)
I20260928 01:57:23.821470 10868 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
I20260928 01:57:23.828625 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 10868
I20260928 01:57:23.828686 10894 log.cc:826] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:23.839720 10894 tablet_bootstrap.cc:492] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=12} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 01:57:23.839850 11002 heartbeater.cc:344] Connected to a master server at 127.9.83.62:34493
I20260928 01:57:23.840118 11002 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:23.840124 10894 tablet_bootstrap.cc:492] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap complete.
I20260928 01:57:23.840651 11002 heartbeater.cc:507] Master 127.9.83.62:34493 requested a full tablet report, sending...
I20260928 01:57:23.841274 10894 ts_tablet_manager.cc:1403] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.004s
I20260928 01:57:23.842442 10450 ts_manager.cc:194] Re-registered known tserver with Master: d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183)
I20260928 01:57:23.843379 10450 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.1:41523
I20260928 01:57:23.843940 10894 raft_consensus.cc:359] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:23.844363 10894 raft_consensus.cc:740] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f0259187c841a98efbc8ed0eb8021a, State: Initialized, Role: FOLLOWER
I20260928 01:57:23.844836 10894 consensus_queue.cc:260] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:23.844962 10894 raft_consensus.cc:399] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 01:57:23.845018 10894 raft_consensus.cc:493] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 01:57:23.845111 10894 raft_consensus.cc:3060] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 3 FOLLOWER]: Advancing to term 4
I20260928 01:57:23.846246 10894 raft_consensus.cc:515] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:23.846613 10894 leader_election.cc:304] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3f0259187c841a98efbc8ed0eb8021a; no voters:
I20260928 01:57:23.846985 10894 leader_election.cc:290] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [CANDIDATE]: Term 4 election: Requested vote from peers
I20260928 01:57:23.847113 11014 raft_consensus.cc:2804] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 4 FOLLOWER]: Leader election won for term 4
I20260928 01:57:23.847324 11014 raft_consensus.cc:697] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 4 LEADER]: Becoming Leader. State: Replica: d3f0259187c841a98efbc8ed0eb8021a, State: Running, Role: LEADER
I20260928 01:57:23.847626 11014 consensus_queue.cc:237] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 15, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:23.847735 11002 heartbeater.cc:499] Master 127.9.83.62:34493 was elected leader, sending a full tablet report...
I20260928 01:57:23.847843 10894 ts_tablet_manager.cc:1434] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 01:57:23.849622 10450 catalog_manager.cc:5838] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a reported cstate change: term changed from 3 to 4. New cstate: current_term: 4 leader_uuid: "d3f0259187c841a98efbc8ed0eb8021a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } health_report { overall_health: HEALTHY } } }
I20260928 01:57:23.892176 10225 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260928 01:57:23.892654 10356 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260928 01:57:23.893569 10936 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
a9dce204f3a642aa84955400c67e32fd | 127.9.83.62:34493 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_add_server_when_underreplicated | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
00af439d64e84c9db541f5414ac1d2e4 | 127.9.83.2:44831 | HEALTHY | <none> | 1 | 0
71a7ac59363f4dd1bc6b7e38506bc135 | 127.9.83.3:41171 | HEALTHY | <none> | 0 | 0
d3f0259187c841a98efbc8ed0eb8021a | 127.9.83.1:41183 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
flush_threshold_mb | 0 | runtime,experimental | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.9.83.1 | experimental | 127.9.83.1:41183
local_ip_for_outbound_sockets | 127.9.83.2 | experimental | 127.9.83.2:44831
local_ip_for_outbound_sockets | 127.9.83.3 | experimental | 127.9.83.3:41171
maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb | hidden | 127.9.83.1:41183
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb | hidden | 127.9.83.2:44831
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb | hidden | 127.9.83.3:41171
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 6bcce401df724b83973071baa1e5828b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
71a7ac59363f4dd1bc6b7e38506bc135 (127.9.83.3:41171): RUNNING
00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831): RUNNING [LEADER]
d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183): not running
State: INITIALIZED
Data state: TABLET_DATA_TOMBSTONED
Last status: Tombstoned
All reported replicas are:
A = 71a7ac59363f4dd1bc6b7e38506bc135
B = 00af439d64e84c9db541f5414ac1d2e4
C = d3f0259187c841a98efbc8ed0eb8021a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 22 | Yes
B | A B* C | 1 | 22 | Yes
C | A B C | 1 | 22 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
other-table | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
test-workload | 2 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 2
Tablets | 2
Replicas | 4
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
I20260928 01:57:23.920771 9548 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260928 01:57:23.922941 11003 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling UndoDeltaBlockGCOp(f7281e2e7d0b4d60a7b5242c208c1d34): 4615 bytes on disk
I20260928 01:57:23.923812 10892 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: UndoDeltaBlockGCOp(f7281e2e7d0b4d60a7b5242c208c1d34) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20260928 01:57:24.116454 10225 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260928 01:57:24.117466 10356 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260928 01:57:24.117686 10936 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
a9dce204f3a642aa84955400c67e32fd | 127.9.83.62:34493 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_add_server_when_underreplicated | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
00af439d64e84c9db541f5414ac1d2e4 | 127.9.83.2:44831 | HEALTHY | <none> | 1 | 0
71a7ac59363f4dd1bc6b7e38506bc135 | 127.9.83.3:41171 | HEALTHY | <none> | 0 | 0
d3f0259187c841a98efbc8ed0eb8021a | 127.9.83.1:41183 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
flush_threshold_mb | 0 | runtime,experimental | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.9.83.1 | experimental | 127.9.83.1:41183
local_ip_for_outbound_sockets | 127.9.83.2 | experimental | 127.9.83.2:44831
local_ip_for_outbound_sockets | 127.9.83.3 | experimental | 127.9.83.3:41171
maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb | hidden | 127.9.83.1:41183
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb | hidden | 127.9.83.2:44831
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb | hidden | 127.9.83.3:41171
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 6bcce401df724b83973071baa1e5828b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
71a7ac59363f4dd1bc6b7e38506bc135 (127.9.83.3:41171): RUNNING
00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831): RUNNING [LEADER]
d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183): not running
State: INITIALIZED
Data state: TABLET_DATA_TOMBSTONED
Last status: Tombstoned
All reported replicas are:
A = 71a7ac59363f4dd1bc6b7e38506bc135
B = 00af439d64e84c9db541f5414ac1d2e4
C = d3f0259187c841a98efbc8ed0eb8021a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 22 | Yes
B | A B* C | 1 | 22 | Yes
C | A B C | 1 | 22 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
other-table | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
test-workload | 2 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 2
Tablets | 2
Replicas | 4
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
I20260928 01:57:24.139370 9548 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260928 01:57:24.413244 10225 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260928 01:57:24.413801 10356 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260928 01:57:24.413805 10936 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
a9dce204f3a642aa84955400c67e32fd | 127.9.83.62:34493 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_add_server_when_underreplicated | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
00af439d64e84c9db541f5414ac1d2e4 | 127.9.83.2:44831 | HEALTHY | <none> | 1 | 0
71a7ac59363f4dd1bc6b7e38506bc135 | 127.9.83.3:41171 | HEALTHY | <none> | 0 | 0
d3f0259187c841a98efbc8ed0eb8021a | 127.9.83.1:41183 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
flush_threshold_mb | 0 | runtime,experimental | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.9.83.1 | experimental | 127.9.83.1:41183
local_ip_for_outbound_sockets | 127.9.83.2 | experimental | 127.9.83.2:44831
local_ip_for_outbound_sockets | 127.9.83.3 | experimental | 127.9.83.3:41171
maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb | hidden | 127.9.83.1:41183
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb | hidden | 127.9.83.2:44831
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb | hidden | 127.9.83.3:41171
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 6bcce401df724b83973071baa1e5828b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
71a7ac59363f4dd1bc6b7e38506bc135 (127.9.83.3:41171): RUNNING
00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831): RUNNING [LEADER]
d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183): not running
State: INITIALIZED
Data state: TABLET_DATA_TOMBSTONED
Last status: Tombstoned
All reported replicas are:
A = 71a7ac59363f4dd1bc6b7e38506bc135
B = 00af439d64e84c9db541f5414ac1d2e4
C = d3f0259187c841a98efbc8ed0eb8021a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 22 | Yes
B | A B* C | 1 | 22 | Yes
C | A B C | 1 | 22 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
other-table | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
test-workload | 2 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 2
Tablets | 2
Replicas | 4
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
I20260928 01:57:24.436550 9548 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260928 01:57:24.516258 11091 tablet_replica.cc:333] stopping tablet replica
I20260928 01:57:24.516443 11091 raft_consensus.cc:2243] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 01:57:24.516520 11091 raft_consensus.cc:2272] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 01:57:24.516652 11091 ts_tablet_manager.cc:933] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Initiating tablet copy from peer 00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831)
I20260928 01:57:24.517207 11091 tablet_copy_client.cc:323] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Beginning tablet copy session from remote peer at address 127.9.83.2:44831
I20260928 01:57:24.520515 10265 tablet_copy_service.cc:140] P 00af439d64e84c9db541f5414ac1d2e4: Received BeginTabletCopySession request for tablet 6bcce401df724b83973071baa1e5828b from peer d3f0259187c841a98efbc8ed0eb8021a ({username='slave'} at 127.9.83.1:60207)
I20260928 01:57:24.520622 10265 tablet_copy_service.cc:171] P 00af439d64e84c9db541f5414ac1d2e4: Re-sending initialization info for existing tablet copy session on tablet 6bcce401df724b83973071baa1e5828b from peer d3f0259187c841a98efbc8ed0eb8021a at {username='slave'} at 127.9.83.1:60207: session_id = d3f0259187c841a98efbc8ed0eb8021a-6bcce401df724b83973071baa1e5828b
I20260928 01:57:24.521801 11091 ts_tablet_manager.cc:1916] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260928 01:57:24.523855 11091 ts_tablet_manager.cc:1929] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.0
I20260928 01:57:24.524170 11091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bcce401df724b83973071baa1e5828b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:24.525218 11091 tablet_copy_client.cc:806] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Starting download of 5 data blocks...
I20260928 01:57:24.530817 11091 tablet_copy_client.cc:670] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Starting download of 1 WAL segments...
I20260928 01:57:24.532455 11091 tablet_copy_client.cc:538] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 01:57:24.533743 11091 tablet_bootstrap.cc:492] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap starting.
I20260928 01:57:24.546525 11091 tablet_bootstrap.cc:492] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap replayed 1/1 log segments. Stats: ops{read=22 overwritten=0 applied=22 ignored=20} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 01:57:24.546838 11091 tablet_bootstrap.cc:492] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap complete.
I20260928 01:57:24.546995 11091 ts_tablet_manager.cc:1403] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.006s
I20260928 01:57:24.547515 11091 raft_consensus.cc:359] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } } peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } attrs { } }
I20260928 01:57:24.547617 11091 raft_consensus.cc:740] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f0259187c841a98efbc8ed0eb8021a, State: Initialized, Role: FOLLOWER
I20260928 01:57:24.547762 11091 consensus_queue.cc:260] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 22, Last appended: 1.22, Last appended by leader: 22, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } } peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } attrs { } }
I20260928 01:57:24.548098 11091 ts_tablet_manager.cc:1434] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260928 01:57:24.548802 10265 tablet_copy_service.cc:342] P 00af439d64e84c9db541f5414ac1d2e4: Request end of tablet copy session d3f0259187c841a98efbc8ed0eb8021a-6bcce401df724b83973071baa1e5828b received from {username='slave'} at 127.9.83.1:60207
I20260928 01:57:24.548903 10265 tablet_copy_service.cc:434] P 00af439d64e84c9db541f5414ac1d2e4: ending tablet copy session d3f0259187c841a98efbc8ed0eb8021a-6bcce401df724b83973071baa1e5828b on tablet 6bcce401df724b83973071baa1e5828b with peer d3f0259187c841a98efbc8ed0eb8021a
I20260928 01:57:24.627003 11003 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 7929 bytes on disk
I20260928 01:57:24.627491 10892 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20260928 01:57:24.819414 10225 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260928 01:57:24.819571 10936 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260928 01:57:24.819893 10356 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
a9dce204f3a642aa84955400c67e32fd | 127.9.83.62:34493 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_add_server_when_underreplicated | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
00af439d64e84c9db541f5414ac1d2e4 | 127.9.83.2:44831 | HEALTHY | <none> | 1 | 0
71a7ac59363f4dd1bc6b7e38506bc135 | 127.9.83.3:41171 | HEALTHY | <none> | 0 | 0
d3f0259187c841a98efbc8ed0eb8021a | 127.9.83.1:41183 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
flush_threshold_mb | 0 | runtime,experimental | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.9.83.1 | experimental | 127.9.83.1:41183
local_ip_for_outbound_sockets | 127.9.83.2 | experimental | 127.9.83.2:44831
local_ip_for_outbound_sockets | 127.9.83.3 | experimental | 127.9.83.3:41171
maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb | hidden | 127.9.83.1:41183
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb | hidden | 127.9.83.2:44831
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb | hidden | 127.9.83.3:41171
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 6bcce401df724b83973071baa1e5828b of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's
71a7ac59363f4dd1bc6b7e38506bc135 (127.9.83.3:41171): RUNNING
00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831): RUNNING [LEADER]
d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183): RUNNING
All reported replicas are:
A = 71a7ac59363f4dd1bc6b7e38506bc135
B = 00af439d64e84c9db541f5414ac1d2e4
C = d3f0259187c841a98efbc8ed0eb8021a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 22 | Yes
B | A B* C | 1 | 22 | Yes
C | A B C | 1 | 22 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+--------------------+---------------+---------+------------+------------------+-------------
other-table | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
test-workload | 2 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 2
Tablets | 2
Replicas | 4
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
I20260928 01:57:24.848352 9548 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260928 01:57:24.947067 10956 raft_consensus.cc:1217] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.21->[1.22-1.22] Dedup: 1.22->[]
I20260928 01:57:25.400429 10356 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260928 01:57:25.404114 10225 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260928 01:57:25.405234 10936 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
a9dce204f3a642aa84955400c67e32fd | 127.9.83.62:34493 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_add_server_when_underreplicated | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
00af439d64e84c9db541f5414ac1d2e4 | 127.9.83.2:44831 | HEALTHY | <none> | 1 | 0
71a7ac59363f4dd1bc6b7e38506bc135 | 127.9.83.3:41171 | HEALTHY | <none> | 0 | 0
d3f0259187c841a98efbc8ed0eb8021a | 127.9.83.1:41183 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
flush_threshold_mb | 0 | runtime,experimental | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.9.83.1 | experimental | 127.9.83.1:41183
local_ip_for_outbound_sockets | 127.9.83.2 | experimental | 127.9.83.2:44831
local_ip_for_outbound_sockets | 127.9.83.3 | experimental | 127.9.83.3:41171
maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb | hidden | 127.9.83.1:41183
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb | hidden | 127.9.83.2:44831
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb | hidden | 127.9.83.3:41171
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
other-table | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
test-workload | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 2
Tablets | 2
Replicas | 4
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260928 01:57:25.427726 9548 log_verifier.cc:126] Checking tablet 6bcce401df724b83973071baa1e5828b
I20260928 01:57:25.457959 9548 log_verifier.cc:177] Verified matching terms for 22 ops in tablet 6bcce401df724b83973071baa1e5828b
I20260928 01:57:25.458087 9548 log_verifier.cc:126] Checking tablet f7281e2e7d0b4d60a7b5242c208c1d34
I20260928 01:57:25.467229 9548 log_verifier.cc:177] Verified matching terms for 16 ops in tablet f7281e2e7d0b4d60a7b5242c208c1d34
I20260928 01:57:25.588037 10245 consensus_queue.cc:237] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 1.22, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } }
I20260928 01:57:25.589416 10376 raft_consensus.cc:1275] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135 [term 1 FOLLOWER]: Refusing update from remote peer 00af439d64e84c9db541f5414ac1d2e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 22. Preceding OpId from leader: term: 1 index: 23. (index mismatch)
I20260928 01:57:25.589901 11152 consensus_queue.cc:1048] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260928 01:57:25.591835 11163 raft_consensus.cc:2955] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [term 1 LEADER]: Committing config change with OpId 1.23: config changed from index 22 to 23, VOTER d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } } }
I20260928 01:57:25.592239 10376 raft_consensus.cc:2955] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135 [term 1 FOLLOWER]: Committing config change with OpId 1.23: config changed from index 22 to 23, VOTER d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } } }
I20260928 01:57:25.595129 10454 catalog_manager.cc:5838] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 reported cstate change: config changed from index 22 to 23, VOTER d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1) evicted. New cstate: current_term: 1 leader_uuid: "00af439d64e84c9db541f5414ac1d2e4" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260928 01:57:25.603011 10936 tablet_service.cc:1558] Processing DeleteTablet for tablet 6bcce401df724b83973071baa1e5828b with delete_type TABLET_DATA_TOMBSTONED (TS d3f0259187c841a98efbc8ed0eb8021a not found in new config with opid_index 23) from {username='slave'} at 127.0.0.1:33362
I20260928 01:57:25.603511 11171 tablet_replica.cc:333] stopping tablet replica
I20260928 01:57:25.603775 11171 raft_consensus.cc:2243] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 01:57:25.604015 11171 raft_consensus.cc:2272] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 01:57:25.606004 11171 ts_tablet_manager.cc:1916] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260928 01:57:25.609647 11171 ts_tablet_manager.cc:1929] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.22
I20260928 01:57:25.609731 11171 log.cc:1199] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Deleting WAL directory at /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal/wals/6bcce401df724b83973071baa1e5828b
I20260928 01:57:25.610373 10443 catalog_manager.cc:5063] TS d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183): tablet 6bcce401df724b83973071baa1e5828b (table test-workload [id=91ee3d2cfd8540cf8d3fee1acc68ce1a]) successfully deleted
W20260928 01:57:25.622844 10995 generic_service.cc:139] {username='slave'} at 127.0.0.1:33370 forcing change of non-runtime-safe flag fault_crash_after_tc_files_fetched
I20260928 01:57:25.622988 10995 generic_service.cc:175] {username='slave'} at 127.0.0.1:33370 changed flags via RPC: fault_crash_after_tc_files_fetched from '0' to '1.0'
I20260928 01:57:25.624341 10245 consensus_queue.cc:237] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } } peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } attrs { } }
I20260928 01:57:25.625973 10376 raft_consensus.cc:1275] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135 [term 1 FOLLOWER]: Refusing update from remote peer 00af439d64e84c9db541f5414ac1d2e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
W20260928 01:57:25.626173 10181 consensus_peers.cc:597] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 -> Peer d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183): Couldn't send request to peer d3f0259187c841a98efbc8ed0eb8021a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260928 01:57:25.626468 11165 consensus_queue.cc:1048] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260928 01:57:25.628132 11163 raft_consensus.cc:2955] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index 23 to 24, VOTER d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } } peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } attrs { } } }
I20260928 01:57:25.628525 10376 raft_consensus.cc:2955] T 6bcce401df724b83973071baa1e5828b P 71a7ac59363f4dd1bc6b7e38506bc135 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index 23 to 24, VOTER d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } } peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } attrs { } } }
I20260928 01:57:25.630904 10454 catalog_manager.cc:5838] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4 reported cstate change: config changed from index 23 to 24, VOTER d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1) added. New cstate: current_term: 1 leader_uuid: "00af439d64e84c9db541f5414ac1d2e4" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } attrs { } health_report { overall_health: UNKNOWN } } }
I20260928 01:57:26.119903 11174 ts_tablet_manager.cc:933] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Initiating tablet copy from peer 00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831)
I20260928 01:57:26.120369 11174 tablet_copy_client.cc:323] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Beginning tablet copy session from remote peer at address 127.9.83.2:44831
I20260928 01:57:26.121017 10265 tablet_copy_service.cc:140] P 00af439d64e84c9db541f5414ac1d2e4: Received BeginTabletCopySession request for tablet 6bcce401df724b83973071baa1e5828b from peer d3f0259187c841a98efbc8ed0eb8021a ({username='slave'} at 127.9.83.1:60207)
I20260928 01:57:26.121140 10265 tablet_copy_service.cc:161] P 00af439d64e84c9db541f5414ac1d2e4: Beginning new tablet copy session on tablet 6bcce401df724b83973071baa1e5828b from peer d3f0259187c841a98efbc8ed0eb8021a at {username='slave'} at 127.9.83.1:60207: session id = d3f0259187c841a98efbc8ed0eb8021a-6bcce401df724b83973071baa1e5828b
I20260928 01:57:26.122076 10265 tablet_copy_source_session.cc:215] T 6bcce401df724b83973071baa1e5828b P 00af439d64e84c9db541f5414ac1d2e4: Tablet Copy: opened 5 blocks and 1 log segments
I20260928 01:57:26.122885 11174 ts_tablet_manager.cc:1916] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260928 01:57:26.124598 11174 ts_tablet_manager.cc:1929] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.22
I20260928 01:57:26.124744 11174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bcce401df724b83973071baa1e5828b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:26.125672 11174 tablet_copy_client.cc:806] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Starting download of 5 data blocks...
I20260928 01:57:26.129925 11174 tablet_copy_client.cc:670] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Starting download of 1 WAL segments...
W20260928 01:57:26.131161 11174 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260928 01:57:26.131210 11174 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260928 01:57:26.131237 11174 fault_injection.cc:59] injecting fault for kudu (pid 10868): FLAGS_fault_crash_after_tc_files_fetched (process will exit)
I20260928 01:57:26.181330 9548 delete_table-itest.cc:613] Test progress: crashed on attempt to copy over tombstoned
I20260928 01:57:26.181674 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.1:41183
--local_ip_for_outbound_sockets=127.9.83.1
--tserver_master_addrs=127.9.83.62:34493
--webserver_port=46379
--webserver_interface=127.9.83.1
--builtin_ntp_servers=127.9.83.20:44475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=0
--maintenance_manager_polling_interval_ms=100 with env {}
W20260928 01:57:26.284655 11176 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:26.284888 11176 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:26.284935 11176 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:26.288424 11176 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:26.288516 11176 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260928 01:57:26.288579 11176 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.1
I20260928 01:57:26.292752 11176 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:44475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.1:41183
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.83.1
--webserver_port=46379
--flush_threshold_mb=0
--tserver_master_addrs=127.9.83.62:34493
--never_fsync=true
--heap_profile_path=/tmp/kudu.11176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:26.293922 11176 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:26.295070 11176 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:26.301599 11181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:26.301679 11184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:26.301718 11176 server_base.cc:1061] running on GCE node
W20260928 01:57:26.301612 11182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:26.302184 11176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:26.302729 11176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:26.303903 11176 hybrid_clock.cc:648] HybridClock initialized: now 1790560646303893 us; error 26 us; skew 500 ppm
I20260928 01:57:26.305711 11176 webserver.cc:533] Webserver started at http://127.9.83.1:46379/ using document root <none> and password file <none>
I20260928 01:57:26.306265 11176 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:26.306316 11176 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:26.309747 11176 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 01:57:26.314597 11195 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:26.315502 11176 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260928 01:57:26.315608 11176 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
uuid: "d3f0259187c841a98efbc8ed0eb8021a"
format_stamp: "Formatted at 2026-09-28 01:57:18 on dist-test-slave-lkt6"
I20260928 01:57:26.315976 11176 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/data
Total live blocks: 5
Total live bytes: 22806
Total live bytes (after alignment): 36864
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:26.325783 11176 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:26.326406 11176 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:26.326635 11176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:26.327239 11176 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:26.328338 11202 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
W20260928 01:57:26.332861 11202 ts_tablet_manager.cc:1863] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_COPYING
I20260928 01:57:26.333874 11202 ts_tablet_manager.cc:1916] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260928 01:57:26.337414 11202 ts_tablet_manager.cc:1929] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.22
I20260928 01:57:26.337518 11202 log.cc:1199] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Deleting WAL directory at /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/wal/wals/6bcce401df724b83973071baa1e5828b
I20260928 01:57:26.338435 11176 ts_tablet_manager.cc:585] Loaded tablet metadata (2 total tablets, 1 live tablets)
I20260928 01:57:26.338500 11176 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260928 01:57:26.338565 11176 ts_tablet_manager.cc:600] Registering tablets (0/2 complete)
I20260928 01:57:26.339287 11176 ts_tablet_manager.cc:616] Registered 1 tablets
I20260928 01:57:26.339327 11176 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 01:57:26.339567 11202 tablet_bootstrap.cc:492] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap starting.
I20260928 01:57:26.348331 11176 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.1:41183
I20260928 01:57:26.348377 11309 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.1:41183 every 8 connection(s)
I20260928 01:57:26.349251 11176 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
I20260928 01:57:26.354534 11202 log.cc:826] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:26.357251 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 11176
I20260928 01:57:26.357411 11310 heartbeater.cc:344] Connected to a master server at 127.9.83.62:34493
I20260928 01:57:26.357638 11310 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:26.358165 11310 heartbeater.cc:507] Master 127.9.83.62:34493 requested a full tablet report, sending...
I20260928 01:57:26.360105 10450 ts_manager.cc:194] Re-registered known tserver with Master: d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183)
I20260928 01:57:26.361176 10450 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.1:46773
I20260928 01:57:26.365651 11202 tablet_bootstrap.cc:492] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap replayed 1/1 log segments. Stats: ops{read=16 overwritten=0 applied=16 ignored=12} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 01:57:26.365957 11202 tablet_bootstrap.cc:492] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap complete.
I20260928 01:57:26.366760 11202 ts_tablet_manager.cc:1403] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.005s
I20260928 01:57:26.368676 11202 raft_consensus.cc:359] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:26.368981 11202 raft_consensus.cc:740] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f0259187c841a98efbc8ed0eb8021a, State: Initialized, Role: FOLLOWER
I20260928 01:57:26.369419 11202 consensus_queue.cc:260] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:26.369532 11202 raft_consensus.cc:399] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 4 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 01:57:26.369596 11202 raft_consensus.cc:493] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 4 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 01:57:26.369685 11202 raft_consensus.cc:3060] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 4 FOLLOWER]: Advancing to term 5
I20260928 01:57:26.370492 11202 raft_consensus.cc:515] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:26.370833 11202 leader_election.cc:304] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3f0259187c841a98efbc8ed0eb8021a; no voters:
I20260928 01:57:26.371130 11202 leader_election.cc:290] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [CANDIDATE]: Term 5 election: Requested vote from peers
I20260928 01:57:26.371227 11321 raft_consensus.cc:2804] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 5 FOLLOWER]: Leader election won for term 5
I20260928 01:57:26.371433 11321 raft_consensus.cc:697] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [term 5 LEADER]: Becoming Leader. State: Replica: d3f0259187c841a98efbc8ed0eb8021a, State: Running, Role: LEADER
I20260928 01:57:26.371714 11310 heartbeater.cc:499] Master 127.9.83.62:34493 was elected leader, sending a full tablet report...
I20260928 01:57:26.371770 11321 consensus_queue.cc:237] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 4.16, Last appended by leader: 16, Current term: 5, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } }
I20260928 01:57:26.371978 11202 ts_tablet_manager.cc:1434] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 01:57:26.373891 10450 catalog_manager.cc:5838] T f7281e2e7d0b4d60a7b5242c208c1d34 P d3f0259187c841a98efbc8ed0eb8021a reported cstate change: term changed from 4 to 5. New cstate: current_term: 5 leader_uuid: "d3f0259187c841a98efbc8ed0eb8021a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } health_report { overall_health: HEALTHY } } }
I20260928 01:57:26.406919 10225 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260928 01:57:26.407029 10356 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260928 01:57:26.410512 11244 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
a9dce204f3a642aa84955400c67e32fd | 127.9.83.62:34493 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_add_server_when_underreplicated | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
00af439d64e84c9db541f5414ac1d2e4 | 127.9.83.2:44831 | HEALTHY | <none> | 1 | 0
71a7ac59363f4dd1bc6b7e38506bc135 | 127.9.83.3:41171 | HEALTHY | <none> | 0 | 0
d3f0259187c841a98efbc8ed0eb8021a | 127.9.83.1:41183 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
flush_threshold_mb | 0 | runtime,experimental | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.9.83.1 | experimental | 127.9.83.1:41183
local_ip_for_outbound_sockets | 127.9.83.2 | experimental | 127.9.83.2:44831
local_ip_for_outbound_sockets | 127.9.83.3 | experimental | 127.9.83.3:41171
maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb | hidden | 127.9.83.1:41183
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb | hidden | 127.9.83.2:44831
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb | hidden | 127.9.83.3:41171
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 6bcce401df724b83973071baa1e5828b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
71a7ac59363f4dd1bc6b7e38506bc135 (127.9.83.3:41171): RUNNING
00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831): RUNNING [LEADER]
d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183): not running
State: INITIALIZED
Data state: TABLET_DATA_TOMBSTONED
Last status: Tombstoned
All reported replicas are:
A = 71a7ac59363f4dd1bc6b7e38506bc135
B = 00af439d64e84c9db541f5414ac1d2e4
C = d3f0259187c841a98efbc8ed0eb8021a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 24 | Yes
B | A B* C | 1 | 24 | Yes
C | A B C | 1 | 24 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
other-table | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
test-workload | 2 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 2
Tablets | 2
Replicas | 4
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
I20260928 01:57:26.433895 9548 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260928 01:57:26.451045 11311 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling UndoDeltaBlockGCOp(f7281e2e7d0b4d60a7b5242c208c1d34): 4615 bytes on disk
I20260928 01:57:26.452070 11200 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: UndoDeltaBlockGCOp(f7281e2e7d0b4d60a7b5242c208c1d34) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20260928 01:57:26.631718 11244 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260928 01:57:26.631718 10356 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260928 01:57:26.631891 10225 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
a9dce204f3a642aa84955400c67e32fd | 127.9.83.62:34493 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_add_server_when_underreplicated | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
00af439d64e84c9db541f5414ac1d2e4 | 127.9.83.2:44831 | HEALTHY | <none> | 1 | 0
71a7ac59363f4dd1bc6b7e38506bc135 | 127.9.83.3:41171 | HEALTHY | <none> | 0 | 0
d3f0259187c841a98efbc8ed0eb8021a | 127.9.83.1:41183 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
flush_threshold_mb | 0 | runtime,experimental | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.9.83.1 | experimental | 127.9.83.1:41183
local_ip_for_outbound_sockets | 127.9.83.2 | experimental | 127.9.83.2:44831
local_ip_for_outbound_sockets | 127.9.83.3 | experimental | 127.9.83.3:41171
maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb | hidden | 127.9.83.1:41183
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb | hidden | 127.9.83.2:44831
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb | hidden | 127.9.83.3:41171
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 6bcce401df724b83973071baa1e5828b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
71a7ac59363f4dd1bc6b7e38506bc135 (127.9.83.3:41171): RUNNING
00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831): RUNNING [LEADER]
d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183): not running
State: INITIALIZED
Data state: TABLET_DATA_TOMBSTONED
Last status: Tombstoned
All reported replicas are:
A = 71a7ac59363f4dd1bc6b7e38506bc135
B = 00af439d64e84c9db541f5414ac1d2e4
C = d3f0259187c841a98efbc8ed0eb8021a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 24 | Yes
B | A B* C | 1 | 24 | Yes
C | A B C | 1 | 24 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
other-table | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
test-workload | 2 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 2
Tablets | 2
Replicas | 4
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
I20260928 01:57:26.660117 9548 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260928 01:57:26.931568 11398 tablet_replica.cc:333] stopping tablet replica
I20260928 01:57:26.931939 11398 raft_consensus.cc:2243] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 01:57:26.932073 11398 raft_consensus.cc:2272] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 01:57:26.932215 11398 ts_tablet_manager.cc:933] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Initiating tablet copy from peer 00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831)
I20260928 01:57:26.932775 11398 tablet_copy_client.cc:323] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Beginning tablet copy session from remote peer at address 127.9.83.2:44831
I20260928 01:57:26.933025 11244 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260928 01:57:26.933732 10356 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260928 01:57:26.933815 10225 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260928 01:57:26.939189 10265 tablet_copy_service.cc:140] P 00af439d64e84c9db541f5414ac1d2e4: Received BeginTabletCopySession request for tablet 6bcce401df724b83973071baa1e5828b from peer d3f0259187c841a98efbc8ed0eb8021a ({username='slave'} at 127.9.83.1:46715)
I20260928 01:57:26.939285 10265 tablet_copy_service.cc:171] P 00af439d64e84c9db541f5414ac1d2e4: Re-sending initialization info for existing tablet copy session on tablet 6bcce401df724b83973071baa1e5828b from peer d3f0259187c841a98efbc8ed0eb8021a at {username='slave'} at 127.9.83.1:46715: session_id = d3f0259187c841a98efbc8ed0eb8021a-6bcce401df724b83973071baa1e5828b
I20260928 01:57:26.943976 11398 ts_tablet_manager.cc:1916] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20260928 01:57:26.946022 11398 ts_tablet_manager.cc:1929] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.22
I20260928 01:57:26.946425 11398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bcce401df724b83973071baa1e5828b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:26.947813 11398 tablet_copy_client.cc:806] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Starting download of 5 data blocks...
I20260928 01:57:26.952823 11398 tablet_copy_client.cc:670] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Starting download of 1 WAL segments...
I20260928 01:57:26.954699 11398 tablet_copy_client.cc:538] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
a9dce204f3a642aa84955400c67e32fd | 127.9.83.62:34493 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_add_server_when_underreplicated | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
00af439d64e84c9db541f5414ac1d2e4 | 127.9.83.2:44831 | HEALTHY | <none> | 1 | 0
71a7ac59363f4dd1bc6b7e38506bc135 | 127.9.83.3:41171 | HEALTHY | <none> | 0 | 0
d3f0259187c841a98efbc8ed0eb8021a | 127.9.83.1:41183 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
flush_threshold_mb | 0 | runtime,experimental | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.9.83.1 | experimental | 127.9.83.1:41183
local_ip_for_outbound_sockets | 127.9.83.2 | experimental | 127.9.83.2:44831
local_ip_for_outbound_sockets | 127.9.83.3 | experimental | 127.9.83.3:41171
maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb | hidden | 127.9.83.1:41183
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb | hidden | 127.9.83.2:44831
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb | hidden | 127.9.83.3:41171
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 6bcce401df724b83973071baa1e5828b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
71a7ac59363f4dd1bc6b7e38506bc135 (127.9.83.3:41171): RUNNING
00af439d64e84c9db541f5414ac1d2e4 (127.9.83.2:44831): RUNNING [LEADER]
d3f0259187c841a98efbc8ed0eb8021a (127.9.83.1:41183): not running
State: STOPPING
Data state: TABLET_DATA_TOMBSTONED
Last status: Tombstoned
All reported replicas are:
A = 71a7ac59363f4dd1bc6b7e38506bc135
B = 00af439d64e84c9db541f5414ac1d2e4
C = d3f0259187c841a98efbc8ed0eb8021a
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+------------------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 1 | 24 | Yes
B | A B* C | 1 | 24 | Yes
C | [config not available] | | |
The cluster doesn't have any matching system tables
Summary by table
I20260928 01:57:26.956140 11398 tablet_bootstrap.cc:492] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap starting.
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
other-table | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
test-workload | 2 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 2
Tablets | 2
Replicas | 4
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 2 table(s) are not healthy
FAILED
I20260928 01:57:26.957402 9548 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20260928 01:57:26.969960 11398 tablet_bootstrap.cc:492] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=0 applied=24 ignored=20} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 01:57:26.970237 11398 tablet_bootstrap.cc:492] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Bootstrap complete.
I20260928 01:57:26.970408 11398 ts_tablet_manager.cc:1403] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260928 01:57:26.970990 11398 raft_consensus.cc:359] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } } peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } attrs { } }
I20260928 01:57:26.971089 11398 raft_consensus.cc:740] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f0259187c841a98efbc8ed0eb8021a, State: Initialized, Role: FOLLOWER
I20260928 01:57:26.971238 11398 consensus_queue.cc:260] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 1.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "71a7ac59363f4dd1bc6b7e38506bc135" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41171 } } peers { permanent_uuid: "00af439d64e84c9db541f5414ac1d2e4" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 44831 } } peers { permanent_uuid: "d3f0259187c841a98efbc8ed0eb8021a" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 41183 } attrs { } }
I20260928 01:57:26.971575 11398 ts_tablet_manager.cc:1434] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20260928 01:57:26.972223 10265 tablet_copy_service.cc:342] P 00af439d64e84c9db541f5414ac1d2e4: Request end of tablet copy session d3f0259187c841a98efbc8ed0eb8021a-6bcce401df724b83973071baa1e5828b received from {username='slave'} at 127.9.83.1:46715
I20260928 01:57:26.972303 10265 tablet_copy_service.cc:434] P 00af439d64e84c9db541f5414ac1d2e4: ending tablet copy session d3f0259187c841a98efbc8ed0eb8021a-6bcce401df724b83973071baa1e5828b on tablet 6bcce401df724b83973071baa1e5828b with peer d3f0259187c841a98efbc8ed0eb8021a
I20260928 01:57:27.055261 11311 maintenance_manager.cc:419] P d3f0259187c841a98efbc8ed0eb8021a: Scheduling UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b): 7929 bytes on disk
I20260928 01:57:27.055757 11200 maintenance_manager.cc:643] P d3f0259187c841a98efbc8ed0eb8021a: UndoDeltaBlockGCOp(6bcce401df724b83973071baa1e5828b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20260928 01:57:27.338919 11264 raft_consensus.cc:1217] T 6bcce401df724b83973071baa1e5828b P d3f0259187c841a98efbc8ed0eb8021a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.23->[1.24-1.24] Dedup: 1.24->[]
I20260928 01:57:27.342531 10356 tablet_service.cc:1467] Tablet server has 0 leaders and 0 scanners
I20260928 01:57:27.342531 11244 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
I20260928 01:57:27.343533 10225 tablet_service.cc:1467] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
a9dce204f3a642aa84955400c67e32fd | 127.9.83.62:34493 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
master_add_server_when_underreplicated | false | hidden | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
00af439d64e84c9db541f5414ac1d2e4 | 127.9.83.2:44831 | HEALTHY | <none> | 1 | 0
71a7ac59363f4dd1bc6b7e38506bc135 | 127.9.83.3:41171 | HEALTHY | <none> | 0 | 0
d3f0259187c841a98efbc8ed0eb8021a | 127.9.83.1:41183 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
-----------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+-------------------------
flush_threshold_mb | 0 | runtime,experimental | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.9.83.1 | experimental | 127.9.83.1:41183
local_ip_for_outbound_sockets | 127.9.83.2 | experimental | 127.9.83.2:44831
local_ip_for_outbound_sockets | 127.9.83.3 | experimental | 127.9.83.3:41171
maintenance_manager_polling_interval_ms | 100 | hidden | all 3 server(s) checked
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb | hidden | 127.9.83.1:41183
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb | hidden | 127.9.83.2:44831
server_dump_info_path | /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb | hidden | 127.9.83.3:41171
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.9.83.20:44475 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
other-table | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
test-workload | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 2
Tablets | 2
Replicas | 4
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20260928 01:57:27.370616 9548 log_verifier.cc:126] Checking tablet 6bcce401df724b83973071baa1e5828b
I20260928 01:57:27.395080 9548 log_verifier.cc:177] Verified matching terms for 24 ops in tablet 6bcce401df724b83973071baa1e5828b
I20260928 01:57:27.395164 9548 log_verifier.cc:126] Checking tablet f7281e2e7d0b4d60a7b5242c208c1d34
I20260928 01:57:27.404721 9548 log_verifier.cc:177] Verified matching terms for 17 ops in tablet f7281e2e7d0b4d60a7b5242c208c1d34
W20260928 01:57:27.920663 9548 scanner-internal.cc:464] Time spent opening tablet: real 0.509s user 0.001s sys 0.000s
I20260928 01:57:27.940490 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 11176
I20260928 01:57:27.945822 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 10156
I20260928 01:57:27.951329 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 10294
I20260928 01:57:27.956665 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 10425
2026-09-28T01:57:27Z chronyd exiting
[ OK ] DeleteTableITest.TestAutoTombstoneAfterCrashDuringTabletCopy (9527 ms)
[ RUN ] DeleteTableITest.TestNoDeleteTombstonedTablets
2026-09-28T01:57:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-28T01:57:27Z Disabled control of system clock
I20260928 01:57:27.988270 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.83.62:40445
--webserver_interface=127.9.83.62
--webserver_port=0
--builtin_ntp_servers=127.9.83.20:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.83.62:40445
--catalog_manager_evict_excess_replicas=false
--master_add_server_when_underreplicated=false with env {}
W20260928 01:57:28.097505 11445 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:28.097721 11445 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:28.097767 11445 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:28.101110 11445 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 01:57:28.101166 11445 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:28.101184 11445 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 01:57:28.101200 11445 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260928 01:57:28.105168 11445 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/wal
--catalog_manager_evict_excess_replicas=false
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.9.83.62:40445
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.83.62:40445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.83.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.11445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:28.106189 11445 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:28.107329 11445 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:28.112634 11450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:28.112581 11451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:28.112583 11453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:28.112797 11445 server_base.cc:1061] running on GCE node
I20260928 01:57:28.113312 11445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:28.114257 11445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:28.115417 11445 hybrid_clock.cc:648] HybridClock initialized: now 1790560648115402 us; error 36 us; skew 500 ppm
I20260928 01:57:28.117081 11445 webserver.cc:533] Webserver started at http://127.9.83.62:44991/ using document root <none> and password file <none>
I20260928 01:57:28.117599 11445 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:28.117653 11445 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:28.117820 11445 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 01:57:28.119356 11445 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/data/instance:
uuid: "d375d09d2b5c41538c6235f015669160"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.119794 11445 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/wal/instance:
uuid: "d375d09d2b5c41538c6235f015669160"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.123023 11445 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 01:57:28.124949 11459 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:28.125890 11445 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 01:57:28.126000 11445 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/wal
uuid: "d375d09d2b5c41538c6235f015669160"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.126082 11445 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:28.137766 11445 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:28.138450 11445 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:28.138623 11445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:28.145406 11511 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.62:40445 every 8 connection(s)
I20260928 01:57:28.145408 11445 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.62:40445
I20260928 01:57:28.146437 11445 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
I20260928 01:57:28.149113 11512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:28.152935 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 11445
I20260928 01:57:28.153033 9548 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/master-0/wal/instance
I20260928 01:57:28.153873 11512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160: Bootstrap starting.
I20260928 01:57:28.156241 11512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160: Neither blocks nor log segments found. Creating new log.
I20260928 01:57:28.157040 11512 log.cc:826] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:28.158679 11512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160: No bootstrap required, opened a new log
I20260928 01:57:28.161480 11512 raft_consensus.cc:359] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d375d09d2b5c41538c6235f015669160" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 40445 } }
I20260928 01:57:28.161692 11512 raft_consensus.cc:385] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 01:57:28.161739 11512 raft_consensus.cc:740] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d375d09d2b5c41538c6235f015669160, State: Initialized, Role: FOLLOWER
I20260928 01:57:28.162149 11512 consensus_queue.cc:260] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d375d09d2b5c41538c6235f015669160" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 40445 } }
I20260928 01:57:28.162287 11512 raft_consensus.cc:399] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 01:57:28.162388 11512 raft_consensus.cc:493] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 01:57:28.162477 11512 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [term 0 FOLLOWER]: Advancing to term 1
I20260928 01:57:28.163219 11512 raft_consensus.cc:515] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d375d09d2b5c41538c6235f015669160" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 40445 } }
I20260928 01:57:28.163491 11512 leader_election.cc:304] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d375d09d2b5c41538c6235f015669160; no voters:
I20260928 01:57:28.163733 11512 leader_election.cc:290] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 01:57:28.163836 11517 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 01:57:28.164026 11517 raft_consensus.cc:697] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [term 1 LEADER]: Becoming Leader. State: Replica: d375d09d2b5c41538c6235f015669160, State: Running, Role: LEADER
I20260928 01:57:28.164320 11517 consensus_queue.cc:237] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d375d09d2b5c41538c6235f015669160" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 40445 } }
I20260928 01:57:28.164717 11512 sys_catalog.cc:565] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 01:57:28.165723 11519 sys_catalog.cc:455] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d375d09d2b5c41538c6235f015669160. Latest consensus state: current_term: 1 leader_uuid: "d375d09d2b5c41538c6235f015669160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d375d09d2b5c41538c6235f015669160" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 40445 } } }
I20260928 01:57:28.165843 11519 sys_catalog.cc:458] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [sys.catalog]: This master's current role is: LEADER
I20260928 01:57:28.166385 11518 sys_catalog.cc:455] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d375d09d2b5c41538c6235f015669160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d375d09d2b5c41538c6235f015669160" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 40445 } } }
I20260928 01:57:28.166473 11518 sys_catalog.cc:458] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160 [sys.catalog]: This master's current role is: LEADER
I20260928 01:57:28.166713 11526 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260928 01:57:28.169548 11526 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260928 01:57:28.174317 11526 catalog_manager.cc:1396] Generated new cluster ID: d4b06114c6fb43658213b420c2159f59
I20260928 01:57:28.174400 11526 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260928 01:57:28.190811 11526 catalog_manager.cc:1419] Generated new certificate authority record
I20260928 01:57:28.191864 11526 catalog_manager.cc:1553] Loading token signing keys...
I20260928 01:57:28.205184 11526 catalog_manager.cc:6219] T 00000000000000000000000000000000 P d375d09d2b5c41538c6235f015669160: Generated new TSK 0
I20260928 01:57:28.205850 11526 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260928 01:57:28.214730 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.1:0
--local_ip_for_outbound_sockets=127.9.83.1
--webserver_interface=127.9.83.1
--webserver_port=0
--tserver_master_addrs=127.9.83.62:40445
--builtin_ntp_servers=127.9.83.20:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260928 01:57:28.323884 11536 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:28.324112 11536 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:28.324169 11536 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:28.327715 11536 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:28.327857 11536 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.1
I20260928 01:57:28.332427 11536 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.83.1
--webserver_port=0
--tserver_master_addrs=127.9.83.62:40445
--never_fsync=true
--heap_profile_path=/tmp/kudu.11536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:28.333549 11536 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:28.334705 11536 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:28.341477 11541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:28.341562 11544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:28.341699 11536 server_base.cc:1061] running on GCE node
W20260928 01:57:28.341471 11542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:28.342129 11536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:28.342690 11536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:28.343842 11536 hybrid_clock.cc:648] HybridClock initialized: now 1790560648343824 us; error 27 us; skew 500 ppm
I20260928 01:57:28.345753 11536 webserver.cc:533] Webserver started at http://127.9.83.1:42463/ using document root <none> and password file <none>
I20260928 01:57:28.346361 11536 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:28.346424 11536 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:28.346619 11536 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 01:57:28.348338 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/data/instance:
uuid: "8e258c6e867d41c1802fa0871761065f"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.348877 11536 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/wal/instance:
uuid: "8e258c6e867d41c1802fa0871761065f"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.353087 11536 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 01:57:28.355464 11550 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:28.356627 11536 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 01:57:28.356755 11536 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/wal
uuid: "8e258c6e867d41c1802fa0871761065f"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.356891 11536 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:28.369510 11536 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:28.370151 11536 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:28.370321 11536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:28.370918 11536 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:28.371858 11536 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 01:57:28.371913 11536 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:28.371966 11536 ts_tablet_manager.cc:616] Registered 0 tablets
I20260928 01:57:28.371989 11536 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:28.380365 11536 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.1:43681
I20260928 01:57:28.380427 11663 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.1:43681 every 8 connection(s)
I20260928 01:57:28.381254 11536 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
I20260928 01:57:28.389428 11664 heartbeater.cc:344] Connected to a master server at 127.9.83.62:40445
I20260928 01:57:28.389642 11664 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:28.390120 11664 heartbeater.cc:507] Master 127.9.83.62:40445 requested a full tablet report, sending...
I20260928 01:57:28.391314 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 11536
I20260928 01:57:28.391427 9548 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-0/wal/instance
I20260928 01:57:28.391579 11476 ts_manager.cc:194] Registered new tserver with Master: 8e258c6e867d41c1802fa0871761065f (127.9.83.1:43681)
I20260928 01:57:28.392771 11476 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.1:60855
I20260928 01:57:28.393637 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.2:0
--local_ip_for_outbound_sockets=127.9.83.2
--webserver_interface=127.9.83.2
--webserver_port=0
--tserver_master_addrs=127.9.83.62:40445
--builtin_ntp_servers=127.9.83.20:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260928 01:57:28.495158 11667 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:28.495385 11667 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:28.495442 11667 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:28.498799 11667 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:28.498921 11667 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.2
I20260928 01:57:28.502724 11667 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.83.2
--webserver_port=0
--tserver_master_addrs=127.9.83.62:40445
--never_fsync=true
--heap_profile_path=/tmp/kudu.11667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:28.503808 11667 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:28.504931 11667 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:28.511127 11672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:28.511132 11673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:28.511279 11667 server_base.cc:1061] running on GCE node
W20260928 01:57:28.511409 11675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:28.511731 11667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:28.512267 11667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:28.513433 11667 hybrid_clock.cc:648] HybridClock initialized: now 1790560648513415 us; error 28 us; skew 500 ppm
I20260928 01:57:28.515144 11667 webserver.cc:533] Webserver started at http://127.9.83.2:43155/ using document root <none> and password file <none>
I20260928 01:57:28.515681 11667 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:28.515729 11667 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:28.515913 11667 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 01:57:28.517417 11667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/data/instance:
uuid: "da431f08920849cea6adbc26a62e7824"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.517836 11667 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/wal/instance:
uuid: "da431f08920849cea6adbc26a62e7824"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.521126 11667 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 01:57:28.523232 11681 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:28.524238 11667 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 01:57:28.524340 11667 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/wal
uuid: "da431f08920849cea6adbc26a62e7824"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.524428 11667 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:28.542178 11667 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:28.542932 11667 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:28.543111 11667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:28.543682 11667 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:28.544656 11667 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 01:57:28.544703 11667 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:28.544755 11667 ts_tablet_manager.cc:616] Registered 0 tablets
I20260928 01:57:28.544781 11667 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:28.553576 11794 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.2:35051 every 8 connection(s)
I20260928 01:57:28.553594 11667 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.2:35051
I20260928 01:57:28.554524 11667 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb
I20260928 01:57:28.559195 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 11667
I20260928 01:57:28.559290 9548 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-1/wal/instance
I20260928 01:57:28.561460 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.3:0
--local_ip_for_outbound_sockets=127.9.83.3
--webserver_interface=127.9.83.3
--webserver_port=0
--tserver_master_addrs=127.9.83.62:40445
--builtin_ntp_servers=127.9.83.20:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260928 01:57:28.563105 11795 heartbeater.cc:344] Connected to a master server at 127.9.83.62:40445
I20260928 01:57:28.563392 11795 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:28.563962 11795 heartbeater.cc:507] Master 127.9.83.62:40445 requested a full tablet report, sending...
I20260928 01:57:28.565047 11476 ts_manager.cc:194] Registered new tserver with Master: da431f08920849cea6adbc26a62e7824 (127.9.83.2:35051)
I20260928 01:57:28.565704 11476 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.2:35765
W20260928 01:57:28.662433 11798 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:28.662644 11798 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:28.662686 11798 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:28.666056 11798 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:28.666159 11798 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.3
I20260928 01:57:28.670051 11798 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.83.3
--webserver_port=0
--tserver_master_addrs=127.9.83.62:40445
--never_fsync=true
--heap_profile_path=/tmp/kudu.11798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:28.671124 11798 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:28.672186 11798 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:28.678546 11804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:28.678546 11803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:28.678565 11806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:28.678903 11798 server_base.cc:1061] running on GCE node
I20260928 01:57:28.679255 11798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:28.679811 11798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:28.680971 11798 hybrid_clock.cc:648] HybridClock initialized: now 1790560648680961 us; error 26 us; skew 500 ppm
I20260928 01:57:28.682689 11798 webserver.cc:533] Webserver started at http://127.9.83.3:37857/ using document root <none> and password file <none>
I20260928 01:57:28.683269 11798 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:28.683329 11798 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:28.683499 11798 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 01:57:28.685075 11798 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/data/instance:
uuid: "1f5e82d9d2954a0d85575b1c67b29a9d"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.685525 11798 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/wal/instance:
uuid: "1f5e82d9d2954a0d85575b1c67b29a9d"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.688778 11798 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 01:57:28.690927 11812 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:28.691994 11798 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 01:57:28.692093 11798 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/wal
uuid: "1f5e82d9d2954a0d85575b1c67b29a9d"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.692174 11798 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:28.714918 11798 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:28.715569 11798 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:28.715724 11798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:28.716264 11798 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:28.717218 11798 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 01:57:28.717264 11798 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:28.717316 11798 ts_tablet_manager.cc:616] Registered 0 tablets
I20260928 01:57:28.717340 11798 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:28.725737 11798 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.3:45657
I20260928 01:57:28.725776 11925 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.3:45657 every 8 connection(s)
I20260928 01:57:28.726637 11798 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb
I20260928 01:57:28.727406 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 11798
I20260928 01:57:28.727519 9548 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-2/wal/instance
I20260928 01:57:28.729748 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.4:0
--local_ip_for_outbound_sockets=127.9.83.4
--webserver_interface=127.9.83.4
--webserver_port=0
--tserver_master_addrs=127.9.83.62:40445
--builtin_ntp_servers=127.9.83.20:41309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260928 01:57:28.736011 11926 heartbeater.cc:344] Connected to a master server at 127.9.83.62:40445
I20260928 01:57:28.736285 11926 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:28.736991 11926 heartbeater.cc:507] Master 127.9.83.62:40445 requested a full tablet report, sending...
I20260928 01:57:28.738123 11476 ts_manager.cc:194] Registered new tserver with Master: 1f5e82d9d2954a0d85575b1c67b29a9d (127.9.83.3:45657)
I20260928 01:57:28.738896 11476 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.3:47479
W20260928 01:57:28.835551 11929 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:28.835776 11929 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:28.835819 11929 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:28.839272 11929 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:28.839402 11929 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.4
I20260928 01:57:28.843544 11929 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:41309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.9.83.4
--webserver_port=0
--tserver_master_addrs=127.9.83.62:40445
--never_fsync=true
--heap_profile_path=/tmp/kudu.11929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.4
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:28.844624 11929 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:28.845693 11929 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:28.851919 11934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:28.851931 11935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:28.851935 11937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:28.852141 11929 server_base.cc:1061] running on GCE node
I20260928 01:57:28.852541 11929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:28.853085 11929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:28.854267 11929 hybrid_clock.cc:648] HybridClock initialized: now 1790560648854253 us; error 25 us; skew 500 ppm
I20260928 01:57:28.856035 11929 webserver.cc:533] Webserver started at http://127.9.83.4:38013/ using document root <none> and password file <none>
I20260928 01:57:28.856601 11929 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:28.856658 11929 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:28.856827 11929 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 01:57:28.858276 11929 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/data/instance:
uuid: "67f566a11c2d423495ccc094ca299245"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.858747 11929 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/wal/instance:
uuid: "67f566a11c2d423495ccc094ca299245"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.861820 11929 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 01:57:28.863832 11943 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:28.864794 11929 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 01:57:28.864898 11929 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/wal
uuid: "67f566a11c2d423495ccc094ca299245"
format_stamp: "Formatted at 2026-09-28 01:57:28 on dist-test-slave-lkt6"
I20260928 01:57:28.864996 11929 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:28.874868 11929 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:28.875425 11929 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:28.875610 11929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:28.876117 11929 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:28.877099 11929 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 01:57:28.877146 11929 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:28.877194 11929 ts_tablet_manager.cc:616] Registered 0 tablets
I20260928 01:57:28.877219 11929 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:28.885483 11929 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.4:40391
I20260928 01:57:28.885506 12056 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.4:40391 every 8 connection(s)
I20260928 01:57:28.886349 11929 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/data/info.pb
I20260928 01:57:28.895404 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 11929
I20260928 01:57:28.895507 9548 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0/minicluster-data/ts-3/wal/instance
I20260928 01:57:28.901710 12057 heartbeater.cc:344] Connected to a master server at 127.9.83.62:40445
I20260928 01:57:28.901921 12057 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:28.902381 12057 heartbeater.cc:507] Master 127.9.83.62:40445 requested a full tablet report, sending...
I20260928 01:57:28.903324 11476 ts_manager.cc:194] Registered new tserver with Master: 67f566a11c2d423495ccc094ca299245 (127.9.83.4:40391)
I20260928 01:57:28.903828 11476 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.4:41265
I20260928 01:57:28.909706 9548 external_mini_cluster.cc:955] 4 TS(s) registered with all masters
I20260928 01:57:28.925168 11476 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48664:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 01:57:28.941663 11991 tablet_service.cc:1511] Processing CreateTablet for tablet 62220c6692fb451e98a236720f8f556b (DEFAULT_TABLE table=test-workload [id=3174afc8436f47c28c1612bf8620d389]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 01:57:28.941704 11860 tablet_service.cc:1511] Processing CreateTablet for tablet 62220c6692fb451e98a236720f8f556b (DEFAULT_TABLE table=test-workload [id=3174afc8436f47c28c1612bf8620d389]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 01:57:28.942255 11598 tablet_service.cc:1511] Processing CreateTablet for tablet 62220c6692fb451e98a236720f8f556b (DEFAULT_TABLE table=test-workload [id=3174afc8436f47c28c1612bf8620d389]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 01:57:28.942584 11860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62220c6692fb451e98a236720f8f556b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:28.942584 11991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62220c6692fb451e98a236720f8f556b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:28.943032 11598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62220c6692fb451e98a236720f8f556b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:28.947006 12079 tablet_bootstrap.cc:492] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245: Bootstrap starting.
I20260928 01:57:28.948184 12081 tablet_bootstrap.cc:492] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d: Bootstrap starting.
I20260928 01:57:28.948191 12080 tablet_bootstrap.cc:492] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f: Bootstrap starting.
I20260928 01:57:28.948912 12079 tablet_bootstrap.cc:654] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245: Neither blocks nor log segments found. Creating new log.
I20260928 01:57:28.949548 12079 log.cc:826] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:28.950130 12081 tablet_bootstrap.cc:654] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d: Neither blocks nor log segments found. Creating new log.
I20260928 01:57:28.950130 12080 tablet_bootstrap.cc:654] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f: Neither blocks nor log segments found. Creating new log.
I20260928 01:57:28.950934 12081 log.cc:826] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:28.950934 12080 log.cc:826] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:28.951450 12079 tablet_bootstrap.cc:492] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245: No bootstrap required, opened a new log
I20260928 01:57:28.951637 12079 ts_tablet_manager.cc:1403] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 01:57:28.952411 12080 tablet_bootstrap.cc:492] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f: No bootstrap required, opened a new log
I20260928 01:57:28.952416 12081 tablet_bootstrap.cc:492] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d: No bootstrap required, opened a new log
I20260928 01:57:28.952580 12081 ts_tablet_manager.cc:1403] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 01:57:28.952580 12080 ts_tablet_manager.cc:1403] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 01:57:28.953845 12079 raft_consensus.cc:359] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } }
I20260928 01:57:28.954013 12079 raft_consensus.cc:385] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 01:57:28.954063 12079 raft_consensus.cc:740] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f566a11c2d423495ccc094ca299245, State: Initialized, Role: FOLLOWER
I20260928 01:57:28.954450 12079 consensus_queue.cc:260] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } }
I20260928 01:57:28.955020 12057 heartbeater.cc:499] Master 127.9.83.62:40445 was elected leader, sending a full tablet report...
I20260928 01:57:28.955240 12079 ts_tablet_manager.cc:1434] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 01:57:28.955444 12081 raft_consensus.cc:359] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } }
I20260928 01:57:28.955440 12080 raft_consensus.cc:359] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } }
I20260928 01:57:28.955617 12081 raft_consensus.cc:385] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 01:57:28.955617 12080 raft_consensus.cc:385] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 01:57:28.955667 12081 raft_consensus.cc:740] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5e82d9d2954a0d85575b1c67b29a9d, State: Initialized, Role: FOLLOWER
I20260928 01:57:28.955667 12080 raft_consensus.cc:740] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e258c6e867d41c1802fa0871761065f, State: Initialized, Role: FOLLOWER
I20260928 01:57:28.956053 12080 consensus_queue.cc:260] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } }
I20260928 01:57:28.956091 12081 consensus_queue.cc:260] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } }
I20260928 01:57:28.956822 12081 ts_tablet_manager.cc:1434] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 01:57:28.956846 11926 heartbeater.cc:499] Master 127.9.83.62:40445 was elected leader, sending a full tablet report...
I20260928 01:57:28.956979 12080 ts_tablet_manager.cc:1434] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 01:57:28.957697 11664 heartbeater.cc:499] Master 127.9.83.62:40445 was elected leader, sending a full tablet report...
W20260928 01:57:28.977773 11927 tablet.cc:2433] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 01:57:29.132453 11665 tablet.cc:2433] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 01:57:29.137162 12058 tablet.cc:2433] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 01:57:29.336480 12085 raft_consensus.cc:493] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 01:57:29.336674 12085 raft_consensus.cc:515] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } }
I20260928 01:57:29.337821 12085 leader_election.cc:290] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e258c6e867d41c1802fa0871761065f (127.9.83.1:43681), 1f5e82d9d2954a0d85575b1c67b29a9d (127.9.83.3:45657)
I20260928 01:57:29.338920 12086 raft_consensus.cc:493] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 01:57:29.339076 12086 raft_consensus.cc:515] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } }
I20260928 01:57:29.340173 12086 leader_election.cc:290] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67f566a11c2d423495ccc094ca299245 (127.9.83.4:40391), 8e258c6e867d41c1802fa0871761065f (127.9.83.1:43681)
I20260928 01:57:29.342329 11618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62220c6692fb451e98a236720f8f556b" candidate_uuid: "67f566a11c2d423495ccc094ca299245" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e258c6e867d41c1802fa0871761065f" is_pre_election: true
I20260928 01:57:29.342577 11618 raft_consensus.cc:2468] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67f566a11c2d423495ccc094ca299245 in term 0.
I20260928 01:57:29.343077 11944 leader_election.cc:304] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67f566a11c2d423495ccc094ca299245, 8e258c6e867d41c1802fa0871761065f; no voters:
I20260928 01:57:29.343233 11880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62220c6692fb451e98a236720f8f556b" candidate_uuid: "67f566a11c2d423495ccc094ca299245" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" is_pre_election: true
I20260928 01:57:29.343395 11880 raft_consensus.cc:2468] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67f566a11c2d423495ccc094ca299245 in term 0.
I20260928 01:57:29.343637 12085 raft_consensus.cc:2804] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 01:57:29.343725 12085 raft_consensus.cc:493] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 01:57:29.343782 12085 raft_consensus.cc:3060] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 0 FOLLOWER]: Advancing to term 1
I20260928 01:57:29.344864 12085 raft_consensus.cc:515] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } }
I20260928 01:57:29.344944 11618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62220c6692fb451e98a236720f8f556b" candidate_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e258c6e867d41c1802fa0871761065f" is_pre_election: true
I20260928 01:57:29.345026 12011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62220c6692fb451e98a236720f8f556b" candidate_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f566a11c2d423495ccc094ca299245" is_pre_election: true
I20260928 01:57:29.345064 11618 raft_consensus.cc:2468] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f5e82d9d2954a0d85575b1c67b29a9d in term 0.
I20260928 01:57:29.345206 12011 raft_consensus.cc:2393] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1f5e82d9d2954a0d85575b1c67b29a9d in current term 1: Already voted for candidate 67f566a11c2d423495ccc094ca299245 in this term.
I20260928 01:57:29.345234 12085 leader_election.cc:290] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [CANDIDATE]: Term 1 election: Requested vote from peers 8e258c6e867d41c1802fa0871761065f (127.9.83.1:43681), 1f5e82d9d2954a0d85575b1c67b29a9d (127.9.83.3:45657)
I20260928 01:57:29.345479 11618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62220c6692fb451e98a236720f8f556b" candidate_uuid: "67f566a11c2d423495ccc094ca299245" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e258c6e867d41c1802fa0871761065f"
I20260928 01:57:29.345526 11813 leader_election.cc:304] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5e82d9d2954a0d85575b1c67b29a9d, 8e258c6e867d41c1802fa0871761065f; no voters:
I20260928 01:57:29.345584 11618 raft_consensus.cc:3060] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f [term 0 FOLLOWER]: Advancing to term 1
I20260928 01:57:29.345712 11880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62220c6692fb451e98a236720f8f556b" candidate_uuid: "67f566a11c2d423495ccc094ca299245" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d"
I20260928 01:57:29.345753 12086 raft_consensus.cc:2804] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 01:57:29.345829 11880 raft_consensus.cc:3060] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 01:57:29.346561 11618 raft_consensus.cc:2468] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67f566a11c2d423495ccc094ca299245 in term 1.
I20260928 01:57:29.346731 11880 raft_consensus.cc:2468] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67f566a11c2d423495ccc094ca299245 in term 1.
I20260928 01:57:29.346845 11944 leader_election.cc:304] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67f566a11c2d423495ccc094ca299245, 8e258c6e867d41c1802fa0871761065f; no voters:
I20260928 01:57:29.346840 12086 raft_consensus.cc:493] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 01:57:29.346915 12086 raft_consensus.cc:3060] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 1 FOLLOWER]: Advancing to term 2
I20260928 01:57:29.346990 12085 raft_consensus.cc:2804] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 01:57:29.347189 12085 raft_consensus.cc:697] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 1 LEADER]: Becoming Leader. State: Replica: 67f566a11c2d423495ccc094ca299245, State: Running, Role: LEADER
I20260928 01:57:29.347455 12085 consensus_queue.cc:237] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } }
I20260928 01:57:29.347667 12086 raft_consensus.cc:515] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } }
I20260928 01:57:29.348232 12086 leader_election.cc:290] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [CANDIDATE]: Term 2 election: Requested vote from peers 67f566a11c2d423495ccc094ca299245 (127.9.83.4:40391), 8e258c6e867d41c1802fa0871761065f (127.9.83.1:43681)
I20260928 01:57:29.348591 11618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62220c6692fb451e98a236720f8f556b" candidate_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e258c6e867d41c1802fa0871761065f"
I20260928 01:57:29.348603 12011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62220c6692fb451e98a236720f8f556b" candidate_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67f566a11c2d423495ccc094ca299245"
I20260928 01:57:29.348696 11618 raft_consensus.cc:3060] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f [term 1 FOLLOWER]: Advancing to term 2
I20260928 01:57:29.349573 11618 raft_consensus.cc:2468] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f5e82d9d2954a0d85575b1c67b29a9d in term 2.
I20260928 01:57:29.349855 11813 leader_election.cc:304] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1f5e82d9d2954a0d85575b1c67b29a9d, 8e258c6e867d41c1802fa0871761065f; no voters: 67f566a11c2d423495ccc094ca299245
I20260928 01:57:29.350025 12086 raft_consensus.cc:2804] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 2 FOLLOWER]: Leader election won for term 2
I20260928 01:57:29.350251 12086 raft_consensus.cc:697] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 2 LEADER]: Becoming Leader. State: Replica: 1f5e82d9d2954a0d85575b1c67b29a9d, State: Running, Role: LEADER
I20260928 01:57:29.350523 12086 consensus_queue.cc:237] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } }
I20260928 01:57:29.350594 11476 catalog_manager.cc:5838] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67f566a11c2d423495ccc094ca299245 (127.9.83.4). New cstate: current_term: 1 leader_uuid: "67f566a11c2d423495ccc094ca299245" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } health_report { overall_health: UNKNOWN } } }
I20260928 01:57:29.353485 11476 catalog_manager.cc:5838] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d reported cstate change: term changed from 1 to 2, leader changed from 67f566a11c2d423495ccc094ca299245 (127.9.83.4) to 1f5e82d9d2954a0d85575b1c67b29a9d (127.9.83.3). New cstate: current_term: 2 leader_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } health_report { overall_health: HEALTHY } } }
I20260928 01:57:29.386417 9548 delete_table-itest.cc:1200] finding leader...
I20260928 01:57:29.567986 11795 heartbeater.cc:499] Master 127.9.83.62:40445 was elected leader, sending a full tablet report...
I20260928 01:57:29.770083 11880 raft_consensus.cc:1240] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [term 2 LEADER]: Rejecting Update request from peer 67f566a11c2d423495ccc094ca299245 for earlier term 1. Current term is 2. Ops: []
I20260928 01:57:29.770854 12085 consensus_queue.cc:1059] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 01:57:29.771044 12085 consensus_queue.cc:1243] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [LEADER]: Peer 1f5e82d9d2954a0d85575b1c67b29a9d log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260928 01:57:29.771198 12092 raft_consensus.cc:3055] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 1 LEADER]: Stepping down as leader of term 1
I20260928 01:57:29.771266 12092 raft_consensus.cc:740] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 67f566a11c2d423495ccc094ca299245, State: Running, Role: LEADER
I20260928 01:57:29.771494 12092 consensus_queue.cc:260] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } }
I20260928 01:57:29.771770 12092 raft_consensus.cc:3060] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 1 FOLLOWER]: Advancing to term 2
I20260928 01:57:29.858795 11618 raft_consensus.cc:1275] T 62220c6692fb451e98a236720f8f556b P 8e258c6e867d41c1802fa0871761065f [term 2 FOLLOWER]: Refusing update from remote peer 1f5e82d9d2954a0d85575b1c67b29a9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260928 01:57:29.859504 12098 consensus_queue.cc:1048] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 01:57:29.864346 12011 raft_consensus.cc:1275] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 2 FOLLOWER]: Refusing update from remote peer 1f5e82d9d2954a0d85575b1c67b29a9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260928 01:57:29.864440 12011 pending_rounds.cc:85] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245: Aborting all ops after (but not including) 0
I20260928 01:57:29.864498 12011 pending_rounds.cc:107] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260928 01:57:29.864557 12011 raft_consensus.cc:2889] T 62220c6692fb451e98a236720f8f556b P 67f566a11c2d423495ccc094ca299245 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260928 01:57:29.865067 12098 consensus_queue.cc:1048] T 62220c6692fb451e98a236720f8f556b P 1f5e82d9d2954a0d85575b1c67b29a9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 01:57:29.874037 9548 delete_table-itest.cc:1207] replacing replica...
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/delete_table-itest.cc:1228: Failure
Failed
Bad status: Illegal state: Replica 67f566a11c2d423495ccc094ca299245 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67f566a11c2d423495ccc094ca299245" member_type: VOTER last_known_addr { host: "127.9.83.4" port: 40391 } } peers { permanent_uuid: "8e258c6e867d41c1802fa0871761065f" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43681 } } peers { permanent_uuid: "1f5e82d9d2954a0d85575b1c67b29a9d" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 45657 } } }
I20260928 01:57:29.875329 9548 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20260928 01:57:29.875383 9548 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 8e258c6e867d41c1802fa0871761065f and pid 11536
************************ BEGIN STACKS **************************
[New LWP 11537]
[New LWP 11538]
[New LWP 11539]
[New LWP 11540]
[New LWP 11546]
[New LWP 11547]
[New LWP 11548]
[New LWP 11551]
[New LWP 11552]
[New LWP 11553]
[New LWP 11554]
[New LWP 11555]
[New LWP 11556]
[New LWP 11557]
[New LWP 11558]
[New LWP 11559]
[New LWP 11560]
[New LWP 11561]
[New LWP 11562]
[New LWP 11563]
[New LWP 11564]
[New LWP 11565]
[New LWP 11566]
[New LWP 11567]
[New LWP 11568]
[New LWP 11569]
[New LWP 11570]
[New LWP 11571]
[New LWP 11572]
[New LWP 11573]
[New LWP 11574]
[New LWP 11575]
[New LWP 11576]
[New LWP 11577]
[New LWP 11578]
[New LWP 11579]
[New LWP 11580]
[New LWP 11581]
[New LWP 11582]
[New LWP 11583]
[New LWP 11584]
[New LWP 11585]
[New LWP 11586]
[New LWP 11587]
[New LWP 11588]
[New LWP 11589]
[New LWP 11590]
[New LWP 11591]
[New LWP 11592]
[New LWP 11593]
[New LWP 11594]
[New LWP 11595]
[New LWP 11596]
[New LWP 11597]
[New LWP 11598]
[New LWP 11599]
[New LWP 11600]
[New LWP 11601]
[New LWP 11602]
[New LWP 11603]
[New LWP 11604]
[New LWP 11605]
[New LWP 11606]
[New LWP 11607]
[New LWP 11608]
[New LWP 11609]
[New LWP 11610]
[New LWP 11611]
[New LWP 11612]
[New LWP 11613]
[New LWP 11614]
[New LWP 11615]
[New LWP 11616]
[New LWP 11617]
[New LWP 11618]
[New LWP 11619]
[New LWP 11620]
[New LWP 11621]
[New LWP 11622]
[New LWP 11623]
[New LWP 11624]
[New LWP 11625]
[New LWP 11626]
[New LWP 11627]
[New LWP 11628]
[New LWP 11629]
[New LWP 11630]
[New LWP 11631]
[New LWP 11632]
[New LWP 11633]
[New LWP 11634]
[New LWP 11635]
[New LWP 11636]
[New LWP 11637]
[New LWP 11638]
[New LWP 11639]
[New LWP 11640]
[New LWP 11641]
[New LWP 11642]
[New LWP 11643]
[New LWP 11644]
[New LWP 11645]
[New LWP 11646]
[New LWP 11647]
[New LWP 11648]
[New LWP 11649]
[New LWP 11650]
[New LWP 11651]
[New LWP 11652]
[New LWP 11653]
[New LWP 11654]
[New LWP 11655]
[New LWP 11656]
[New LWP 11657]
[New LWP 11658]
[New LWP 11659]
[New LWP 11660]
[New LWP 11661]
[New LWP 11662]
[New LWP 11663]
[New LWP 11664]
[New LWP 11665]
[New LWP 12100]
0x00007f687f402d50 in ?? ()
Id Target Id Frame
* 1 LWP 11536 "kudu" 0x00007f687f402d50 in ?? ()
2 LWP 11537 "kudu" 0x00007f687f3fefb9 in ?? ()
3 LWP 11538 "kudu" 0x00007f687f3fefb9 in ?? ()
4 LWP 11539 "kudu" 0x00007f687f3fefb9 in ?? ()
5 LWP 11540 "kernel-watcher-" 0x00007f687f3fefb9 in ?? ()
6 LWP 11546 "ntp client-1154" 0x00007f687f4029e2 in ?? ()
7 LWP 11547 "file cache-evic" 0x00007f687f3fefb9 in ?? ()
8 LWP 11548 "sq_acceptor" 0x00007f687c747bb9 in ?? ()
9 LWP 11551 "rpc reactor-115" 0x00007f687c754947 in ?? ()
10 LWP 11552 "rpc reactor-115" 0x00007f687c754947 in ?? ()
11 LWP 11553 "rpc reactor-115" 0x00007f687c754947 in ?? ()
12 LWP 11554 "rpc reactor-115" 0x00007f687c754947 in ?? ()
13 LWP 11555 "MaintenanceMgr " 0x00007f687f3fead3 in ?? ()
14 LWP 11556 "txn-status-mana" 0x00007f687f3fefb9 in ?? ()
15 LWP 11557 "collect_and_rem" 0x00007f687f3fefb9 in ?? ()
16 LWP 11558 "tc-session-exp-" 0x00007f687f3fefb9 in ?? ()
17 LWP 11559 "rpc worker-1155" 0x00007f687f3fead3 in ?? ()
18 LWP 11560 "rpc worker-1156" 0x00007f687f3fead3 in ?? ()
19 LWP 11561 "rpc worker-1156" 0x00007f687f3fead3 in ?? ()
20 LWP 11562 "rpc worker-1156" 0x00007f687f3fead3 in ?? ()
21 LWP 11563 "rpc worker-1156" 0x00007f687f3fead3 in ?? ()
22 LWP 11564 "rpc worker-1156" 0x00007f687f3fead3 in ?? ()
23 LWP 11565 "rpc worker-1156" 0x00007f687f3fead3 in ?? ()
24 LWP 11566 "rpc worker-1156" 0x00007f687f3fead3 in ?? ()
25 LWP 11567 "rpc worker-1156" 0x00007f687f3fead3 in ?? ()
26 LWP 11568 "rpc worker-1156" 0x00007f687f3fead3 in ?? ()
27 LWP 11569 "rpc worker-1156" 0x00007f687f3fead3 in ?? ()
28 LWP 11570 "rpc worker-1157" 0x00007f687f3fead3 in ?? ()
29 LWP 11571 "rpc worker-1157" 0x00007f687f3fead3 in ?? ()
30 LWP 11572 "rpc worker-1157" 0x00007f687f3fead3 in ?? ()
31 LWP 11573 "rpc worker-1157" 0x00007f687f3fead3 in ?? ()
32 LWP 11574 "rpc worker-1157" 0x00007f687f3fead3 in ?? ()
33 LWP 11575 "rpc worker-1157" 0x00007f687f3fead3 in ?? ()
34 LWP 11576 "rpc worker-1157" 0x00007f687f3fead3 in ?? ()
35 LWP 11577 "rpc worker-1157" 0x00007f687f3fead3 in ?? ()
36 LWP 11578 "rpc worker-1157" 0x00007f687f3fead3 in ?? ()
37 LWP 11579 "rpc worker-1157" 0x00007f687f3fead3 in ?? ()
38 LWP 11580 "rpc worker-1158" 0x00007f687f3fead3 in ?? ()
39 LWP 11581 "rpc worker-1158" 0x00007f687f3fead3 in ?? ()
40 LWP 11582 "rpc worker-1158" 0x00007f687f3fead3 in ?? ()
41 LWP 11583 "rpc worker-1158" 0x00007f687f3fead3 in ?? ()
42 LWP 11584 "rpc worker-1158" 0x00007f687f3fead3 in ?? ()
43 LWP 11585 "rpc worker-1158" 0x00007f687f3fead3 in ?? ()
44 LWP 11586 "rpc worker-1158" 0x00007f687f3fead3 in ?? ()
45 LWP 11587 "rpc worker-1158" 0x00007f687f3fead3 in ?? ()
46 LWP 11588 "rpc worker-1158" 0x00007f687f3fead3 in ?? ()
47 LWP 11589 "rpc worker-1158" 0x00007f687f3fead3 in ?? ()
48 LWP 11590 "rpc worker-1159" 0x00007f687f3fead3 in ?? ()
49 LWP 11591 "rpc worker-1159" 0x00007f687f3fead3 in ?? ()
50 LWP 11592 "rpc worker-1159" 0x00007f687f3fead3 in ?? ()
51 LWP 11593 "rpc worker-1159" 0x00007f687f3fead3 in ?? ()
52 LWP 11594 "rpc worker-1159" 0x00007f687f3fead3 in ?? ()
53 LWP 11595 "rpc worker-1159" 0x00007f687f3fead3 in ?? ()
54 LWP 11596 "rpc worker-1159" 0x00007f687f3fead3 in ?? ()
55 LWP 11597 "rpc worker-1159" 0x00007f687f3fead3 in ?? ()
56 LWP 11598 "rpc worker-1159" 0x00007f687f3fead3 in ?? ()
57 LWP 11599 "rpc worker-1159" 0x00007f687f3fead3 in ?? ()
58 LWP 11600 "rpc worker-1160" 0x00007f687f3fead3 in ?? ()
59 LWP 11601 "rpc worker-1160" 0x00007f687f3fead3 in ?? ()
60 LWP 11602 "rpc worker-1160" 0x00007f687f3fead3 in ?? ()
61 LWP 11603 "rpc worker-1160" 0x00007f687f3fead3 in ?? ()
62 LWP 11604 "rpc worker-1160" 0x00007f687f3fead3 in ?? ()
63 LWP 11605 "rpc worker-1160" 0x00007f687f3fead3 in ?? ()
64 LWP 11606 "rpc worker-1160" 0x00007f687f3fead3 in ?? ()
65 LWP 11607 "rpc worker-1160" 0x00007f687f3fead3 in ?? ()
66 LWP 11608 "rpc worker-1160" 0x00007f687f3fead3 in ?? ()
67 LWP 11609 "rpc worker-1160" 0x00007f687f3fead3 in ?? ()
68 LWP 11610 "rpc worker-1161" 0x00007f687f3fead3 in ?? ()
69 LWP 11611 "rpc worker-1161" 0x00007f687f3fead3 in ?? ()
70 LWP 11612 "rpc worker-1161" 0x00007f687f3fead3 in ?? ()
71 LWP 11613 "rpc worker-1161" 0x00007f687f3fead3 in ?? ()
72 LWP 11614 "rpc worker-1161" 0x00007f687f3fead3 in ?? ()
73 LWP 11615 "rpc worker-1161" 0x00007f687f3fead3 in ?? ()
74 LWP 11616 "rpc worker-1161" 0x00007f687f3fead3 in ?? ()
75 LWP 11617 "rpc worker-1161" 0x00007f687f3fead3 in ?? ()
76 LWP 11618 "rpc worker-1161" 0x00007f687f3fead3 in ?? ()
77 LWP 11619 "rpc worker-1161" 0x00007f687f3fead3 in ?? ()
78 LWP 11620 "rpc worker-1162" 0x00007f687f3fead3 in ?? ()
79 LWP 11621 "rpc worker-1162" 0x00007f687f3fead3 in ?? ()
80 LWP 11622 "rpc worker-1162" 0x00007f687f3fead3 in ?? ()
81 LWP 11623 "rpc worker-1162" 0x00007f687f3fead3 in ?? ()
82 LWP 11624 "rpc worker-1162" 0x00007f687f3fead3 in ?? ()
83 LWP 11625 "rpc worker-1162" 0x00007f687f3fead3 in ?? ()
84 LWP 11626 "rpc worker-1162" 0x00007f687f3fead3 in ?? ()
85 LWP 11627 "rpc worker-1162" 0x00007f687f3fead3 in ?? ()
86 LWP 11628 "rpc worker-1162" 0x00007f687f3fead3 in ?? ()
87 LWP 11629 "rpc worker-1162" 0x00007f687f3fead3 in ?? ()
88 LWP 11630 "rpc worker-1163" 0x00007f687f3fead3 in ?? ()
89 LWP 11631 "rpc worker-1163" 0x00007f687f3fead3 in ?? ()
90 LWP 11632 "rpc worker-1163" 0x00007f687f3fead3 in ?? ()
91 LWP 11633 "rpc worker-1163" 0x00007f687f3fead3 in ?? ()
92 LWP 11634 "rpc worker-1163" 0x00007f687f3fead3 in ?? ()
93 LWP 11635 "rpc worker-1163" 0x00007f687f3fead3 in ?? ()
94 LWP 11636 "rpc worker-1163" 0x00007f687f3fead3 in ?? ()
95 LWP 11637 "rpc worker-1163" 0x00007f687f3fead3 in ?? ()
96 LWP 11638 "rpc worker-1163" 0x00007f687f3fead3 in ?? ()
97 LWP 11639 "rpc worker-1163" 0x00007f687f3fead3 in ?? ()
98 LWP 11640 "rpc worker-1164" 0x00007f687f3fead3 in ?? ()
99 LWP 11641 "rpc worker-1164" 0x00007f687f3fead3 in ?? ()
100 LWP 11642 "rpc worker-1164" 0x00007f687f3fead3 in ?? ()
101 LWP 11643 "rpc worker-1164" 0x00007f687f3fead3 in ?? ()
102 LWP 11644 "rpc worker-1164" 0x00007f687f3fead3 in ?? ()
103 LWP 11645 "rpc worker-1164" 0x00007f687f3fead3 in ?? ()
104 LWP 11646 "rpc worker-1164" 0x00007f687f3fead3 in ?? ()
105 LWP 11647 "rpc worker-1164" 0x00007f687f3fead3 in ?? ()
106 LWP 11648 "rpc worker-1164" 0x00007f687f3fead3 in ?? ()
107 LWP 11649 "rpc worker-1164" 0x00007f687f3fead3 in ?? ()
108 LWP 11650 "rpc worker-1165" 0x00007f687f3fead3 in ?? ()
109 LWP 11651 "rpc worker-1165" 0x00007f687f3fead3 in ?? ()
110 LWP 11652 "rpc worker-1165" 0x00007f687f3fead3 in ?? ()
111 LWP 11653 "rpc worker-1165" 0x00007f687f3fead3 in ?? ()
112 LWP 11654 "rpc worker-1165" 0x00007f687f3fead3 in ?? ()
113 LWP 11655 "rpc worker-1165" 0x00007f687f3fead3 in ?? ()
114 LWP 11656 "rpc worker-1165" 0x00007f687f3fead3 in ?? ()
115 LWP 11657 "rpc worker-1165" 0x00007f687f3fead3 in ?? ()
116 LWP 11658 "rpc worker-1165" 0x00007f687f3fead3 in ?? ()
117 LWP 11659 "diag-logger-116" 0x00007f687f3fefb9 in ?? ()
118 LWP 11660 "result-tracker-" 0x00007f687f3fefb9 in ?? ()
119 LWP 11661 "excess-log-dele" 0x00007f687f3fefb9 in ?? ()
120 LWP 11662 "tcmalloc-memory" 0x00007f687f3fefb9 in ?? ()
121 LWP 11663 "acceptor-11663" 0x00007f687c755fc7 in ?? ()
122 LWP 11664 "heartbeat-11664" 0x00007f687f3fefb9 in ?? ()
123 LWP 11665 "maintenance_sch" 0x00007f687f3fefb9 in ?? ()
124 LWP 12100 "wal-append [wor" 0x00007f687f3fefb9 in ?? ()
Thread 124 (LWP 12100):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f683505a0f0 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x000055b205ba8134 in ?? ()
#5 0x00007f683505a110 in ?? ()
#6 0x0000000000000003 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f683505a130 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007f683505a1c0 in ?? ()
#12 0x00007f683505a260 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 123 (LWP 11665):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f683585b330 in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000055b20591de60 in ?? ()
#5 0x00007f683585b350 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 11664):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f683605c310 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x000055b20586d944 in ?? ()
#5 0x00007f683605c330 in ?? ()
#6 0x0000000000000007 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f683605c350 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007f683605c3e0 in ?? ()
#12 0x00007f683605c4b8 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 121 (LWP 11663):
#0 0x00007f687c755fc7 in ?? ()
#1 0x00007f683685d128 in ?? ()
#2 0x00007f683685d12c in ?? ()
#3 0x00007f683685d130 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 120 (LWP 11662):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f683705e4f0 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffc9160b250 in ?? ()
#5 0x00007f683705e510 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 11661):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f683785f480 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 11660):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f68380604e0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000055b205828470 in ?? ()
#5 0x00007f6838060500 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 11659):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f6838861450 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000055b205ab82e0 in ?? ()
#5 0x00007f6838861470 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 116 (LWP 11658):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 11657):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 11656):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 11655):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 11654):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 11653):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 11652):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 11651):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 11650):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 11649):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 11648):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 11647):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 11646):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 11645):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 11644):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 11643):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 11642):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 11641):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 11640):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 11639):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 11638):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 11637):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 11636):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 11635):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 11634):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 11633):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 11632):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 11631):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 11630):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 11629):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 11628):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 11627):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 11626):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 11625):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 11624):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 11623):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 11622):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 11621):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 11620):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 11619):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 11618):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000055b205a80318 in ?? ()
#4 0x00007f684d08a430 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f684d08a450 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 11617):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 11616):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 11615):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 11614):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 11613):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 11612):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 11611):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 11610):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 11609):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 11608):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 11607):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 11606):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 11605):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 11604):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 11603):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 11602):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 11601):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 11600):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 11599):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 11598):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000055b205a7700c in ?? ()
#4 0x00007f685709e430 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f685709e450 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000055b205a71c30 in ?? ()
#9 0x00007f687f3fe770 in ?? ()
#10 0x00007f685709e450 in ?? ()
#11 0x0000000100000000 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 11597):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 11596):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 11595):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 11594):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 11593):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 11592):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 11591):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 11590):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 11589):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 11588):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 11587):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 11586):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 11585):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 11584):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 11583):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 11582):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 11581):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 11580):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 11579):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 11578):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 11577):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 11576):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 11575):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 11574):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 11573):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 11572):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 11571):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 11570):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 11569):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 11568):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 11567):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 11566):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 11565):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 11564):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 11563):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 11562):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 11561):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 11560):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 11559):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 11558):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f686b0c6390 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 15 (LWP 11557):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f686b8c7540 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 14 (LWP 11556):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f686c0c80c0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 13 (LWP 11555):
#0 0x00007f687f3fead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 11554):
#0 0x00007f687c754947 in ?? ()
#1 0x00007f686d0ca580 in ?? ()
#2 0x00007f687fa4922b in ?? ()
#3 0x00007f686d0ca580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 11 (LWP 11553):
#0 0x00007f687c754947 in ?? ()
#1 0x00007f686d8cb580 in ?? ()
#2 0x00007f687fa4922b in ?? ()
#3 0x00007f686d8cb580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 10 (LWP 11552):
#0 0x00007f687c754947 in ?? ()
#1 0x00007f686e0cc580 in ?? ()
#2 0x00007f687fa4922b in ?? ()
#3 0x00007f686e0cc580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 9 (LWP 11551):
#0 0x00007f687c754947 in ?? ()
#1 0x00007f686fcb0580 in ?? ()
#2 0x00007f687fa4922b in ?? ()
#3 0x00007f686fcb0580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 8 (LWP 11548):
#0 0x00007f687c747bb9 in ?? ()
#1 0x00007f68714b3800 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 7 (LWP 11547):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f6870cb24e0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 11546):
#0 0x00007f687f4029e2 in ?? ()
#1 0x00007f68704b1330 in ?? ()
#2 0x00007f68704b1380 in ?? ()
#3 0x00007f68704b1350 in ?? ()
#4 0x0000000000000001 in ?? ()
#5 0x00007f68704b16c0 in ?? ()
#6 0x000055b20591f890 in ?? ()
#7 0x000055b205785b10 in ?? ()
#8 0x000055b205785b00 in ?? ()
#9 0x00007ffc91609800 in ?? ()
#10 0x00007f687ff58dd2 in ?? ()
#11 0x000055b2058cb5d0 in ?? ()
#12 0x000055b205828c30 in ?? ()
#13 0x00007f68704b1390 in ?? ()
#14 0x00007f687e86eee0 in ?? ()
#15 0x00007f6800000010 in ?? ()
#16 0x00007f687e9a6b5f in ?? ()
#17 0x00007f68704b13b0 in ?? ()
#18 0x000055b2058101e0 in ?? ()
#19 0x00007f687e86eee0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 5 (LWP 11540):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f68724b52b0 in ?? ()
#2 0x000000000000000a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000055b205785d08 in ?? ()
#5 0x00007f68724b52d0 in ?? ()
#6 0x0000000000000014 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 11539):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f6872cb65c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000055b2058ce438 in ?? ()
#5 0x00007f6872cb65e0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 3 (LWP 11538):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f68734b75c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000055b2058ce2f8 in ?? ()
#5 0x00007f68734b75e0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (LWP 11537):
#0 0x00007f687f3fefb9 in ?? ()
#1 0x00007f6873cb85c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000055b2058ce1b8 in ?? ()
#5 0x00007f6873cb85e0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 11536):
#0 0x00007f687f402d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20260928 01:57:30.470402 9548 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID da431f08920849cea6adbc26a62e7824 and pid 11667
************************ BEGIN STACKS **************************
[New LWP 11668]
[New LWP 11669]
[New LWP 11670]
[New LWP 11671]
[New LWP 11677]
[New LWP 11678]
[New LWP 11679]
[New LWP 11682]
[New LWP 11683]
[New LWP 11684]
[New LWP 11685]
[New LWP 11686]
[New LWP 11687]
[New LWP 11688]
[New LWP 11689]
[New LWP 11690]
[New LWP 11691]
[New LWP 11692]
[New LWP 11693]
[New LWP 11694]
[New LWP 11695]
[New LWP 11696]
[New LWP 11697]
[New LWP 11698]
[New LWP 11699]
[New LWP 11700]
[New LWP 11701]
[New LWP 11702]
[New LWP 11703]
[New LWP 11704]
[New LWP 11705]
[New LWP 11706]
[New LWP 11707]
[New LWP 11708]
[New LWP 11709]
[New LWP 11710]
[New LWP 11711]
[New LWP 11712]
[New LWP 11713]
[New LWP 11714]
[New LWP 11715]
[New LWP 11716]
[New LWP 11717]
[New LWP 11718]
[New LWP 11719]
[New LWP 11720]
[New LWP 11721]
[New LWP 11722]
[New LWP 11723]
[New LWP 11724]
[New LWP 11725]
[New LWP 11726]
[New LWP 11727]
[New LWP 11728]
[New LWP 11729]
[New LWP 11730]
[New LWP 11731]
[New LWP 11732]
[New LWP 11733]
[New LWP 11734]
[New LWP 11735]
[New LWP 11736]
[New LWP 11737]
[New LWP 11738]
[New LWP 11739]
[New LWP 11740]
[New LWP 11741]
[New LWP 11742]
[New LWP 11743]
[New LWP 11744]
[New LWP 11745]
[New LWP 11746]
[New LWP 11747]
[New LWP 11748]
[New LWP 11749]
[New LWP 11750]
[New LWP 11751]
[New LWP 11752]
[New LWP 11753]
[New LWP 11754]
[New LWP 11755]
[New LWP 11756]
[New LWP 11757]
[New LWP 11758]
[New LWP 11759]
[New LWP 11760]
[New LWP 11761]
[New LWP 11762]
[New LWP 11763]
[New LWP 11764]
[New LWP 11765]
[New LWP 11766]
[New LWP 11767]
[New LWP 11768]
[New LWP 11769]
[New LWP 11770]
[New LWP 11771]
[New LWP 11772]
[New LWP 11773]
[New LWP 11774]
[New LWP 11775]
[New LWP 11776]
[New LWP 11777]
[New LWP 11778]
[New LWP 11779]
[New LWP 11780]
[New LWP 11781]
[New LWP 11782]
[New LWP 11783]
[New LWP 11784]
[New LWP 11785]
[New LWP 11786]
[New LWP 11787]
[New LWP 11788]
[New LWP 11789]
[New LWP 11790]
[New LWP 11791]
[New LWP 11792]
[New LWP 11793]
[New LWP 11794]
[New LWP 11795]
[New LWP 11796]
0x00007f25d4d2cd50 in ?? ()
Id Target Id Frame
* 1 LWP 11667 "kudu" 0x00007f25d4d2cd50 in ?? ()
2 LWP 11668 "kudu" 0x00007f25d4d28fb9 in ?? ()
3 LWP 11669 "kudu" 0x00007f25d4d28fb9 in ?? ()
4 LWP 11670 "kudu" 0x00007f25d4d28fb9 in ?? ()
5 LWP 11671 "kernel-watcher-" 0x00007f25d4d28fb9 in ?? ()
6 LWP 11677 "ntp client-1167" 0x00007f25d4d2c9e2 in ?? ()
7 LWP 11678 "file cache-evic" 0x00007f25d4d28fb9 in ?? ()
8 LWP 11679 "sq_acceptor" 0x00007f25d2071bb9 in ?? ()
9 LWP 11682 "rpc reactor-116" 0x00007f25d207e947 in ?? ()
10 LWP 11683 "rpc reactor-116" 0x00007f25d207e947 in ?? ()
11 LWP 11684 "rpc reactor-116" 0x00007f25d207e947 in ?? ()
12 LWP 11685 "rpc reactor-116" 0x00007f25d207e947 in ?? ()
13 LWP 11686 "MaintenanceMgr " 0x00007f25d4d28ad3 in ?? ()
14 LWP 11687 "txn-status-mana" 0x00007f25d4d28fb9 in ?? ()
15 LWP 11688 "collect_and_rem" 0x00007f25d4d28fb9 in ?? ()
16 LWP 11689 "tc-session-exp-" 0x00007f25d4d28fb9 in ?? ()
17 LWP 11690 "rpc worker-1169" 0x00007f25d4d28ad3 in ?? ()
18 LWP 11691 "rpc worker-1169" 0x00007f25d4d28ad3 in ?? ()
19 LWP 11692 "rpc worker-1169" 0x00007f25d4d28ad3 in ?? ()
20 LWP 11693 "rpc worker-1169" 0x00007f25d4d28ad3 in ?? ()
21 LWP 11694 "rpc worker-1169" 0x00007f25d4d28ad3 in ?? ()
22 LWP 11695 "rpc worker-1169" 0x00007f25d4d28ad3 in ?? ()
23 LWP 11696 "rpc worker-1169" 0x00007f25d4d28ad3 in ?? ()
24 LWP 11697 "rpc worker-1169" 0x00007f25d4d28ad3 in ?? ()
25 LWP 11698 "rpc worker-1169" 0x00007f25d4d28ad3 in ?? ()
26 LWP 11699 "rpc worker-1169" 0x00007f25d4d28ad3 in ?? ()
27 LWP 11700 "rpc worker-1170" 0x00007f25d4d28ad3 in ?? ()
28 LWP 11701 "rpc worker-1170" 0x00007f25d4d28ad3 in ?? ()
29 LWP 11702 "rpc worker-1170" 0x00007f25d4d28ad3 in ?? ()
30 LWP 11703 "rpc worker-1170" 0x00007f25d4d28ad3 in ?? ()
31 LWP 11704 "rpc worker-1170" 0x00007f25d4d28ad3 in ?? ()
32 LWP 11705 "rpc worker-1170" 0x00007f25d4d28ad3 in ?? ()
33 LWP 11706 "rpc worker-1170" 0x00007f25d4d28ad3 in ?? ()
34 LWP 11707 "rpc worker-1170" 0x00007f25d4d28ad3 in ?? ()
35 LWP 11708 "rpc worker-1170" 0x00007f25d4d28ad3 in ?? ()
36 LWP 11709 "rpc worker-1170" 0x00007f25d4d28ad3 in ?? ()
37 LWP 11710 "rpc worker-1171" 0x00007f25d4d28ad3 in ?? ()
38 LWP 11711 "rpc worker-1171" 0x00007f25d4d28ad3 in ?? ()
39 LWP 11712 "rpc worker-1171" 0x00007f25d4d28ad3 in ?? ()
40 LWP 11713 "rpc worker-1171" 0x00007f25d4d28ad3 in ?? ()
41 LWP 11714 "rpc worker-1171" 0x00007f25d4d28ad3 in ?? ()
42 LWP 11715 "rpc worker-1171" 0x00007f25d4d28ad3 in ?? ()
43 LWP 11716 "rpc worker-1171" 0x00007f25d4d28ad3 in ?? ()
44 LWP 11717 "rpc worker-1171" 0x00007f25d4d28ad3 in ?? ()
45 LWP 11718 "rpc worker-1171" 0x00007f25d4d28ad3 in ?? ()
46 LWP 11719 "rpc worker-1171" 0x00007f25d4d28ad3 in ?? ()
47 LWP 11720 "rpc worker-1172" 0x00007f25d4d28ad3 in ?? ()
48 LWP 11721 "rpc worker-1172" 0x00007f25d4d28ad3 in ?? ()
49 LWP 11722 "rpc worker-1172" 0x00007f25d4d28ad3 in ?? ()
50 LWP 11723 "rpc worker-1172" 0x00007f25d4d28ad3 in ?? ()
51 LWP 11724 "rpc worker-1172" 0x00007f25d4d28ad3 in ?? ()
52 LWP 11725 "rpc worker-1172" 0x00007f25d4d28ad3 in ?? ()
53 LWP 11726 "rpc worker-1172" 0x00007f25d4d28ad3 in ?? ()
54 LWP 11727 "rpc worker-1172" 0x00007f25d4d28ad3 in ?? ()
55 LWP 11728 "rpc worker-1172" 0x00007f25d4d28ad3 in ?? ()
56 LWP 11729 "rpc worker-1172" 0x00007f25d4d28ad3 in ?? ()
57 LWP 11730 "rpc worker-1173" 0x00007f25d4d28ad3 in ?? ()
58 LWP 11731 "rpc worker-1173" 0x00007f25d4d28ad3 in ?? ()
59 LWP 11732 "rpc worker-1173" 0x00007f25d4d28ad3 in ?? ()
60 LWP 11733 "rpc worker-1173" 0x00007f25d4d28ad3 in ?? ()
61 LWP 11734 "rpc worker-1173" 0x00007f25d4d28ad3 in ?? ()
62 LWP 11735 "rpc worker-1173" 0x00007f25d4d28ad3 in ?? ()
63 LWP 11736 "rpc worker-1173" 0x00007f25d4d28ad3 in ?? ()
64 LWP 11737 "rpc worker-1173" 0x00007f25d4d28ad3 in ?? ()
65 LWP 11738 "rpc worker-1173" 0x00007f25d4d28ad3 in ?? ()
66 LWP 11739 "rpc worker-1173" 0x00007f25d4d28ad3 in ?? ()
67 LWP 11740 "rpc worker-1174" 0x00007f25d4d28ad3 in ?? ()
68 LWP 11741 "rpc worker-1174" 0x00007f25d4d28ad3 in ?? ()
69 LWP 11742 "rpc worker-1174" 0x00007f25d4d28ad3 in ?? ()
70 LWP 11743 "rpc worker-1174" 0x00007f25d4d28ad3 in ?? ()
71 LWP 11744 "rpc worker-1174" 0x00007f25d4d28ad3 in ?? ()
72 LWP 11745 "rpc worker-1174" 0x00007f25d4d28ad3 in ?? ()
73 LWP 11746 "rpc worker-1174" 0x00007f25d4d28ad3 in ?? ()
74 LWP 11747 "rpc worker-1174" 0x00007f25d4d28ad3 in ?? ()
75 LWP 11748 "rpc worker-1174" 0x00007f25d4d28ad3 in ?? ()
76 LWP 11749 "rpc worker-1174" 0x00007f25d4d28ad3 in ?? ()
77 LWP 11750 "rpc worker-1175" 0x00007f25d4d28ad3 in ?? ()
78 LWP 11751 "rpc worker-1175" 0x00007f25d4d28ad3 in ?? ()
79 LWP 11752 "rpc worker-1175" 0x00007f25d4d28ad3 in ?? ()
80 LWP 11753 "rpc worker-1175" 0x00007f25d4d28ad3 in ?? ()
81 LWP 11754 "rpc worker-1175" 0x00007f25d4d28ad3 in ?? ()
82 LWP 11755 "rpc worker-1175" 0x00007f25d4d28ad3 in ?? ()
83 LWP 11756 "rpc worker-1175" 0x00007f25d4d28ad3 in ?? ()
84 LWP 11757 "rpc worker-1175" 0x00007f25d4d28ad3 in ?? ()
85 LWP 11758 "rpc worker-1175" 0x00007f25d4d28ad3 in ?? ()
86 LWP 11759 "rpc worker-1175" 0x00007f25d4d28ad3 in ?? ()
87 LWP 11760 "rpc worker-1176" 0x00007f25d4d28ad3 in ?? ()
88 LWP 11761 "rpc worker-1176" 0x00007f25d4d28ad3 in ?? ()
89 LWP 11762 "rpc worker-1176" 0x00007f25d4d28ad3 in ?? ()
90 LWP 11763 "rpc worker-1176" 0x00007f25d4d28ad3 in ?? ()
91 LWP 11764 "rpc worker-1176" 0x00007f25d4d28ad3 in ?? ()
92 LWP 11765 "rpc worker-1176" 0x00007f25d4d28ad3 in ?? ()
93 LWP 11766 "rpc worker-1176" 0x00007f25d4d28ad3 in ?? ()
94 LWP 11767 "rpc worker-1176" 0x00007f25d4d28ad3 in ?? ()
95 LWP 11768 "rpc worker-1176" 0x00007f25d4d28ad3 in ?? ()
96 LWP 11769 "rpc worker-1176" 0x00007f25d4d28ad3 in ?? ()
97 LWP 11770 "rpc worker-1177" 0x00007f25d4d28ad3 in ?? ()
98 LWP 11771 "rpc worker-1177" 0x00007f25d4d28ad3 in ?? ()
99 LWP 11772 "rpc worker-1177" 0x00007f25d4d28ad3 in ?? ()
100 LWP 11773 "rpc worker-1177" 0x00007f25d4d28ad3 in ?? ()
101 LWP 11774 "rpc worker-1177" 0x00007f25d4d28ad3 in ?? ()
102 LWP 11775 "rpc worker-1177" 0x00007f25d4d28ad3 in ?? ()
103 LWP 11776 "rpc worker-1177" 0x00007f25d4d28ad3 in ?? ()
104 LWP 11777 "rpc worker-1177" 0x00007f25d4d28ad3 in ?? ()
105 LWP 11778 "rpc worker-1177" 0x00007f25d4d28ad3 in ?? ()
106 LWP 11779 "rpc worker-1177" 0x00007f25d4d28ad3 in ?? ()
107 LWP 11780 "rpc worker-1178" 0x00007f25d4d28ad3 in ?? ()
108 LWP 11781 "rpc worker-1178" 0x00007f25d4d28ad3 in ?? ()
109 LWP 11782 "rpc worker-1178" 0x00007f25d4d28ad3 in ?? ()
110 LWP 11783 "rpc worker-1178" 0x00007f25d4d28ad3 in ?? ()
111 LWP 11784 "rpc worker-1178" 0x00007f25d4d28ad3 in ?? ()
112 LWP 11785 "rpc worker-1178" 0x00007f25d4d28ad3 in ?? ()
113 LWP 11786 "rpc worker-1178" 0x00007f25d4d28ad3 in ?? ()
114 LWP 11787 "rpc worker-1178" 0x00007f25d4d28ad3 in ?? ()
115 LWP 11788 "rpc worker-1178" 0x00007f25d4d28ad3 in ?? ()
116 LWP 11789 "rpc worker-1178" 0x00007f25d4d28ad3 in ?? ()
117 LWP 11790 "diag-logger-117" 0x00007f25d4d28fb9 in ?? ()
118 LWP 11791 "result-tracker-" 0x00007f25d4d28fb9 in ?? ()
119 LWP 11792 "excess-log-dele" 0x00007f25d4d28fb9 in ?? ()
120 LWP 11793 "tcmalloc-memory" 0x00007f25d4d28fb9 in ?? ()
121 LWP 11794 "acceptor-11794" 0x00007f25d207ffc7 in ?? ()
122 LWP 11795 "heartbeat-11795" 0x00007f25d4d28fb9 in ?? ()
123 LWP 11796 "maintenance_sch" 0x00007f25d4d28fb9 in ?? ()
Thread 123 (LWP 11796):
#0 0x00007f25d4d28fb9 in ?? ()
#1 0x00007f258af7e330 in ?? ()
#2 0x0000000000000009 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000056141d2bbe60 in ?? ()
#5 0x00007f258af7e350 in ?? ()
#6 0x0000000000000012 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 11795):
#0 0x00007f25d4d28fb9 in ?? ()
#1 0x00007f258b77f310 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000056141d20b940 in ?? ()
#5 0x00007f258b77f330 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 11794):
#0 0x00007f25d207ffc7 in ?? ()
#1 0x00007f258bf80128 in ?? ()
#2 0x00007f258bf8012c in ?? ()
#3 0x00007f258bf80130 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 120 (LWP 11793):
#0 0x00007f25d4d28fb9 in ?? ()
#1 0x00007f258c7814f0 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffca9df43e0 in ?? ()
#5 0x00007f258c781510 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 11792):
#0 0x00007f25d4d28fb9 in ?? ()
#1 0x00007f258cf82480 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 11791):
#0 0x00007f25d4d28fb9 in ?? ()
#1 0x00007f258d7834e0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000056141d1c6470 in ?? ()
#5 0x00007f258d783500 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 11790):
#0 0x00007f25d4d28fb9 in ?? ()
#1 0x00007f258df84450 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000056141d4562e0 in ?? ()
#5 0x00007f258df84470 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 116 (LWP 11789):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 11788):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 11787):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 11786):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 11785):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 11784):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 11783):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 11782):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 11781):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 11780):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 11779):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 11778):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 11777):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 11776):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 11775):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 11774):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 11773):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 11772):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 11771):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 11770):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 11769):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 11768):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 11767):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 11766):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 11765):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 11764):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 11763):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 11762):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 11761):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 11760):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 11759):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 11758):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 11757):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 11756):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 11755):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 11754):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 11753):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 11752):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 11751):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 11750):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 11749):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000056141d41e31c in ?? ()
#4 0x00007f25a27ad430 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f25a27ad450 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000056141d419330 in ?? ()
#9 0x00007f25d4d28770 in ?? ()
#10 0x00007f25a27ad450 in ?? ()
#11 0x0000000100000000 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 11748):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 11747):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 11746):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 11745):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 11744):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 11743):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 11742):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 11741):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 11740):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 11739):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 11738):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 11737):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 11736):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 11735):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 11734):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 11733):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 11732):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 11731):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 11730):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 11729):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 11728):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 11727):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 11726):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 11725):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 11724):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 11723):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 11722):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 11721):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 11720):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 11719):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 11718):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 11717):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 11716):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 11715):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 11714):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 11713):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 11712):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 11711):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 11710):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 11709):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 11708):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 11707):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 11706):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 11705):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 11704):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 11703):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 11702):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 11701):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 11700):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 11699):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 11698):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 11697):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 11696):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 11695):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 11694):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 11693):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 11692):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 11691):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 11690):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 11689):
#0 0x00007f25d4d28fb9 in ?? ()
#1 0x00007f25c07e9390 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 15 (LWP 11688):
#0 0x00007f25d4d28fb9 in ?? ()
#1 0x00007f25c0fea540 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 14 (LWP 11687):
#0 0x00007f25d4d28fb9 in ?? ()
#1 0x00007f25c17eb0c0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 13 (LWP 11686):
#0 0x00007f25d4d28ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 11685):
#0 0x00007f25d207e947 in ?? ()
#1 0x00007f25c27ed580 in ?? ()
#2 0x00007f25d537322b in ?? ()
#3 0x00007f25c27ed580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 11 (LWP 11684):
#0 0x00007f25d207e947 in ?? ()
#1 0x00007f25c2fee580 in ?? ()
#2 0x00007f25d537322b in ?? ()
#3 0x00007f25c2fee580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 10 (LWP 11683):
#0 0x00007f25d207e947 in ?? ()
#1 0x00007f25c37ef580 in ?? ()
#2 0x00007f25d537322b in ?? ()
#3 0x00007f25c37ef580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 9 (LWP 11682):
#0 0x00007f25d207e947 in ?? ()
#1 0x00007f25c55da580 in ?? ()
#2 0x00007f25d537322b in ?? ()
#3 0x00007f25c55da580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 8 (LWP 11679):
#0 0x00007f25d2071bb9 in ?? ()
#1 0x00007f25c6ddd800 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 7 (LWP 11678):
#0 0x00007f25d4d28fb9 in ?? ()
#1 0x00007f25c65dc4e0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 11677):
#0 0x00007f25d4d2c9e2 in ?? ()
#1 0x00007f25c5ddb330 in ?? ()
#2 0x00007f25c5ddb380 in ?? ()
#3 0x00007f25c5ddb350 in ?? ()
#4 0x0000000000000001 in ?? ()
#5 0x00007f25c5ddb6c0 in ?? ()
#6 0x000056141d2bd890 in ?? ()
#7 0x000056141d123b10 in ?? ()
#8 0x000056141d123b00 in ?? ()
#9 0x00007ffca9df2990 in ?? ()
#10 0x00007f25d5882dd2 in ?? ()
#11 0x000056141d22fd10 in ?? ()
#12 0x000056141d1c6c30 in ?? ()
#13 0x00007f25c5ddb390 in ?? ()
#14 0x00007f25d4198ee0 in ?? ()
#15 0x00007f2500000010 in ?? ()
#16 0x00007f25d42d0b5f in ?? ()
#17 0x00007f25c5ddb3b0 in ?? ()
#18 0x000056141d1ae1e0 in ?? ()
#19 0x00007f25d4198ee0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 5 (LWP 11671):
#0 0x00007f25d4d28fb9 in ?? ()
#1 0x00007f25c7ddf2b0 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000056141d123d08 in ?? ()
#5 0x00007f25c7ddf2d0 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 11670):
#0 0x00007f25d4d28fb9 in ?? ()
#1 0x00007f25c85e05c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000056141d26c438 in ?? ()
#5 0x00007f25c85e05e0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 3 (LWP 11669):
#0 0x00007f25d4d28fb9 in ?? ()
#1 0x00007f25c8de15c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000056141d26c2f8 in ?? ()
#5 0x00007f25c8de15e0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (LWP 11668):
#0 0x00007f25d4d28fb9 in ?? ()
#1 0x00007f25c95e25c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000056141d26c1b8 in ?? ()
#5 0x00007f25c95e25e0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 11667):
#0 0x00007f25d4d2cd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20260928 01:57:31.073663 9548 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID 1f5e82d9d2954a0d85575b1c67b29a9d and pid 11798
************************ BEGIN STACKS **************************
[New LWP 11799]
[New LWP 11800]
[New LWP 11801]
[New LWP 11802]
[New LWP 11808]
[New LWP 11809]
[New LWP 11810]
[New LWP 11813]
[New LWP 11814]
[New LWP 11815]
[New LWP 11816]
[New LWP 11817]
[New LWP 11818]
[New LWP 11819]
[New LWP 11820]
[New LWP 11821]
[New LWP 11822]
[New LWP 11823]
[New LWP 11824]
[New LWP 11825]
[New LWP 11826]
[New LWP 11827]
[New LWP 11828]
[New LWP 11829]
[New LWP 11830]
[New LWP 11831]
[New LWP 11832]
[New LWP 11833]
[New LWP 11834]
[New LWP 11835]
[New LWP 11836]
[New LWP 11837]
[New LWP 11838]
[New LWP 11839]
[New LWP 11840]
[New LWP 11841]
[New LWP 11842]
[New LWP 11843]
[New LWP 11844]
[New LWP 11845]
[New LWP 11846]
[New LWP 11847]
[New LWP 11848]
[New LWP 11849]
[New LWP 11850]
[New LWP 11851]
[New LWP 11852]
[New LWP 11853]
[New LWP 11854]
[New LWP 11855]
[New LWP 11856]
[New LWP 11857]
[New LWP 11858]
[New LWP 11859]
[New LWP 11860]
[New LWP 11861]
[New LWP 11862]
[New LWP 11863]
[New LWP 11864]
[New LWP 11865]
[New LWP 11866]
[New LWP 11867]
[New LWP 11868]
[New LWP 11869]
[New LWP 11870]
[New LWP 11871]
[New LWP 11872]
[New LWP 11873]
[New LWP 11874]
[New LWP 11875]
[New LWP 11876]
[New LWP 11877]
[New LWP 11878]
[New LWP 11879]
[New LWP 11880]
[New LWP 11881]
[New LWP 11882]
[New LWP 11883]
[New LWP 11884]
[New LWP 11885]
[New LWP 11886]
[New LWP 11887]
[New LWP 11888]
[New LWP 11889]
[New LWP 11890]
[New LWP 11891]
[New LWP 11892]
[New LWP 11893]
[New LWP 11894]
[New LWP 11895]
[New LWP 11896]
[New LWP 11897]
[New LWP 11898]
[New LWP 11899]
[New LWP 11900]
[New LWP 11901]
[New LWP 11902]
[New LWP 11903]
[New LWP 11904]
[New LWP 11905]
[New LWP 11906]
[New LWP 11907]
[New LWP 11908]
[New LWP 11909]
[New LWP 11910]
[New LWP 11911]
[New LWP 11912]
[New LWP 11913]
[New LWP 11914]
[New LWP 11915]
[New LWP 11916]
[New LWP 11917]
[New LWP 11918]
[New LWP 11919]
[New LWP 11920]
[New LWP 11921]
[New LWP 11922]
[New LWP 11923]
[New LWP 11924]
[New LWP 11925]
[New LWP 11926]
[New LWP 11927]
[New LWP 12098]
0x00007fd812404d50 in ?? ()
Id Target Id Frame
* 1 LWP 11798 "kudu" 0x00007fd812404d50 in ?? ()
2 LWP 11799 "kudu" 0x00007fd812400fb9 in ?? ()
3 LWP 11800 "kudu" 0x00007fd812400fb9 in ?? ()
4 LWP 11801 "kudu" 0x00007fd812400fb9 in ?? ()
5 LWP 11802 "kernel-watcher-" 0x00007fd812400fb9 in ?? ()
6 LWP 11808 "ntp client-1180" 0x00007fd8124049e2 in ?? ()
7 LWP 11809 "file cache-evic" 0x00007fd812400fb9 in ?? ()
8 LWP 11810 "sq_acceptor" 0x00007fd80f749bb9 in ?? ()
9 LWP 11813 "rpc reactor-118" 0x00007fd80f756947 in ?? ()
10 LWP 11814 "rpc reactor-118" 0x00007fd80f756947 in ?? ()
11 LWP 11815 "rpc reactor-118" 0x00007fd80f756947 in ?? ()
12 LWP 11816 "rpc reactor-118" 0x00007fd80f756947 in ?? ()
13 LWP 11817 "MaintenanceMgr " 0x00007fd812400ad3 in ?? ()
14 LWP 11818 "txn-status-mana" 0x00007fd812400fb9 in ?? ()
15 LWP 11819 "collect_and_rem" 0x00007fd812400fb9 in ?? ()
16 LWP 11820 "tc-session-exp-" 0x00007fd812400fb9 in ?? ()
17 LWP 11821 "rpc worker-1182" 0x00007fd812400ad3 in ?? ()
18 LWP 11822 "rpc worker-1182" 0x00007fd812400ad3 in ?? ()
19 LWP 11823 "rpc worker-1182" 0x00007fd812400ad3 in ?? ()
20 LWP 11824 "rpc worker-1182" 0x00007fd812400ad3 in ?? ()
21 LWP 11825 "rpc worker-1182" 0x00007fd812400ad3 in ?? ()
22 LWP 11826 "rpc worker-1182" 0x00007fd812400ad3 in ?? ()
23 LWP 11827 "rpc worker-1182" 0x00007fd812400ad3 in ?? ()
24 LWP 11828 "rpc worker-1182" 0x00007fd812400ad3 in ?? ()
25 LWP 11829 "rpc worker-1182" 0x00007fd812400ad3 in ?? ()
26 LWP 11830 "rpc worker-1183" 0x00007fd812400ad3 in ?? ()
27 LWP 11831 "rpc worker-1183" 0x00007fd812400ad3 in ?? ()
28 LWP 11832 "rpc worker-1183" 0x00007fd812400ad3 in ?? ()
29 LWP 11833 "rpc worker-1183" 0x00007fd812400ad3 in ?? ()
30 LWP 11834 "rpc worker-1183" 0x00007fd812400ad3 in ?? ()
31 LWP 11835 "rpc worker-1183" 0x00007fd812400ad3 in ?? ()
32 LWP 11836 "rpc worker-1183" 0x00007fd812400ad3 in ?? ()
33 LWP 11837 "rpc worker-1183" 0x00007fd812400ad3 in ?? ()
34 LWP 11838 "rpc worker-1183" 0x00007fd812400ad3 in ?? ()
35 LWP 11839 "rpc worker-1183" 0x00007fd812400ad3 in ?? ()
36 LWP 11840 "rpc worker-1184" 0x00007fd812400ad3 in ?? ()
37 LWP 11841 "rpc worker-1184" 0x00007fd812400ad3 in ?? ()
38 LWP 11842 "rpc worker-1184" 0x00007fd812400ad3 in ?? ()
39 LWP 11843 "rpc worker-1184" 0x00007fd812400ad3 in ?? ()
40 LWP 11844 "rpc worker-1184" 0x00007fd812400ad3 in ?? ()
41 LWP 11845 "rpc worker-1184" 0x00007fd812400ad3 in ?? ()
42 LWP 11846 "rpc worker-1184" 0x00007fd812400ad3 in ?? ()
43 LWP 11847 "rpc worker-1184" 0x00007fd812400ad3 in ?? ()
44 LWP 11848 "rpc worker-1184" 0x00007fd812400ad3 in ?? ()
45 LWP 11849 "rpc worker-1184" 0x00007fd812400ad3 in ?? ()
46 LWP 11850 "rpc worker-1185" 0x00007fd812400ad3 in ?? ()
47 LWP 11851 "rpc worker-1185" 0x00007fd812400ad3 in ?? ()
48 LWP 11852 "rpc worker-1185" 0x00007fd812400ad3 in ?? ()
49 LWP 11853 "rpc worker-1185" 0x00007fd812400ad3 in ?? ()
50 LWP 11854 "rpc worker-1185" 0x00007fd812400ad3 in ?? ()
51 LWP 11855 "rpc worker-1185" 0x00007fd812400ad3 in ?? ()
52 LWP 11856 "rpc worker-1185" 0x00007fd812400ad3 in ?? ()
53 LWP 11857 "rpc worker-1185" 0x00007fd812400ad3 in ?? ()
54 LWP 11858 "rpc worker-1185" 0x00007fd812400ad3 in ?? ()
55 LWP 11859 "rpc worker-1185" 0x00007fd812400ad3 in ?? ()
56 LWP 11860 "rpc worker-1186" 0x00007fd812400ad3 in ?? ()
57 LWP 11861 "rpc worker-1186" 0x00007fd812400ad3 in ?? ()
58 LWP 11862 "rpc worker-1186" 0x00007fd812400ad3 in ?? ()
59 LWP 11863 "rpc worker-1186" 0x00007fd812400ad3 in ?? ()
60 LWP 11864 "rpc worker-1186" 0x00007fd812400ad3 in ?? ()
61 LWP 11865 "rpc worker-1186" 0x00007fd812400ad3 in ?? ()
62 LWP 11866 "rpc worker-1186" 0x00007fd812400ad3 in ?? ()
63 LWP 11867 "rpc worker-1186" 0x00007fd812400ad3 in ?? ()
64 LWP 11868 "rpc worker-1186" 0x00007fd812400ad3 in ?? ()
65 LWP 11869 "rpc worker-1186" 0x00007fd812400ad3 in ?? ()
66 LWP 11870 "rpc worker-1187" 0x00007fd812400ad3 in ?? ()
67 LWP 11871 "rpc worker-1187" 0x00007fd812400ad3 in ?? ()
68 LWP 11872 "rpc worker-1187" 0x00007fd812400ad3 in ?? ()
69 LWP 11873 "rpc worker-1187" 0x00007fd812400ad3 in ?? ()
70 LWP 11874 "rpc worker-1187" 0x00007fd812400ad3 in ?? ()
71 LWP 11875 "rpc worker-1187" 0x00007fd812400ad3 in ?? ()
72 LWP 11876 "rpc worker-1187" 0x00007fd812400ad3 in ?? ()
73 LWP 11877 "rpc worker-1187" 0x00007fd812400ad3 in ?? ()
74 LWP 11878 "rpc worker-1187" 0x00007fd812400ad3 in ?? ()
75 LWP 11879 "rpc worker-1187" 0x00007fd812400ad3 in ?? ()
76 LWP 11880 "rpc worker-1188" 0x00007fd812400ad3 in ?? ()
77 LWP 11881 "rpc worker-1188" 0x00007fd812400ad3 in ?? ()
78 LWP 11882 "rpc worker-1188" 0x00007fd812400ad3 in ?? ()
79 LWP 11883 "rpc worker-1188" 0x00007fd812400ad3 in ?? ()
80 LWP 11884 "rpc worker-1188" 0x00007fd812400ad3 in ?? ()
81 LWP 11885 "rpc worker-1188" 0x00007fd812400ad3 in ?? ()
82 LWP 11886 "rpc worker-1188" 0x00007fd812400ad3 in ?? ()
83 LWP 11887 "rpc worker-1188" 0x00007fd812400ad3 in ?? ()
84 LWP 11888 "rpc worker-1188" 0x00007fd812400ad3 in ?? ()
85 LWP 11889 "rpc worker-1188" 0x00007fd812400ad3 in ?? ()
86 LWP 11890 "rpc worker-1189" 0x00007fd812400ad3 in ?? ()
87 LWP 11891 "rpc worker-1189" 0x00007fd812400ad3 in ?? ()
88 LWP 11892 "rpc worker-1189" 0x00007fd812400ad3 in ?? ()
89 LWP 11893 "rpc worker-1189" 0x00007fd812400ad3 in ?? ()
90 LWP 11894 "rpc worker-1189" 0x00007fd812400ad3 in ?? ()
91 LWP 11895 "rpc worker-1189" 0x00007fd812400ad3 in ?? ()
92 LWP 11896 "rpc worker-1189" 0x00007fd812400ad3 in ?? ()
93 LWP 11897 "rpc worker-1189" 0x00007fd812400ad3 in ?? ()
94 LWP 11898 "rpc worker-1189" 0x00007fd812400ad3 in ?? ()
95 LWP 11899 "rpc worker-1189" 0x00007fd812400ad3 in ?? ()
96 LWP 11900 "rpc worker-1190" 0x00007fd812400ad3 in ?? ()
97 LWP 11901 "rpc worker-1190" 0x00007fd812400ad3 in ?? ()
98 LWP 11902 "rpc worker-1190" 0x00007fd812400ad3 in ?? ()
99 LWP 11903 "rpc worker-1190" 0x00007fd812400ad3 in ?? ()
100 LWP 11904 "rpc worker-1190" 0x00007fd812400ad3 in ?? ()
101 LWP 11905 "rpc worker-1190" 0x00007fd812400ad3 in ?? ()
102 LWP 11906 "rpc worker-1190" 0x00007fd812400ad3 in ?? ()
103 LWP 11907 "rpc worker-1190" 0x00007fd812400ad3 in ?? ()
104 LWP 11908 "rpc worker-1190" 0x00007fd812400ad3 in ?? ()
105 LWP 11909 "rpc worker-1190" 0x00007fd812400ad3 in ?? ()
106 LWP 11910 "rpc worker-1191" 0x00007fd812400ad3 in ?? ()
107 LWP 11911 "rpc worker-1191" 0x00007fd812400ad3 in ?? ()
108 LWP 11912 "rpc worker-1191" 0x00007fd812400ad3 in ?? ()
109 LWP 11913 "rpc worker-1191" 0x00007fd812400ad3 in ?? ()
110 LWP 11914 "rpc worker-1191" 0x00007fd812400ad3 in ?? ()
111 LWP 11915 "rpc worker-1191" 0x00007fd812400ad3 in ?? ()
112 LWP 11916 "rpc worker-1191" 0x00007fd812400ad3 in ?? ()
113 LWP 11917 "rpc worker-1191" 0x00007fd812400ad3 in ?? ()
114 LWP 11918 "rpc worker-1191" 0x00007fd812400ad3 in ?? ()
115 LWP 11919 "rpc worker-1191" 0x00007fd812400ad3 in ?? ()
116 LWP 11920 "rpc worker-1192" 0x00007fd812400ad3 in ?? ()
117 LWP 11921 "diag-logger-119" 0x00007fd812400fb9 in ?? ()
118 LWP 11922 "result-tracker-" 0x00007fd812400fb9 in ?? ()
119 LWP 11923 "excess-log-dele" 0x00007fd812400fb9 in ?? ()
120 LWP 11924 "tcmalloc-memory" 0x00007fd812400fb9 in ?? ()
121 LWP 11925 "acceptor-11925" 0x00007fd80f757fc7 in ?? ()
122 LWP 11926 "heartbeat-11926" 0x00007fd812400fb9 in ?? ()
123 LWP 11927 "maintenance_sch" 0x00007fd812400fb9 in ?? ()
124 LWP 12098 "raft [worker]-1" 0x00007fd812400fb9 in ?? ()
Thread 124 (LWP 12098):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd7c6058310 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00007fd7c60586a4 in ?? ()
#5 0x00007fd7c6058330 in ?? ()
#6 0x0000000000000023 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fd7c6058350 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007fd7c60583f0 in ?? ()
#12 0x00007fd7c60583e0 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 123 (LWP 11927):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd7c885d330 in ?? ()
#2 0x000000000000000b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00005635e8b27e60 in ?? ()
#5 0x00007fd7c885d350 in ?? ()
#6 0x0000000000000016 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 11926):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd7c905e310 in ?? ()
#2 0x0000000000000005 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00005635e8a77940 in ?? ()
#5 0x00007fd7c905e330 in ?? ()
#6 0x000000000000000a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 11925):
#0 0x00007fd80f757fc7 in ?? ()
#1 0x00007fd7c985f128 in ?? ()
#2 0x00007fd7c985f12c in ?? ()
#3 0x00007fd7c985f130 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 120 (LWP 11924):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd7ca0604f0 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffdcf04d3a0 in ?? ()
#5 0x00007fd7ca060510 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 11923):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd7ca861480 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 11922):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd7cb0624e0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00005635e8a32470 in ?? ()
#5 0x00007fd7cb062500 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 11921):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd7cb863450 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00005635e8cc22e0 in ?? ()
#5 0x00007fd7cb863470 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 116 (LWP 11920):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 11919):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 11918):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 11917):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 11916):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 11915):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 11914):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 11913):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 11912):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 11911):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 11910):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 11909):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 11908):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 11907):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 11906):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 11905):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 11904):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 11903):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 11902):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 11901):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 11900):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 11899):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 11898):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 11897):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 11896):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 11895):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 11894):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 11893):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 11892):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 11891):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 11890):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 11889):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 11888):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 11887):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 11886):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 11885):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 11884):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 11883):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 11882):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 11881):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 11880):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00005635e8c8a31c in ?? ()
#4 0x00007fd7e008c430 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd7e008c450 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00005635e8c85330 in ?? ()
#9 0x00007fd812400770 in ?? ()
#10 0x00007fd7e008c450 in ?? ()
#11 0x0000000100000000 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 11879):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 11878):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 11877):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 11876):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 11875):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 11874):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 11873):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 11872):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 11871):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 11870):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 11869):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 11868):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 11867):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 11866):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 11865):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 11864):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 11863):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 11862):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 11861):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 11860):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00005635e8c8100c in ?? ()
#4 0x00007fd7ea0a0430 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd7ea0a0450 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00005635e8c7bc30 in ?? ()
#9 0x00007fd812400770 in ?? ()
#10 0x00007fd7ea0a0450 in ?? ()
#11 0x0000000100000000 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 11859):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 11858):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 11857):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 11856):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 11855):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 11854):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 11853):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 11852):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 11851):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 11850):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 11849):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 11848):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 11847):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 11846):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 11845):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 11844):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 11843):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 11842):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 11841):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 11840):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 11839):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 11838):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 11837):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 11836):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 11835):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 11834):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 11833):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 11832):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 11831):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 11830):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 11829):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 11828):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 11827):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 11826):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 11825):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 11824):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 11823):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 11822):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 11821):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 11820):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd7fe0c8390 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 15 (LWP 11819):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd7fe8c9540 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 14 (LWP 11818):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd7ff0ca0c0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 13 (LWP 11817):
#0 0x00007fd812400ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 11816):
#0 0x00007fd80f756947 in ?? ()
#1 0x00007fd8000cc580 in ?? ()
#2 0x00007fd812a4b22b in ?? ()
#3 0x00007fd8000cc580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 11 (LWP 11815):
#0 0x00007fd80f756947 in ?? ()
#1 0x00007fd8008cd580 in ?? ()
#2 0x00007fd812a4b22b in ?? ()
#3 0x00007fd8008cd580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 10 (LWP 11814):
#0 0x00007fd80f756947 in ?? ()
#1 0x00007fd8010ce580 in ?? ()
#2 0x00007fd812a4b22b in ?? ()
#3 0x00007fd8010ce580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 9 (LWP 11813):
#0 0x00007fd80f756947 in ?? ()
#1 0x00007fd802cb2580 in ?? ()
#2 0x00007fd812a4b22b in ?? ()
#3 0x00007fd802cb2580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 8 (LWP 11810):
#0 0x00007fd80f749bb9 in ?? ()
#1 0x00007fd8044b5800 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 7 (LWP 11809):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd803cb44e0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 11808):
#0 0x00007fd8124049e2 in ?? ()
#1 0x00007fd8034b3330 in ?? ()
#2 0x00007fd8034b3380 in ?? ()
#3 0x00007fd8034b3350 in ?? ()
#4 0x0000000000000001 in ?? ()
#5 0x00007fd8034b36c0 in ?? ()
#6 0x00005635e8b29890 in ?? ()
#7 0x00005635e898fb10 in ?? ()
#8 0x00005635e898fb00 in ?? ()
#9 0x00007ffdcf04b950 in ?? ()
#10 0x00007fd812f5add2 in ?? ()
#11 0x00005635e8aa0330 in ?? ()
#12 0x00005635e8a32c30 in ?? ()
#13 0x00007fd8034b3390 in ?? ()
#14 0x00007fd811870ee0 in ?? ()
#15 0x00007fd800000010 in ?? ()
#16 0x00007fd8119a8b5f in ?? ()
#17 0x00007fd8034b33b0 in ?? ()
#18 0x00005635e8a1a1e0 in ?? ()
#19 0x00007fd811870ee0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 5 (LWP 11802):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd8054b72b0 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00005635e898fd08 in ?? ()
#5 0x00007fd8054b72d0 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 11801):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd805cb85c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00005635e8ad8438 in ?? ()
#5 0x00007fd805cb85e0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 3 (LWP 11800):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd8064b95c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00005635e8ad82f8 in ?? ()
#5 0x00007fd8064b95e0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (LWP 11799):
#0 0x00007fd812400fb9 in ?? ()
#1 0x00007fd806cba5c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00005635e8ad81b8 in ?? ()
#5 0x00007fd806cba5e0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 11798):
#0 0x00007fd812404d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20260928 01:57:31.673731 9548 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 3 with UUID 67f566a11c2d423495ccc094ca299245 and pid 11929
************************ BEGIN STACKS **************************
[New LWP 11930]
[New LWP 11931]
[New LWP 11932]
[New LWP 11933]
[New LWP 11939]
[New LWP 11940]
[New LWP 11941]
[New LWP 11944]
[New LWP 11945]
[New LWP 11946]
[New LWP 11947]
[New LWP 11948]
[New LWP 11949]
[New LWP 11950]
[New LWP 11951]
[New LWP 11952]
[New LWP 11953]
[New LWP 11954]
[New LWP 11955]
[New LWP 11956]
[New LWP 11957]
[New LWP 11958]
[New LWP 11959]
[New LWP 11960]
[New LWP 11961]
[New LWP 11962]
[New LWP 11963]
[New LWP 11964]
[New LWP 11965]
[New LWP 11966]
[New LWP 11967]
[New LWP 11968]
[New LWP 11969]
[New LWP 11970]
[New LWP 11971]
[New LWP 11972]
[New LWP 11973]
[New LWP 11974]
[New LWP 11975]
[New LWP 11976]
[New LWP 11977]
[New LWP 11978]
[New LWP 11979]
[New LWP 11980]
[New LWP 11981]
[New LWP 11982]
[New LWP 11983]
[New LWP 11984]
[New LWP 11985]
[New LWP 11986]
[New LWP 11987]
[New LWP 11988]
[New LWP 11989]
[New LWP 11990]
[New LWP 11991]
[New LWP 11992]
[New LWP 11993]
[New LWP 11994]
[New LWP 11995]
[New LWP 11996]
[New LWP 11997]
[New LWP 11998]
[New LWP 11999]
[New LWP 12000]
[New LWP 12001]
[New LWP 12002]
[New LWP 12003]
[New LWP 12004]
[New LWP 12005]
[New LWP 12006]
[New LWP 12007]
[New LWP 12008]
[New LWP 12009]
[New LWP 12010]
[New LWP 12011]
[New LWP 12012]
[New LWP 12013]
[New LWP 12014]
[New LWP 12015]
[New LWP 12016]
[New LWP 12017]
[New LWP 12018]
[New LWP 12019]
[New LWP 12020]
[New LWP 12021]
[New LWP 12022]
[New LWP 12023]
[New LWP 12024]
[New LWP 12025]
[New LWP 12026]
[New LWP 12027]
[New LWP 12028]
[New LWP 12029]
[New LWP 12030]
[New LWP 12031]
[New LWP 12032]
[New LWP 12033]
[New LWP 12034]
[New LWP 12035]
[New LWP 12036]
[New LWP 12037]
[New LWP 12038]
[New LWP 12039]
[New LWP 12040]
[New LWP 12041]
[New LWP 12042]
[New LWP 12043]
[New LWP 12044]
[New LWP 12045]
[New LWP 12046]
[New LWP 12047]
[New LWP 12048]
[New LWP 12049]
[New LWP 12050]
[New LWP 12051]
[New LWP 12052]
[New LWP 12053]
[New LWP 12054]
[New LWP 12055]
[New LWP 12056]
[New LWP 12057]
[New LWP 12058]
0x00007fa90c033d50 in ?? ()
Id Target Id Frame
* 1 LWP 11929 "kudu" 0x00007fa90c033d50 in ?? ()
2 LWP 11930 "kudu" 0x00007fa90c02ffb9 in ?? ()
3 LWP 11931 "kudu" 0x00007fa90c02ffb9 in ?? ()
4 LWP 11932 "kudu" 0x00007fa90c02ffb9 in ?? ()
5 LWP 11933 "kernel-watcher-" 0x00007fa90c02ffb9 in ?? ()
6 LWP 11939 "ntp client-1193" 0x00007fa90c0339e2 in ?? ()
7 LWP 11940 "file cache-evic" 0x00007fa90c02ffb9 in ?? ()
8 LWP 11941 "sq_acceptor" 0x00007fa909378bb9 in ?? ()
9 LWP 11944 "rpc reactor-119" 0x00007fa909385947 in ?? ()
10 LWP 11945 "rpc reactor-119" 0x00007fa909385947 in ?? ()
11 LWP 11946 "rpc reactor-119" 0x00007fa909385947 in ?? ()
12 LWP 11947 "rpc reactor-119" 0x00007fa909385947 in ?? ()
13 LWP 11948 "MaintenanceMgr " 0x00007fa90c02fad3 in ?? ()
14 LWP 11949 "txn-status-mana" 0x00007fa90c02ffb9 in ?? ()
15 LWP 11950 "collect_and_rem" 0x00007fa90c02ffb9 in ?? ()
16 LWP 11951 "tc-session-exp-" 0x00007fa90c02ffb9 in ?? ()
17 LWP 11952 "rpc worker-1195" 0x00007fa90c02fad3 in ?? ()
18 LWP 11953 "rpc worker-1195" 0x00007fa90c02fad3 in ?? ()
19 LWP 11954 "rpc worker-1195" 0x00007fa90c02fad3 in ?? ()
20 LWP 11955 "rpc worker-1195" 0x00007fa90c02fad3 in ?? ()
21 LWP 11956 "rpc worker-1195" 0x00007fa90c02fad3 in ?? ()
22 LWP 11957 "rpc worker-1195" 0x00007fa90c02fad3 in ?? ()
23 LWP 11958 "rpc worker-1195" 0x00007fa90c02fad3 in ?? ()
24 LWP 11959 "rpc worker-1195" 0x00007fa90c02fad3 in ?? ()
25 LWP 11960 "rpc worker-1196" 0x00007fa90c02fad3 in ?? ()
26 LWP 11961 "rpc worker-1196" 0x00007fa90c02fad3 in ?? ()
27 LWP 11962 "rpc worker-1196" 0x00007fa90c02fad3 in ?? ()
28 LWP 11963 "rpc worker-1196" 0x00007fa90c02fad3 in ?? ()
29 LWP 11964 "rpc worker-1196" 0x00007fa90c02fad3 in ?? ()
30 LWP 11965 "rpc worker-1196" 0x00007fa90c02fad3 in ?? ()
31 LWP 11966 "rpc worker-1196" 0x00007fa90c02fad3 in ?? ()
32 LWP 11967 "rpc worker-1196" 0x00007fa90c02fad3 in ?? ()
33 LWP 11968 "rpc worker-1196" 0x00007fa90c02fad3 in ?? ()
34 LWP 11969 "rpc worker-1196" 0x00007fa90c02fad3 in ?? ()
35 LWP 11970 "rpc worker-1197" 0x00007fa90c02fad3 in ?? ()
36 LWP 11971 "rpc worker-1197" 0x00007fa90c02fad3 in ?? ()
37 LWP 11972 "rpc worker-1197" 0x00007fa90c02fad3 in ?? ()
38 LWP 11973 "rpc worker-1197" 0x00007fa90c02fad3 in ?? ()
39 LWP 11974 "rpc worker-1197" 0x00007fa90c02fad3 in ?? ()
40 LWP 11975 "rpc worker-1197" 0x00007fa90c02fad3 in ?? ()
41 LWP 11976 "rpc worker-1197" 0x00007fa90c02fad3 in ?? ()
42 LWP 11977 "rpc worker-1197" 0x00007fa90c02fad3 in ?? ()
43 LWP 11978 "rpc worker-1197" 0x00007fa90c02fad3 in ?? ()
44 LWP 11979 "rpc worker-1197" 0x00007fa90c02fad3 in ?? ()
45 LWP 11980 "rpc worker-1198" 0x00007fa90c02fad3 in ?? ()
46 LWP 11981 "rpc worker-1198" 0x00007fa90c02fad3 in ?? ()
47 LWP 11982 "rpc worker-1198" 0x00007fa90c02fad3 in ?? ()
48 LWP 11983 "rpc worker-1198" 0x00007fa90c02fad3 in ?? ()
49 LWP 11984 "rpc worker-1198" 0x00007fa90c02fad3 in ?? ()
50 LWP 11985 "rpc worker-1198" 0x00007fa90c02fad3 in ?? ()
51 LWP 11986 "rpc worker-1198" 0x00007fa90c02fad3 in ?? ()
52 LWP 11987 "rpc worker-1198" 0x00007fa90c02fad3 in ?? ()
53 LWP 11988 "rpc worker-1198" 0x00007fa90c02fad3 in ?? ()
54 LWP 11989 "rpc worker-1198" 0x00007fa90c02fad3 in ?? ()
55 LWP 11990 "rpc worker-1199" 0x00007fa90c02fad3 in ?? ()
56 LWP 11991 "rpc worker-1199" 0x00007fa90c02fad3 in ?? ()
57 LWP 11992 "rpc worker-1199" 0x00007fa90c02fad3 in ?? ()
58 LWP 11993 "rpc worker-1199" 0x00007fa90c02fad3 in ?? ()
59 LWP 11994 "rpc worker-1199" 0x00007fa90c02fad3 in ?? ()
60 LWP 11995 "rpc worker-1199" 0x00007fa90c02fad3 in ?? ()
61 LWP 11996 "rpc worker-1199" 0x00007fa90c02fad3 in ?? ()
62 LWP 11997 "rpc worker-1199" 0x00007fa90c02fad3 in ?? ()
63 LWP 11998 "rpc worker-1199" 0x00007fa90c02fad3 in ?? ()
64 LWP 11999 "rpc worker-1199" 0x00007fa90c02fad3 in ?? ()
65 LWP 12000 "rpc worker-1200" 0x00007fa90c02fad3 in ?? ()
66 LWP 12001 "rpc worker-1200" 0x00007fa90c02fad3 in ?? ()
67 LWP 12002 "rpc worker-1200" 0x00007fa90c02fad3 in ?? ()
68 LWP 12003 "rpc worker-1200" 0x00007fa90c02fad3 in ?? ()
69 LWP 12004 "rpc worker-1200" 0x00007fa90c02fad3 in ?? ()
70 LWP 12005 "rpc worker-1200" 0x00007fa90c02fad3 in ?? ()
71 LWP 12006 "rpc worker-1200" 0x00007fa90c02fad3 in ?? ()
72 LWP 12007 "rpc worker-1200" 0x00007fa90c02fad3 in ?? ()
73 LWP 12008 "rpc worker-1200" 0x00007fa90c02fad3 in ?? ()
74 LWP 12009 "rpc worker-1200" 0x00007fa90c02fad3 in ?? ()
75 LWP 12010 "rpc worker-1201" 0x00007fa90c02fad3 in ?? ()
76 LWP 12011 "rpc worker-1201" 0x00007fa90c02fad3 in ?? ()
77 LWP 12012 "rpc worker-1201" 0x00007fa90c02fad3 in ?? ()
78 LWP 12013 "rpc worker-1201" 0x00007fa90c02fad3 in ?? ()
79 LWP 12014 "rpc worker-1201" 0x00007fa90c02fad3 in ?? ()
80 LWP 12015 "rpc worker-1201" 0x00007fa90c02fad3 in ?? ()
81 LWP 12016 "rpc worker-1201" 0x00007fa90c02fad3 in ?? ()
82 LWP 12017 "rpc worker-1201" 0x00007fa90c02fad3 in ?? ()
83 LWP 12018 "rpc worker-1201" 0x00007fa90c02fad3 in ?? ()
84 LWP 12019 "rpc worker-1201" 0x00007fa90c02fad3 in ?? ()
85 LWP 12020 "rpc worker-1202" 0x00007fa90c02fad3 in ?? ()
86 LWP 12021 "rpc worker-1202" 0x00007fa90c02fad3 in ?? ()
87 LWP 12022 "rpc worker-1202" 0x00007fa90c02fad3 in ?? ()
88 LWP 12023 "rpc worker-1202" 0x00007fa90c02fad3 in ?? ()
89 LWP 12024 "rpc worker-1202" 0x00007fa90c02fad3 in ?? ()
90 LWP 12025 "rpc worker-1202" 0x00007fa90c02fad3 in ?? ()
91 LWP 12026 "rpc worker-1202" 0x00007fa90c02fad3 in ?? ()
92 LWP 12027 "rpc worker-1202" 0x00007fa90c02fad3 in ?? ()
93 LWP 12028 "rpc worker-1202" 0x00007fa90c02fad3 in ?? ()
94 LWP 12029 "rpc worker-1202" 0x00007fa90c02fad3 in ?? ()
95 LWP 12030 "rpc worker-1203" 0x00007fa90c02fad3 in ?? ()
96 LWP 12031 "rpc worker-1203" 0x00007fa90c02fad3 in ?? ()
97 LWP 12032 "rpc worker-1203" 0x00007fa90c02fad3 in ?? ()
98 LWP 12033 "rpc worker-1203" 0x00007fa90c02fad3 in ?? ()
99 LWP 12034 "rpc worker-1203" 0x00007fa90c02fad3 in ?? ()
100 LWP 12035 "rpc worker-1203" 0x00007fa90c02fad3 in ?? ()
101 LWP 12036 "rpc worker-1203" 0x00007fa90c02fad3 in ?? ()
102 LWP 12037 "rpc worker-1203" 0x00007fa90c02fad3 in ?? ()
103 LWP 12038 "rpc worker-1203" 0x00007fa90c02fad3 in ?? ()
104 LWP 12039 "rpc worker-1203" 0x00007fa90c02fad3 in ?? ()
105 LWP 12040 "rpc worker-1204" 0x00007fa90c02fad3 in ?? ()
106 LWP 12041 "rpc worker-1204" 0x00007fa90c02fad3 in ?? ()
107 LWP 12042 "rpc worker-1204" 0x00007fa90c02fad3 in ?? ()
108 LWP 12043 "rpc worker-1204" 0x00007fa90c02fad3 in ?? ()
109 LWP 12044 "rpc worker-1204" 0x00007fa90c02fad3 in ?? ()
110 LWP 12045 "rpc worker-1204" 0x00007fa90c02fad3 in ?? ()
111 LWP 12046 "rpc worker-1204" 0x00007fa90c02fad3 in ?? ()
112 LWP 12047 "rpc worker-1204" 0x00007fa90c02fad3 in ?? ()
113 LWP 12048 "rpc worker-1204" 0x00007fa90c02fad3 in ?? ()
114 LWP 12049 "rpc worker-1204" 0x00007fa90c02fad3 in ?? ()
115 LWP 12050 "rpc worker-1205" 0x00007fa90c02fad3 in ?? ()
116 LWP 12051 "rpc worker-1205" 0x00007fa90c02fad3 in ?? ()
117 LWP 12052 "diag-logger-120" 0x00007fa90c02ffb9 in ?? ()
118 LWP 12053 "result-tracker-" 0x00007fa90c02ffb9 in ?? ()
119 LWP 12054 "excess-log-dele" 0x00007fa90c02ffb9 in ?? ()
120 LWP 12055 "tcmalloc-memory" 0x00007fa90c02ffb9 in ?? ()
121 LWP 12056 "acceptor-12056" 0x00007fa909386fc7 in ?? ()
122 LWP 12057 "heartbeat-12057" 0x00007fa90c02ffb9 in ?? ()
123 LWP 12058 "maintenance_sch" 0x00007fa90c02ffb9 in ?? ()
Thread 123 (LWP 12058):
#0 0x00007fa90c02ffb9 in ?? ()
#1 0x00007fa8c248c330 in ?? ()
#2 0x000000000000000d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000055b1ebac9e60 in ?? ()
#5 0x00007fa8c248c350 in ?? ()
#6 0x000000000000001a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 12057):
#0 0x00007fa90c02ffb9 in ?? ()
#1 0x00007fa8c2c8d310 in ?? ()
#2 0x0000000000000005 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x000055b1eba19944 in ?? ()
#5 0x00007fa8c2c8d330 in ?? ()
#6 0x000000000000000b in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fa8c2c8d350 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007fa8c2c8d3e0 in ?? ()
#12 0x00007fa8c2c8d4b8 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 121 (LWP 12056):
#0 0x00007fa909386fc7 in ?? ()
#1 0x00007fa8c348e128 in ?? ()
#2 0x00007fa8c348e12c in ?? ()
#3 0x00007fa8c348e130 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 120 (LWP 12055):
#0 0x00007fa90c02ffb9 in ?? ()
#1 0x00007fa8c3c8f4f0 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff49724560 in ?? ()
#5 0x00007fa8c3c8f510 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 12054):
#0 0x00007fa90c02ffb9 in ?? ()
#1 0x00007fa8c4490480 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 12053):
#0 0x00007fa90c02ffb9 in ?? ()
#1 0x00007fa8c4c914e0 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000055b1eb9d4470 in ?? ()
#5 0x00007fa8c4c91500 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 12052):
#0 0x00007fa90c02ffb9 in ?? ()
#1 0x00007fa8c5492450 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000055b1ebc642e0 in ?? ()
#5 0x00007fa8c5492470 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 116 (LWP 12051):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 12050):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 12049):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 12048):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 12047):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 12046):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 12045):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 12044):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 12043):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 12042):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 12041):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 12040):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 12039):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 12038):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 12037):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 12036):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 12035):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 12034):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 12033):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 12032):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 12031):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 12030):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 12029):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 12028):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 12027):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 12026):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 12025):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 12024):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 12023):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 12022):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 12021):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 12020):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 12019):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 12018):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 12017):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 12016):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 12015):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 12014):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 12013):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 12012):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 12011):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x000000000000005c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000055b1ebc2c318 in ?? ()
#4 0x00007fa8d9cbb430 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa8d9cbb450 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 12010):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 12009):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 12008):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 12007):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 12006):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 12005):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 12004):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 12003):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 12002):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 12001):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 12000):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 11999):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 11998):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 11997):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 11996):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 11995):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 11994):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 11993):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 11992):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 11991):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000055b1ebc2300c in ?? ()
#4 0x00007fa8e3ccf430 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa8e3ccf450 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000055b1ebc1dc30 in ?? ()
#9 0x00007fa90c02f770 in ?? ()
#10 0x00007fa8e3ccf450 in ?? ()
#11 0x0000000100000000 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 11990):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 11989):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 11988):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 11987):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 11986):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 11985):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 11984):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 11983):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 11982):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 11981):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 11980):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 11979):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 11978):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 11977):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 11976):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 11975):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 11974):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 11973):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 11972):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 11971):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 11970):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 11969):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 11968):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 11967):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 11966):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 11965):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 11964):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 11963):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 11962):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 11961):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 11960):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 11959):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 11958):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 11957):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 11956):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 11955):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 11954):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 11953):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 11952):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 11951):
#0 0x00007fa90c02ffb9 in ?? ()
#1 0x00007fa8f7cf7390 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 15 (LWP 11950):
#0 0x00007fa90c02ffb9 in ?? ()
#1 0x00007fa8f84f8540 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 14 (LWP 11949):
#0 0x00007fa90c02ffb9 in ?? ()
#1 0x00007fa8f8cf90c0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 13 (LWP 11948):
#0 0x00007fa90c02fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 11947):
#0 0x00007fa909385947 in ?? ()
#1 0x00007fa8f9cfb580 in ?? ()
#2 0x00007fa90c67a22b in ?? ()
#3 0x00007fa8f9cfb580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 11 (LWP 11946):
#0 0x00007fa909385947 in ?? ()
#1 0x00007fa8fa4fc580 in ?? ()
#2 0x00007fa90c67a22b in ?? ()
#3 0x00007fa8fa4fc580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 10 (LWP 11945):
#0 0x00007fa909385947 in ?? ()
#1 0x00007fa8facfd580 in ?? ()
#2 0x00007fa90c67a22b in ?? ()
#3 0x00007fa8facfd580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 9 (LWP 11944):
#0 0x00007fa909385947 in ?? ()
#1 0x00007fa8fc8e1580 in ?? ()
#2 0x00007fa90c67a22b in ?? ()
#3 0x00007fa8fc8e1580 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 8 (LWP 11941):
#0 0x00007fa909378bb9 in ?? ()
#1 0x00007fa8fe0e4800 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 7 (LWP 11940):
#0 0x00007fa90c02ffb9 in ?? ()
#1 0x00007fa8fd8e34e0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 11939):
#0 0x00007fa90c0339e2 in ?? ()
#1 0x00007fa8fd0e2330 in ?? ()
#2 0x00007fa8fd0e2380 in ?? ()
#3 0x00007fa8fd0e2350 in ?? ()
#4 0x0000000000000001 in ?? ()
#5 0x00007fa8fd0e26c0 in ?? ()
#6 0x000055b1ebacb890 in ?? ()
#7 0x000055b1eb931b10 in ?? ()
#8 0x000055b1eb931b00 in ?? ()
#9 0x00007fff49722b10 in ?? ()
#10 0x00007fa90cb89dd2 in ?? ()
#11 0x000055b1eba43030 in ?? ()
#12 0x000055b1eb9d4c30 in ?? ()
#13 0x00007fa8fd0e2390 in ?? ()
#14 0x00007fa90b49fee0 in ?? ()
#15 0x00007fa800000010 in ?? ()
#16 0x00007fa90b5d7b5f in ?? ()
#17 0x00007fa8fd0e23b0 in ?? ()
#18 0x000055b1eb9bc1e0 in ?? ()
#19 0x00007fa90b49fee0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 5 (LWP 11933):
#0 0x00007fa90c02ffb9 in ?? ()
#1 0x00007fa8ff0e62b0 in ?? ()
#2 0x0000000000000010 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000055b1eb931d08 in ?? ()
#5 0x00007fa8ff0e62d0 in ?? ()
#6 0x0000000000000020 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 11932):
#0 0x00007fa90c02ffb9 in ?? ()
#1 0x00007fa8ff8e75c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000055b1eba7a438 in ?? ()
#5 0x00007fa8ff8e75e0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 3 (LWP 11931):
#0 0x00007fa90c02ffb9 in ?? ()
#1 0x00007fa9000e85c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000055b1eba7a2f8 in ?? ()
#5 0x00007fa9000e85e0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (LWP 11930):
#0 0x00007fa90c02ffb9 in ?? ()
#1 0x00007fa9008e95c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000055b1eba7a1b8 in ?? ()
#5 0x00007fa9008e95e0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 11929):
#0 0x00007fa90c033d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20260928 01:57:32.271800 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 11536
I20260928 01:57:32.277346 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 11667
I20260928 01:57:32.282697 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 11798
I20260928 01:57:32.288259 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 11929
I20260928 01:57:32.293341 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 11445
2026-09-28T01:57:32Z chronyd exiting
I20260928 01:57:32.310009 9548 test_util.cc:182] -----------------------------------------------
I20260928 01:57:32.310091 9548 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.DeleteTableITest.TestNoDeleteTombstonedTablets.1790560638442226-9548-0
[ FAILED ] DeleteTableITest.TestNoDeleteTombstonedTablets (4329 ms)
[----------] 2 tests from DeleteTableITest (13857 ms total)
[----------] 1 test from Params/DeleteTableWhileScanInProgressParamTest
[ RUN ] Params/DeleteTableWhileScanInProgressParamTest.Test/1
2026-09-28T01:57:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-28T01:57:32Z Disabled control of system clock
I20260928 01:57:32.318141 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.83.62:37899
--webserver_interface=127.9.83.62
--webserver_port=0
--builtin_ntp_servers=127.9.83.20:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.83.62:37899 with env {}
W20260928 01:57:32.416296 12141 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:32.416510 12141 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:32.416546 12141 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:32.419833 12141 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 01:57:32.419886 12141 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:32.419905 12141 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 01:57:32.419930 12141 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260928 01:57:32.423856 12141 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.83.62:37899
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.83.62:37899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.83.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.12141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:32.424880 12141 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:32.425961 12141 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:32.431149 12146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:32.431155 12147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:32.431155 12149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:32.431561 12141 server_base.cc:1061] running on GCE node
I20260928 01:57:32.432000 12141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:32.432893 12141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:32.434057 12141 hybrid_clock.cc:648] HybridClock initialized: now 1790560652434041 us; error 35 us; skew 500 ppm
I20260928 01:57:32.435926 12141 webserver.cc:533] Webserver started at http://127.9.83.62:37617/ using document root <none> and password file <none>
I20260928 01:57:32.436482 12141 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:32.436539 12141 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:32.436707 12141 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 01:57:32.438197 12141 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/data/instance:
uuid: "d1d868702e2f4dc6afad51e179fce858"
format_stamp: "Formatted at 2026-09-28 01:57:32 on dist-test-slave-lkt6"
I20260928 01:57:32.438683 12141 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/wal/instance:
uuid: "d1d868702e2f4dc6afad51e179fce858"
format_stamp: "Formatted at 2026-09-28 01:57:32 on dist-test-slave-lkt6"
I20260928 01:57:32.441901 12141 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 01:57:32.444046 12155 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:32.445096 12141 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 01:57:32.445214 12141 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/wal
uuid: "d1d868702e2f4dc6afad51e179fce858"
format_stamp: "Formatted at 2026-09-28 01:57:32 on dist-test-slave-lkt6"
I20260928 01:57:32.445304 12141 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:32.461854 12141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:32.462558 12141 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:32.462733 12141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:32.469810 12141 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.62:37899
I20260928 01:57:32.469821 12207 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.62:37899 every 8 connection(s)
I20260928 01:57:32.470861 12141 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/data/info.pb
I20260928 01:57:32.473182 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 12141
I20260928 01:57:32.473306 9548 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/master-0/wal/instance
I20260928 01:57:32.473806 12208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:32.479193 12208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858: Bootstrap starting.
I20260928 01:57:32.481324 12208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858: Neither blocks nor log segments found. Creating new log.
I20260928 01:57:32.482036 12208 log.cc:826] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:32.483644 12208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858: No bootstrap required, opened a new log
I20260928 01:57:32.486104 12208 raft_consensus.cc:359] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d868702e2f4dc6afad51e179fce858" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 37899 } }
I20260928 01:57:32.486316 12208 raft_consensus.cc:385] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 01:57:32.486371 12208 raft_consensus.cc:740] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d868702e2f4dc6afad51e179fce858, State: Initialized, Role: FOLLOWER
I20260928 01:57:32.486809 12208 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d868702e2f4dc6afad51e179fce858" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 37899 } }
I20260928 01:57:32.486953 12208 raft_consensus.cc:399] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 01:57:32.487025 12208 raft_consensus.cc:493] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 01:57:32.487108 12208 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [term 0 FOLLOWER]: Advancing to term 1
I20260928 01:57:32.488051 12208 raft_consensus.cc:515] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d868702e2f4dc6afad51e179fce858" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 37899 } }
I20260928 01:57:32.488397 12208 leader_election.cc:304] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1d868702e2f4dc6afad51e179fce858; no voters:
I20260928 01:57:32.488704 12208 leader_election.cc:290] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 01:57:32.488796 12213 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 01:57:32.489004 12213 raft_consensus.cc:697] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [term 1 LEADER]: Becoming Leader. State: Replica: d1d868702e2f4dc6afad51e179fce858, State: Running, Role: LEADER
I20260928 01:57:32.489301 12213 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d868702e2f4dc6afad51e179fce858" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 37899 } }
I20260928 01:57:32.489807 12208 sys_catalog.cc:565] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 01:57:32.490340 12214 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1d868702e2f4dc6afad51e179fce858. Latest consensus state: current_term: 1 leader_uuid: "d1d868702e2f4dc6afad51e179fce858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d868702e2f4dc6afad51e179fce858" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 37899 } } }
I20260928 01:57:32.490449 12214 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [sys.catalog]: This master's current role is: LEADER
I20260928 01:57:32.490738 12215 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1d868702e2f4dc6afad51e179fce858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1d868702e2f4dc6afad51e179fce858" member_type: VOTER last_known_addr { host: "127.9.83.62" port: 37899 } } }
I20260928 01:57:32.490869 12215 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858 [sys.catalog]: This master's current role is: LEADER
I20260928 01:57:32.491134 12217 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260928 01:57:32.494156 12217 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260928 01:57:32.499046 12217 catalog_manager.cc:1396] Generated new cluster ID: db3b13e711654ead950274be41618922
I20260928 01:57:32.499110 12217 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260928 01:57:32.515522 12217 catalog_manager.cc:1419] Generated new certificate authority record
I20260928 01:57:32.516656 12217 catalog_manager.cc:1553] Loading token signing keys...
I20260928 01:57:32.524581 12217 catalog_manager.cc:6219] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858: Generated new TSK 0
I20260928 01:57:32.525238 12217 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260928 01:57:32.532249 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.1:0
--local_ip_for_outbound_sockets=127.9.83.1
--webserver_interface=127.9.83.1
--webserver_port=0
--tserver_master_addrs=127.9.83.62:37899
--builtin_ntp_servers=127.9.83.20:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_compaction_budget_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001 with env {}
W20260928 01:57:32.632629 12232 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:32.632848 12232 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:32.632929 12232 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:32.636327 12232 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:32.636418 12232 flags.cc:432] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260928 01:57:32.636447 12232 flags.cc:432] Enabled experimental flag: --tablet_compaction_budget_mb=1
W20260928 01:57:32.636482 12232 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260928 01:57:32.636551 12232 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.1
I20260928 01:57:32.640544 12232 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.83.1
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=1
--flush_threshold_mb=1
--tserver_master_addrs=127.9.83.62:37899
--never_fsync=true
--heap_profile_path=/tmp/kudu.12232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.1
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:32.641589 12232 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:32.642684 12232 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:32.648792 12240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:32.648856 12238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:32.648798 12237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:32.649014 12232 server_base.cc:1061] running on GCE node
I20260928 01:57:32.649823 12232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:32.650365 12232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:32.651571 12232 hybrid_clock.cc:648] HybridClock initialized: now 1790560652651552 us; error 32 us; skew 500 ppm
I20260928 01:57:32.653174 12232 webserver.cc:533] Webserver started at http://127.9.83.1:46603/ using document root <none> and password file <none>
I20260928 01:57:32.653736 12232 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:32.653787 12232 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:32.653963 12232 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 01:57:32.655469 12232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/data/instance:
uuid: "3cf31b55f6f941a9a9759e7d971a6011"
format_stamp: "Formatted at 2026-09-28 01:57:32 on dist-test-slave-lkt6"
I20260928 01:57:32.655891 12232 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/wal/instance:
uuid: "3cf31b55f6f941a9a9759e7d971a6011"
format_stamp: "Formatted at 2026-09-28 01:57:32 on dist-test-slave-lkt6"
I20260928 01:57:32.659085 12232 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 01:57:32.661211 12246 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:32.662215 12232 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 01:57:32.662318 12232 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/wal
uuid: "3cf31b55f6f941a9a9759e7d971a6011"
format_stamp: "Formatted at 2026-09-28 01:57:32 on dist-test-slave-lkt6"
I20260928 01:57:32.662456 12232 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:32.692775 12232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:32.693421 12232 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:32.693593 12232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:32.694146 12232 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:32.695129 12232 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 01:57:32.695179 12232 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:32.695230 12232 ts_tablet_manager.cc:616] Registered 0 tablets
I20260928 01:57:32.695253 12232 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:32.703599 12232 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.1:43331
I20260928 01:57:32.703631 12359 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.1:43331 every 8 connection(s)
I20260928 01:57:32.704540 12232 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/data/info.pb
I20260928 01:57:32.707679 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 12232
I20260928 01:57:32.707821 9548 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/wal/instance
I20260928 01:57:32.710115 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.2:0
--local_ip_for_outbound_sockets=127.9.83.2
--webserver_interface=127.9.83.2
--webserver_port=0
--tserver_master_addrs=127.9.83.62:37899
--builtin_ntp_servers=127.9.83.20:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_compaction_budget_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001 with env {}
I20260928 01:57:32.714203 12360 heartbeater.cc:344] Connected to a master server at 127.9.83.62:37899
I20260928 01:57:32.714443 12360 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:32.715003 12360 heartbeater.cc:507] Master 127.9.83.62:37899 requested a full tablet report, sending...
I20260928 01:57:32.716571 12171 ts_manager.cc:194] Registered new tserver with Master: 3cf31b55f6f941a9a9759e7d971a6011 (127.9.83.1:43331)
I20260928 01:57:32.718166 12171 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.1:52675
W20260928 01:57:32.811139 12363 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:32.811347 12363 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:32.811398 12363 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:32.814685 12363 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:32.814792 12363 flags.cc:432] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260928 01:57:32.814819 12363 flags.cc:432] Enabled experimental flag: --tablet_compaction_budget_mb=1
W20260928 01:57:32.814858 12363 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260928 01:57:32.814905 12363 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.2
I20260928 01:57:32.818791 12363 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.83.2
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=1
--flush_threshold_mb=1
--tserver_master_addrs=127.9.83.62:37899
--never_fsync=true
--heap_profile_path=/tmp/kudu.12363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.2
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:32.819787 12363 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:32.820828 12363 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:32.826987 12371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:32.826959 12369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:32.826964 12368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:32.827179 12363 server_base.cc:1061] running on GCE node
I20260928 01:57:32.827589 12363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:32.828097 12363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:32.829253 12363 hybrid_clock.cc:648] HybridClock initialized: now 1790560652829240 us; error 31 us; skew 500 ppm
I20260928 01:57:32.830973 12363 webserver.cc:533] Webserver started at http://127.9.83.2:33751/ using document root <none> and password file <none>
I20260928 01:57:32.831496 12363 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:32.831547 12363 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:32.831717 12363 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 01:57:32.833200 12363 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/data/instance:
uuid: "e256f95dcf414d9d870b13e21fcd1734"
format_stamp: "Formatted at 2026-09-28 01:57:32 on dist-test-slave-lkt6"
I20260928 01:57:32.833626 12363 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/wal/instance:
uuid: "e256f95dcf414d9d870b13e21fcd1734"
format_stamp: "Formatted at 2026-09-28 01:57:32 on dist-test-slave-lkt6"
I20260928 01:57:32.836814 12363 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260928 01:57:32.839003 12377 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:32.840098 12363 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 01:57:32.840205 12363 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/wal
uuid: "e256f95dcf414d9d870b13e21fcd1734"
format_stamp: "Formatted at 2026-09-28 01:57:32 on dist-test-slave-lkt6"
I20260928 01:57:32.840296 12363 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:32.856549 12363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:32.857189 12363 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:32.857352 12363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:32.857889 12363 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:32.858862 12363 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 01:57:32.858911 12363 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:32.858961 12363 ts_tablet_manager.cc:616] Registered 0 tablets
I20260928 01:57:32.858986 12363 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:32.867295 12363 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.2:39611
I20260928 01:57:32.867337 12490 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.2:39611 every 8 connection(s)
I20260928 01:57:32.868155 12363 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/data/info.pb
I20260928 01:57:32.876051 12491 heartbeater.cc:344] Connected to a master server at 127.9.83.62:37899
I20260928 01:57:32.876164 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 12363
I20260928 01:57:32.876282 9548 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/wal/instance
I20260928 01:57:32.876297 12491 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:32.876801 12491 heartbeater.cc:507] Master 127.9.83.62:37899 requested a full tablet report, sending...
I20260928 01:57:32.877784 12172 ts_manager.cc:194] Registered new tserver with Master: e256f95dcf414d9d870b13e21fcd1734 (127.9.83.2:39611)
I20260928 01:57:32.878314 9548 external_mini_cluster.cc:1372] Running /tmp/dist-test-task6vbGbg/build/debug/bin/kudu
/tmp/dist-test-task6vbGbg/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.83.3:0
--local_ip_for_outbound_sockets=127.9.83.3
--webserver_interface=127.9.83.3
--webserver_port=0
--tserver_master_addrs=127.9.83.62:37899
--builtin_ntp_servers=127.9.83.20:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--flush_threshold_mb=1
--tablet_compaction_budget_mb=1
--tablet_delta_store_major_compact_min_ratio=0.001 with env {}
I20260928 01:57:32.878317 12172 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.2:57339
W20260928 01:57:32.976461 12494 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 01:57:32.976665 12494 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 01:57:32.976722 12494 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260928 01:57:32.980032 12494 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260928 01:57:32.980108 12494 flags.cc:432] Enabled experimental flag: --tablet_delta_store_major_compact_min_ratio=0.001
W20260928 01:57:32.980123 12494 flags.cc:432] Enabled experimental flag: --tablet_compaction_budget_mb=1
W20260928 01:57:32.980139 12494 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=1
W20260928 01:57:32.980206 12494 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.83.3
I20260928 01:57:32.984045 12494 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.83.20:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.83.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.83.3
--webserver_port=0
--tablet_delta_store_major_compact_min_ratio=0.001
--tablet_compaction_budget_mb=1
--flush_threshold_mb=1
--tserver_master_addrs=127.9.83.62:37899
--never_fsync=true
--heap_profile_path=/tmp/kudu.12494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.83.3
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3b18f0139b1d21a072b7353269301fba16dca54e
build type DEBUG
built by None at 28 Sep 2026 01:43:13 UTC on bdcb31816ec0
build id 14031
I20260928 01:57:32.985095 12494 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 01:57:32.986183 12494 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260928 01:57:32.992475 12502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:32.992524 12500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 01:57:32.992578 12499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 01:57:32.992846 12494 server_base.cc:1061] running on GCE node
I20260928 01:57:32.993160 12494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 01:57:32.993646 12494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 01:57:32.994818 12494 hybrid_clock.cc:648] HybridClock initialized: now 1790560652994805 us; error 24 us; skew 500 ppm
I20260928 01:57:32.996441 12494 webserver.cc:533] Webserver started at http://127.9.83.3:44609/ using document root <none> and password file <none>
I20260928 01:57:32.996955 12494 fs_manager.cc:362] Metadata directory not provided
I20260928 01:57:32.997002 12494 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 01:57:32.997154 12494 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 01:57:32.998600 12494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/data/instance:
uuid: "a6209579c03e48dbb59a247e1d0ed07c"
format_stamp: "Formatted at 2026-09-28 01:57:32 on dist-test-slave-lkt6"
I20260928 01:57:32.999061 12494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/wal/instance:
uuid: "a6209579c03e48dbb59a247e1d0ed07c"
format_stamp: "Formatted at 2026-09-28 01:57:32 on dist-test-slave-lkt6"
I20260928 01:57:33.002125 12494 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 01:57:33.004177 12508 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:33.005126 12494 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 01:57:33.005218 12494 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/data,/tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/wal
uuid: "a6209579c03e48dbb59a247e1d0ed07c"
format_stamp: "Formatted at 2026-09-28 01:57:32 on dist-test-slave-lkt6"
I20260928 01:57:33.005290 12494 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 01:57:33.034713 12494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260928 01:57:33.035399 12494 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 01:57:33.035557 12494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 01:57:33.036108 12494 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260928 01:57:33.037132 12494 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 01:57:33.037184 12494 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:33.037225 12494 ts_tablet_manager.cc:616] Registered 0 tablets
I20260928 01:57:33.037251 12494 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 01:57:33.045938 12494 rpc_server.cc:307] RPC server started. Bound to: 127.9.83.3:41503
I20260928 01:57:33.045974 12621 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.83.3:41503 every 8 connection(s)
I20260928 01:57:33.046940 12494 server_base.cc:1193] Dumped server information to /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/data/info.pb
I20260928 01:57:33.053799 9548 external_mini_cluster.cc:1434] Started /tmp/dist-test-task6vbGbg/build/debug/bin/kudu as pid 12494
I20260928 01:57:33.053923 9548 external_mini_cluster.cc:1448] Reading /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/wal/instance
I20260928 01:57:33.055408 12622 heartbeater.cc:344] Connected to a master server at 127.9.83.62:37899
I20260928 01:57:33.055670 12622 heartbeater.cc:461] Registering TS with master...
I20260928 01:57:33.056167 12622 heartbeater.cc:507] Master 127.9.83.62:37899 requested a full tablet report, sending...
I20260928 01:57:33.057106 12171 ts_manager.cc:194] Registered new tserver with Master: a6209579c03e48dbb59a247e1d0ed07c (127.9.83.3:41503)
I20260928 01:57:33.057626 12171 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.9.83.3:43611
I20260928 01:57:33.067996 9548 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260928 01:57:33.083555 12171 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34346:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 01:57:33.084645 12171 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 01:57:33.100507 12556 tablet_service.cc:1511] Processing CreateTablet for tablet 0e01bb2faf0f46549a09f971bf6ef2e2 (DEFAULT_TABLE table=test-workload [id=38ae7dc437a34dc0989e7a76083e7a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 01:57:33.100507 12294 tablet_service.cc:1511] Processing CreateTablet for tablet 0e01bb2faf0f46549a09f971bf6ef2e2 (DEFAULT_TABLE table=test-workload [id=38ae7dc437a34dc0989e7a76083e7a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 01:57:33.100507 12425 tablet_service.cc:1511] Processing CreateTablet for tablet 0e01bb2faf0f46549a09f971bf6ef2e2 (DEFAULT_TABLE table=test-workload [id=38ae7dc437a34dc0989e7a76083e7a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 01:57:33.101374 12556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e01bb2faf0f46549a09f971bf6ef2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:33.101389 12425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e01bb2faf0f46549a09f971bf6ef2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:33.101389 12294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e01bb2faf0f46549a09f971bf6ef2e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 01:57:33.106215 12644 tablet_bootstrap.cc:492] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c: Bootstrap starting.
I20260928 01:57:33.107358 12646 tablet_bootstrap.cc:492] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011: Bootstrap starting.
I20260928 01:57:33.108495 12644 tablet_bootstrap.cc:654] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c: Neither blocks nor log segments found. Creating new log.
I20260928 01:57:33.109180 12646 tablet_bootstrap.cc:654] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011: Neither blocks nor log segments found. Creating new log.
I20260928 01:57:33.109305 12644 log.cc:826] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:33.109851 12646 log.cc:826] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:33.110520 12645 tablet_bootstrap.cc:492] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734: Bootstrap starting.
I20260928 01:57:33.110894 12644 tablet_bootstrap.cc:492] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c: No bootstrap required, opened a new log
I20260928 01:57:33.111092 12644 ts_tablet_manager.cc:1403] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 01:57:33.111470 12646 tablet_bootstrap.cc:492] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011: No bootstrap required, opened a new log
I20260928 01:57:33.111671 12646 ts_tablet_manager.cc:1403] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 01:57:33.112783 12645 tablet_bootstrap.cc:654] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734: Neither blocks nor log segments found. Creating new log.
I20260928 01:57:33.113584 12645 log.cc:826] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734: Log is configured to *not* fsync() on all Append() calls
I20260928 01:57:33.113701 12644 raft_consensus.cc:359] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e256f95dcf414d9d870b13e21fcd1734" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 39611 } } peers { permanent_uuid: "a6209579c03e48dbb59a247e1d0ed07c" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41503 } } peers { permanent_uuid: "3cf31b55f6f941a9a9759e7d971a6011" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43331 } }
I20260928 01:57:33.113900 12644 raft_consensus.cc:385] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 01:57:33.113948 12644 raft_consensus.cc:740] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6209579c03e48dbb59a247e1d0ed07c, State: Initialized, Role: FOLLOWER
I20260928 01:57:33.114424 12644 consensus_queue.cc:260] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e256f95dcf414d9d870b13e21fcd1734" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 39611 } } peers { permanent_uuid: "a6209579c03e48dbb59a247e1d0ed07c" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41503 } } peers { permanent_uuid: "3cf31b55f6f941a9a9759e7d971a6011" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43331 } }
I20260928 01:57:33.114657 12646 raft_consensus.cc:359] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e256f95dcf414d9d870b13e21fcd1734" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 39611 } } peers { permanent_uuid: "a6209579c03e48dbb59a247e1d0ed07c" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41503 } } peers { permanent_uuid: "3cf31b55f6f941a9a9759e7d971a6011" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43331 } }
I20260928 01:57:33.114861 12646 raft_consensus.cc:385] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 01:57:33.114904 12646 raft_consensus.cc:740] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf31b55f6f941a9a9759e7d971a6011, State: Initialized, Role: FOLLOWER
I20260928 01:57:33.115278 12644 ts_tablet_manager.cc:1434] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 01:57:33.115346 12646 consensus_queue.cc:260] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e256f95dcf414d9d870b13e21fcd1734" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 39611 } } peers { permanent_uuid: "a6209579c03e48dbb59a247e1d0ed07c" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41503 } } peers { permanent_uuid: "3cf31b55f6f941a9a9759e7d971a6011" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43331 } }
I20260928 01:57:33.115618 12622 heartbeater.cc:499] Master 127.9.83.62:37899 was elected leader, sending a full tablet report...
I20260928 01:57:33.115710 12645 tablet_bootstrap.cc:492] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734: No bootstrap required, opened a new log
I20260928 01:57:33.115900 12645 ts_tablet_manager.cc:1403] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 01:57:33.116103 12360 heartbeater.cc:499] Master 127.9.83.62:37899 was elected leader, sending a full tablet report...
I20260928 01:57:33.116183 12646 ts_tablet_manager.cc:1434] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 01:57:33.118867 12645 raft_consensus.cc:359] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e256f95dcf414d9d870b13e21fcd1734" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 39611 } } peers { permanent_uuid: "a6209579c03e48dbb59a247e1d0ed07c" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41503 } } peers { permanent_uuid: "3cf31b55f6f941a9a9759e7d971a6011" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43331 } }
I20260928 01:57:33.119047 12645 raft_consensus.cc:385] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 01:57:33.119100 12645 raft_consensus.cc:740] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e256f95dcf414d9d870b13e21fcd1734, State: Initialized, Role: FOLLOWER
I20260928 01:57:33.119521 12645 consensus_queue.cc:260] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e256f95dcf414d9d870b13e21fcd1734" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 39611 } } peers { permanent_uuid: "a6209579c03e48dbb59a247e1d0ed07c" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41503 } } peers { permanent_uuid: "3cf31b55f6f941a9a9759e7d971a6011" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43331 } }
I20260928 01:57:33.120239 12491 heartbeater.cc:499] Master 127.9.83.62:37899 was elected leader, sending a full tablet report...
I20260928 01:57:33.120504 12645 ts_tablet_manager.cc:1434] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 01:57:33.129794 12651 raft_consensus.cc:493] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 01:57:33.130008 12651 raft_consensus.cc:515] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e256f95dcf414d9d870b13e21fcd1734" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 39611 } } peers { permanent_uuid: "a6209579c03e48dbb59a247e1d0ed07c" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41503 } } peers { permanent_uuid: "3cf31b55f6f941a9a9759e7d971a6011" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43331 } }
I20260928 01:57:33.131124 12651 leader_election.cc:290] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e256f95dcf414d9d870b13e21fcd1734 (127.9.83.2:39611), a6209579c03e48dbb59a247e1d0ed07c (127.9.83.3:41503)
I20260928 01:57:33.135191 12445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e01bb2faf0f46549a09f971bf6ef2e2" candidate_uuid: "3cf31b55f6f941a9a9759e7d971a6011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e256f95dcf414d9d870b13e21fcd1734" is_pre_election: true
I20260928 01:57:33.135394 12576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e01bb2faf0f46549a09f971bf6ef2e2" candidate_uuid: "3cf31b55f6f941a9a9759e7d971a6011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6209579c03e48dbb59a247e1d0ed07c" is_pre_election: true
I20260928 01:57:33.135465 12445 raft_consensus.cc:2468] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf31b55f6f941a9a9759e7d971a6011 in term 0.
I20260928 01:57:33.135635 12576 raft_consensus.cc:2468] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf31b55f6f941a9a9759e7d971a6011 in term 0.
I20260928 01:57:33.135931 12250 leader_election.cc:304] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf31b55f6f941a9a9759e7d971a6011, e256f95dcf414d9d870b13e21fcd1734; no voters:
I20260928 01:57:33.136200 12651 raft_consensus.cc:2804] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 01:57:33.136270 12651 raft_consensus.cc:493] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 01:57:33.136315 12651 raft_consensus.cc:3060] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [term 0 FOLLOWER]: Advancing to term 1
I20260928 01:57:33.137192 12651 raft_consensus.cc:515] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e256f95dcf414d9d870b13e21fcd1734" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 39611 } } peers { permanent_uuid: "a6209579c03e48dbb59a247e1d0ed07c" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41503 } } peers { permanent_uuid: "3cf31b55f6f941a9a9759e7d971a6011" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43331 } }
I20260928 01:57:33.137550 12651 leader_election.cc:290] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [CANDIDATE]: Term 1 election: Requested vote from peers e256f95dcf414d9d870b13e21fcd1734 (127.9.83.2:39611), a6209579c03e48dbb59a247e1d0ed07c (127.9.83.3:41503)
I20260928 01:57:33.137945 12576 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e01bb2faf0f46549a09f971bf6ef2e2" candidate_uuid: "3cf31b55f6f941a9a9759e7d971a6011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6209579c03e48dbb59a247e1d0ed07c"
I20260928 01:57:33.137957 12445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0e01bb2faf0f46549a09f971bf6ef2e2" candidate_uuid: "3cf31b55f6f941a9a9759e7d971a6011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e256f95dcf414d9d870b13e21fcd1734"
I20260928 01:57:33.138059 12445 raft_consensus.cc:3060] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734 [term 0 FOLLOWER]: Advancing to term 1
I20260928 01:57:33.138067 12576 raft_consensus.cc:3060] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c [term 0 FOLLOWER]: Advancing to term 1
I20260928 01:57:33.138928 12445 raft_consensus.cc:2468] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf31b55f6f941a9a9759e7d971a6011 in term 1.
I20260928 01:57:33.138940 12576 raft_consensus.cc:2468] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf31b55f6f941a9a9759e7d971a6011 in term 1.
I20260928 01:57:33.139223 12250 leader_election.cc:304] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf31b55f6f941a9a9759e7d971a6011, e256f95dcf414d9d870b13e21fcd1734; no voters:
I20260928 01:57:33.139392 12651 raft_consensus.cc:2804] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 01:57:33.139628 12651 raft_consensus.cc:697] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [term 1 LEADER]: Becoming Leader. State: Replica: 3cf31b55f6f941a9a9759e7d971a6011, State: Running, Role: LEADER
I20260928 01:57:33.139961 12651 consensus_queue.cc:237] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e256f95dcf414d9d870b13e21fcd1734" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 39611 } } peers { permanent_uuid: "a6209579c03e48dbb59a247e1d0ed07c" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41503 } } peers { permanent_uuid: "3cf31b55f6f941a9a9759e7d971a6011" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43331 } }
I20260928 01:57:33.143667 12171 catalog_manager.cc:5838] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cf31b55f6f941a9a9759e7d971a6011 (127.9.83.1). New cstate: current_term: 1 leader_uuid: "3cf31b55f6f941a9a9759e7d971a6011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e256f95dcf414d9d870b13e21fcd1734" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 39611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6209579c03e48dbb59a247e1d0ed07c" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cf31b55f6f941a9a9759e7d971a6011" member_type: VOTER last_known_addr { host: "127.9.83.1" port: 43331 } health_report { overall_health: HEALTHY } } }
I20260928 01:57:33.182811 12445 raft_consensus.cc:1275] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734 [term 1 FOLLOWER]: Refusing update from remote peer 3cf31b55f6f941a9a9759e7d971a6011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 01:57:33.182811 12576 raft_consensus.cc:1275] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c [term 1 FOLLOWER]: Refusing update from remote peer 3cf31b55f6f941a9a9759e7d971a6011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 01:57:33.183492 12654 consensus_queue.cc:1048] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6209579c03e48dbb59a247e1d0ed07c" member_type: VOTER last_known_addr { host: "127.9.83.3" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 01:57:33.183656 12651 consensus_queue.cc:1048] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e256f95dcf414d9d870b13e21fcd1734" member_type: VOTER last_known_addr { host: "127.9.83.2" port: 39611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 01:57:33.192507 12663 mvcc.cc:204] Tried to move back new op lower bound from 7334136435437318144 to 7334136435264028672. Current Snapshot: MvccSnapshot[applied={T|T < 7334136435430035456}]
I20260928 01:57:33.194247 12662 mvcc.cc:204] Tried to move back new op lower bound from 7334136435437318144 to 7334136435264028672. Current Snapshot: MvccSnapshot[applied={T|T < 7334136435430035456}]
I20260928 01:57:33.957048 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:34.051110 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:34.120491 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:34.347190 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.390s user 0.239s sys 0.012s Metrics: {"bytes_written":301415,"cfile_init":1,"compiler_manager_pool.queue_time_us":13332,"dirs.queue_time_us":26738,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":1976,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1324,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10400,"thread_start_us":191,"threads_started":2}
I20260928 01:57:34.348387 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 96447 bytes on disk
I20260928 01:57:34.349121 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20260928 01:57:34.673900 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.622s user 0.265s sys 0.008s Metrics: {"bytes_written":321208,"cfile_init":1,"compiler_manager_pool.queue_time_us":8651,"dirs.queue_time_us":9409,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":728,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1419,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11150,"thread_start_us":165,"threads_started":2}
I20260928 01:57:34.675076 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 103918 bytes on disk
I20260928 01:57:34.675729 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20260928 01:57:34.685405 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.564s user 0.321s sys 0.004s Metrics: {"bytes_written":330697,"cfile_init":1,"compiler_manager_pool.queue_time_us":68585,"dirs.queue_time_us":20737,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1031,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1592,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11500,"thread_start_us":208,"threads_started":2}
I20260928 01:57:34.686455 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 107324 bytes on disk
I20260928 01:57:34.687407 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20260928 01:57:35.103751 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:35.176837 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:35.336344 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.232s user 0.185s sys 0.000s Metrics: {"bytes_written":274940,"cfile_init":1,"dirs.queue_time_us":9374,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":11348,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1272,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9550,"thread_start_us":81,"threads_started":1}
I20260928 01:57:35.339721 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 89354 bytes on disk
I20260928 01:57:35.342315 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20260928 01:57:35.438597 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:35.449519 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.272s user 0.181s sys 0.004s Metrics: {"bytes_written":275021,"cfile_init":1,"dirs.queue_time_us":192,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":2784,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7141,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"rows_written":9550,"spinlock_wait_cycles":127744,"thread_start_us":90,"threads_started":1}
I20260928 01:57:35.450443 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 89142 bytes on disk
I20260928 01:57:35.450930 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20260928 01:57:35.737073 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.298s user 0.209s sys 0.003s Metrics: {"bytes_written":319945,"cfile_init":1,"dirs.queue_time_us":6278,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":870,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1465,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11100,"thread_start_us":78,"threads_started":1}
I20260928 01:57:35.738505 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 103773 bytes on disk
I20260928 01:57:35.739192 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20260928 01:57:36.094432 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:36.202162 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:36.470511 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.376s user 0.188s sys 0.004s Metrics: {"bytes_written":286877,"cfile_init":1,"dirs.queue_time_us":1201,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1701,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1232,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10000,"spinlock_wait_cycles":151936,"thread_start_us":89,"threads_started":1}
I20260928 01:57:36.471257 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 93047 bytes on disk
I20260928 01:57:36.471676 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20260928 01:57:36.490509 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:36.573997 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.372s user 0.197s sys 0.000s Metrics: {"bytes_written":292073,"cfile_init":1,"dirs.queue_time_us":1240,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":27515,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1329,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10200,"thread_start_us":82,"threads_started":1}
I20260928 01:57:36.574679 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 94634 bytes on disk
I20260928 01:57:36.575464 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20260928 01:57:36.738505 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.248s user 0.211s sys 0.000s Metrics: {"bytes_written":318837,"cfile_init":1,"dirs.queue_time_us":1393,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":5971,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1423,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11050,"thread_start_us":77,"threads_started":1}
I20260928 01:57:36.744374 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 102313 bytes on disk
I20260928 01:57:36.746009 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20260928 01:57:37.224080 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:37.326959 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:37.507227 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:37.663777 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.440s user 0.220s sys 0.000s Metrics: {"bytes_written":338181,"cfile_init":1,"dirs.queue_time_us":22091,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1295,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3452,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":11800,"spinlock_wait_cycles":150784,"thread_start_us":128,"threads_started":1}
I20260928 01:57:37.666760 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 110932 bytes on disk
I20260928 01:57:37.667289 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20260928 01:57:37.770972 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.444s user 0.218s sys 0.000s Metrics: {"bytes_written":333033,"cfile_init":1,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1426,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11600,"thread_start_us":75,"threads_started":1}
I20260928 01:57:37.771667 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 108683 bytes on disk
I20260928 01:57:37.772259 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20260928 01:57:37.898366 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.391s user 0.202s sys 0.000s Metrics: {"bytes_written":308042,"cfile_init":1,"dirs.queue_time_us":229,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":700,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1424,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10650,"spinlock_wait_cycles":3415424,"thread_start_us":79,"threads_started":1}
I20260928 01:57:37.899192 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 98314 bytes on disk
I20260928 01:57:37.899652 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20260928 01:57:38.427131 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:38.523591 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:38.651181 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:38.752686 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.325s user 0.212s sys 0.001s Metrics: {"bytes_written":318881,"cfile_init":1,"dirs.queue_time_us":201,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":5475,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1328,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11050,"thread_start_us":92,"threads_started":1}
I20260928 01:57:38.753470 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 103230 bytes on disk
I20260928 01:57:38.753966 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20260928 01:57:38.864476 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.341s user 0.214s sys 0.000s Metrics: {"bytes_written":318914,"cfile_init":1,"dirs.queue_time_us":1328,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":22214,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":17924,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":11050,"thread_start_us":83,"threads_started":1}
I20260928 01:57:38.865255 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 102976 bytes on disk
I20260928 01:57:38.865779 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20260928 01:57:38.947342 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.296s user 0.192s sys 0.001s Metrics: {"bytes_written":290950,"cfile_init":1,"dirs.queue_time_us":4829,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":935,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1346,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10150,"thread_start_us":79,"threads_started":1}
I20260928 01:57:38.948042 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 93783 bytes on disk
I20260928 01:57:38.948716 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20260928 01:57:39.505465 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:39.617281 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:39.700323 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:39.911214 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.406s user 0.187s sys 0.001s Metrics: {"bytes_written":280245,"cfile_init":1,"dirs.queue_time_us":13188,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1616,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1350,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750,"thread_start_us":104,"threads_started":1}
I20260928 01:57:39.911918 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 90042 bytes on disk
I20260928 01:57:39.920382 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":972,"lbm_reads_lt_1ms":4}
I20260928 01:57:40.018175 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.401s user 0.192s sys 0.000s Metrics: {"bytes_written":290913,"cfile_init":1,"dirs.queue_time_us":1988,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":4320,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1261,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10150,"thread_start_us":72,"threads_started":1}
I20260928 01:57:40.022707 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 93581 bytes on disk
I20260928 01:57:40.023168 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20260928 01:57:40.284801 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.584s user 0.183s sys 0.002s Metrics: {"bytes_written":273737,"cfile_init":1,"dirs.queue_time_us":8742,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":2362,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1506,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9500,"spinlock_wait_cycles":129280,"thread_start_us":91,"threads_started":1}
I20260928 01:57:40.285630 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 87561 bytes on disk
I20260928 01:57:40.286303 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20260928 01:57:40.671957 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:40.774817 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:40.787570 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:41.146706 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.475s user 0.202s sys 0.004s Metrics: {"bytes_written":312158,"cfile_init":1,"dirs.queue_time_us":27263,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":6983,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7663,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10800,"spinlock_wait_cycles":131328,"thread_start_us":91,"threads_started":1}
I20260928 01:57:41.147567 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 99787 bytes on disk
I20260928 01:57:41.148002 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20260928 01:57:41.187695 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.400s user 0.191s sys 0.008s Metrics: {"bytes_written":300435,"cfile_init":1,"dirs.queue_time_us":9997,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":9617,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4520,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10350,"spinlock_wait_cycles":135552,"thread_start_us":77,"threads_started":1}
I20260928 01:57:41.188581 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 95820 bytes on disk
I20260928 01:57:41.189781 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20260928 01:57:41.299563 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.525s user 0.195s sys 0.004s Metrics: {"bytes_written":300420,"cfile_init":1,"dirs.queue_time_us":15947,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1551,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1438,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10350,"thread_start_us":76,"threads_started":1}
I20260928 01:57:41.300444 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 95720 bytes on disk
I20260928 01:57:41.309924 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20260928 01:57:41.900028 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:41.945771 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:42.062652 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:42.279520 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.379s user 0.199s sys 0.001s Metrics: {"bytes_written":305698,"cfile_init":1,"dirs.queue_time_us":2127,"dirs.run_cpu_time_us":135,"dirs.run_wall_time_us":724,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1323,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":10550,"thread_start_us":77,"threads_started":1}
I20260928 01:57:42.280471 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 98023 bytes on disk
I20260928 01:57:42.281802 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20260928 01:57:42.401739 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.456s user 0.189s sys 0.000s Metrics: {"bytes_written":280298,"cfile_init":1,"dirs.queue_time_us":184,"dirs.run_cpu_time_us":144,"dirs.run_wall_time_us":1655,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1282,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9750,"spinlock_wait_cycles":131840,"thread_start_us":98,"threads_started":1}
I20260928 01:57:42.402652 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 90663 bytes on disk
I20260928 01:57:42.416379 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20260928 01:57:42.513082 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.450s user 0.205s sys 0.000s Metrics: {"bytes_written":317563,"cfile_init":1,"dirs.queue_time_us":151,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":759,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1366,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11000,"thread_start_us":76,"threads_started":1}
I20260928 01:57:42.514176 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 102986 bytes on disk
I20260928 01:57:42.514665 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20260928 01:57:43.168056 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:43.266220 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:43.283746 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2): perf score=1.000000
I20260928 01:57:43.577960 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.410s user 0.195s sys 0.004s Metrics: {"bytes_written":290862,"cfile_init":1,"dirs.queue_time_us":7403,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":961,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1237,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":10150,"thread_start_us":76,"threads_started":1}
I20260928 01:57:43.579412 12492 maintenance_manager.cc:419] P e256f95dcf414d9d870b13e21fcd1734: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 95150 bytes on disk
I20260928 01:57:43.584534 12382 maintenance_manager.cc:643] P e256f95dcf414d9d870b13e21fcd1734: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20260928 01:57:43.665784 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.399s user 0.195s sys 0.000s Metrics: {"bytes_written":281675,"cfile_init":1,"dirs.queue_time_us":1143,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":936,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1274,"lbm_writes_lt_1ms":32,"peak_mem_usage":0,"rows_written":9800,"thread_start_us":73,"threads_started":1}
I20260928 01:57:43.666782 12623 maintenance_manager.cc:419] P a6209579c03e48dbb59a247e1d0ed07c: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 91772 bytes on disk
I20260928 01:57:43.667364 12513 maintenance_manager.cc:643] P a6209579c03e48dbb59a247e1d0ed07c: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20260928 01:57:43.710222 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: FlushMRSOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.426s user 0.211s sys 0.001s Metrics: {"bytes_written":319832,"cfile_init":1,"dirs.queue_time_us":1134,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":6594,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1448,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":11100,"thread_start_us":74,"threads_started":1}
I20260928 01:57:43.711114 12361 maintenance_manager.cc:419] P 3cf31b55f6f941a9a9759e7d971a6011: Scheduling UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2): 104704 bytes on disk
I20260928 01:57:43.717060 12251 maintenance_manager.cc:643] P 3cf31b55f6f941a9a9759e7d971a6011: UndoDeltaBlockGCOp(0e01bb2faf0f46549a09f971bf6ef2e2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"spinlock_wait_cycles":768}
I20260928 01:57:43.988323 12171 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34342:
table { table_name: "test-workload" } modify_external_catalogs: true
I20260928 01:57:43.988488 12171 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34342:
table { table_name: "test-workload" } modify_external_catalogs: true
I20260928 01:57:43.991573 12171 catalog_manager.cc:6133] T 00000000000000000000000000000000 P d1d868702e2f4dc6afad51e179fce858: Sending DeleteTablet for 3 replicas of tablet 0e01bb2faf0f46549a09f971bf6ef2e2
I20260928 01:57:43.992403 12425 tablet_service.cc:1558] Processing DeleteTablet for tablet 0e01bb2faf0f46549a09f971bf6ef2e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 01:57:43 UTC) from {username='slave'} at 127.0.0.1:47124
I20260928 01:57:43.992545 12294 tablet_service.cc:1558] Processing DeleteTablet for tablet 0e01bb2faf0f46549a09f971bf6ef2e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 01:57:43 UTC) from {username='slave'} at 127.0.0.1:37908
I20260928 01:57:43.992601 12556 tablet_service.cc:1558] Processing DeleteTablet for tablet 0e01bb2faf0f46549a09f971bf6ef2e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 01:57:43 UTC) from {username='slave'} at 127.0.0.1:41456
I20260928 01:57:43.992841 12732 tablet_replica.cc:333] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734: stopping tablet replica
I20260928 01:57:43.992944 12733 tablet_replica.cc:333] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011: stopping tablet replica
I20260928 01:57:43.992942 12734 tablet_replica.cc:333] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c: stopping tablet replica
I20260928 01:57:43.993116 12732 raft_consensus.cc:2243] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 01:57:43.993172 12734 raft_consensus.cc:2243] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 01:57:43.993172 12733 raft_consensus.cc:2243] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [term 1 LEADER]: Raft consensus shutting down.
I20260928 01:57:43.993348 12732 raft_consensus.cc:2272] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 01:57:43.993384 12734 raft_consensus.cc:2272] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 01:57:43.993858 12733 raft_consensus.cc:2272] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 01:57:43.994499 12732 ts_tablet_manager.cc:1916] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 01:57:43.994968 12734 ts_tablet_manager.cc:1916] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 01:57:43.995215 12733 ts_tablet_manager.cc:1916] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 01:57:43.997824 12732 ts_tablet_manager.cc:1929] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2007
I20260928 01:57:43.997917 12732 log.cc:1199] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734: Deleting WAL directory at /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-1/wal/wals/0e01bb2faf0f46549a09f971bf6ef2e2
I20260928 01:57:43.999107 12732 ts_tablet_manager.cc:1950] T 0e01bb2faf0f46549a09f971bf6ef2e2 P e256f95dcf414d9d870b13e21fcd1734: Deleting consensus metadata
I20260928 01:57:43.999758 12159 catalog_manager.cc:5063] TS e256f95dcf414d9d870b13e21fcd1734 (127.9.83.2:39611): tablet 0e01bb2faf0f46549a09f971bf6ef2e2 (table test-workload [id=38ae7dc437a34dc0989e7a76083e7a01]) successfully deleted
I20260928 01:57:43.999812 12734 ts_tablet_manager.cc:1929] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2007
I20260928 01:57:44.000173 12734 log.cc:1199] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c: Deleting WAL directory at /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-2/wal/wals/0e01bb2faf0f46549a09f971bf6ef2e2
I20260928 01:57:44.000360 12733 ts_tablet_manager.cc:1929] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2007
I20260928 01:57:44.000514 12733 log.cc:1199] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011: Deleting WAL directory at /tmp/dist-test-task6vbGbg/test-tmp/delete_table-itest.4.Params_DeleteTableWhileScanInProgressParamTest.Test_1.1790560638442226-9548-0/minicluster-data/ts-0/wal/wals/0e01bb2faf0f46549a09f971bf6ef2e2
I20260928 01:57:44.001367 12734 ts_tablet_manager.cc:1950] T 0e01bb2faf0f46549a09f971bf6ef2e2 P a6209579c03e48dbb59a247e1d0ed07c: Deleting consensus metadata
I20260928 01:57:44.001672 12733 ts_tablet_manager.cc:1950] T 0e01bb2faf0f46549a09f971bf6ef2e2 P 3cf31b55f6f941a9a9759e7d971a6011: Deleting consensus metadata
I20260928 01:57:44.002872 12157 catalog_manager.cc:5063] TS a6209579c03e48dbb59a247e1d0ed07c (127.9.83.3:41503): tablet 0e01bb2faf0f46549a09f971bf6ef2e2 (table test-workload [id=38ae7dc437a34dc0989e7a76083e7a01]) successfully deleted
I20260928 01:57:44.004101 12156 catalog_manager.cc:5063] TS 3cf31b55f6f941a9a9759e7d971a6011 (127.9.83.1:43331): tablet 0e01bb2faf0f46549a09f971bf6ef2e2 (table test-workload [id=38ae7dc437a34dc0989e7a76083e7a01]) successfully deleted
I20260928 01:57:44.838830 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 12232
I20260928 01:57:44.846076 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 12363
I20260928 01:57:44.851883 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 12494
I20260928 01:57:44.857579 9548 external_mini_cluster.cc:1664] Killing /tmp/dist-test-task6vbGbg/build/debug/bin/kudu with pid 12141
2026-09-28T01:57:44Z chronyd exiting
[ OK ] Params/DeleteTableWhileScanInProgressParamTest.Test/1 (12567 ms)
[----------] 1 test from Params/DeleteTableWhileScanInProgressParamTest (12567 ms total)
[----------] Global test environment tear-down
[==========] 3 tests from 2 test suites ran. (26425 ms total)
[ PASSED ] 2 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] DeleteTableITest.TestNoDeleteTombstonedTablets
1 FAILED TEST