Diagnosed failure

UpdateScanDeltaCompactionTest.TestAll: Unrecognized error type. Please see the error log for more information.

Full log

[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from UpdateScanDeltaCompactionTest
[ RUN      ] UpdateScanDeltaCompactionTest.TestAll
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:52:32.732517 32601 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.214.126:36937
I20240502 01:52:32.734191 32601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:32.735550 32601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:32.747004 32608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.747622 32610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.748163 32607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.748653 32601 server_base.cc:1034] running on GCE node
I20240502 01:52:32.749495 32601 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:32.749652 32601 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:32.749722 32601 hybrid_clock.cc:648] HybridClock initialized: now 1714614752749705 us; error 0 us; skew 500 ppm
I20240502 01:52:32.750041 32601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:32.758519 32601 webserver.cc:460] Webserver started at http://127.31.214.126:38377/ using document root <none> and password file <none>
I20240502 01:52:32.759410 32601 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:32.759517 32601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:32.759876 32601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:32.763399 32601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOAnHmO/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714614752658155-32601-0/minicluster-data/master-0-root/instance:
uuid: "4548ee4662a24496ab6f455c26efb5c4"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-jw2p"
I20240502 01:52:32.769491 32601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20240502 01:52:32.772904 32615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:32.774466 32601 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240502 01:52:32.774649 32601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOAnHmO/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714614752658155-32601-0/minicluster-data/master-0-root
uuid: "4548ee4662a24496ab6f455c26efb5c4"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-jw2p"
I20240502 01:52:32.774894 32601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOAnHmO/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714614752658155-32601-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskOAnHmO/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714614752658155-32601-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskOAnHmO/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714614752658155-32601-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:32.799343 32601 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:32.800235 32601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:32.800529 32601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:32.820032 32601 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.126:36937
I20240502 01:52:32.820070 32666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.126:36937 every 8 connection(s)
I20240502 01:52:32.824009 32667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:32.837285 32667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4: Bootstrap starting.
I20240502 01:52:32.841275 32667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:32.842381 32667 log.cc:824] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:32.845264 32667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4: No bootstrap required, opened a new log
I20240502 01:52:32.852357 32667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4548ee4662a24496ab6f455c26efb5c4" member_type: VOTER }
I20240502 01:52:32.852727 32667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:32.852828 32667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4548ee4662a24496ab6f455c26efb5c4, State: Initialized, Role: FOLLOWER
I20240502 01:52:32.853601 32667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4548ee4662a24496ab6f455c26efb5c4" member_type: VOTER }
I20240502 01:52:32.853821 32667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:32.853926 32667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:32.854061 32667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:32.856132 32667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4548ee4662a24496ab6f455c26efb5c4" member_type: VOTER }
I20240502 01:52:32.856596 32667 leader_election.cc:302] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4548ee4662a24496ab6f455c26efb5c4; no voters: 
I20240502 01:52:32.857221 32667 leader_election.cc:288] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:52:32.857360 32670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:32.858071 32670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [term 1 LEADER]: Becoming Leader. State: Replica: 4548ee4662a24496ab6f455c26efb5c4, State: Running, Role: LEADER
I20240502 01:52:32.858712 32670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4548ee4662a24496ab6f455c26efb5c4" member_type: VOTER }
I20240502 01:52:32.859380 32667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:32.861933 32672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4548ee4662a24496ab6f455c26efb5c4. Latest consensus state: current_term: 1 leader_uuid: "4548ee4662a24496ab6f455c26efb5c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4548ee4662a24496ab6f455c26efb5c4" member_type: VOTER } }
I20240502 01:52:32.862279 32672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:32.862524 32671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4548ee4662a24496ab6f455c26efb5c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4548ee4662a24496ab6f455c26efb5c4" member_type: VOTER } }
I20240502 01:52:32.862784 32671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:32.864233 32679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:32.870115 32679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:32.870759 32601 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 01:52:32.880503 32679 catalog_manager.cc:1347] Generated new cluster ID: 6e3b164ae39b49ac8e36c930aea06f2e
I20240502 01:52:32.880664 32679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:32.889097 32679 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:32.890359 32679 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:32.904886 32679 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4548ee4662a24496ab6f455c26efb5c4: Generated new TSK 0
I20240502 01:52:32.905869 32679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:32.936554 32601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:32.941207 32689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.942510 32690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:32.942768 32692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:32.943518 32601 server_base.cc:1034] running on GCE node
I20240502 01:52:32.943874 32601 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 01:52:32.943987 32601 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 01:52:32.944057 32601 hybrid_clock.cc:648] HybridClock initialized: now 1714614752944031 us; error 0 us; skew 500 ppm
I20240502 01:52:32.944315 32601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:32.945883 32601 webserver.cc:460] Webserver started at http://127.31.214.65:41113/ using document root <none> and password file <none>
I20240502 01:52:32.946208 32601 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:32.946314 32601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:32.946493 32601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:32.947466 32601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOAnHmO/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714614752658155-32601-0/minicluster-data/ts-0-root/instance:
uuid: "0e9e50bf3b4a4811af65a83150bacc54"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-jw2p"
I20240502 01:52:32.951177 32601 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.001s	sys 0.003s
I20240502 01:52:32.953747 32697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:32.954190 32601 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.001s
I20240502 01:52:32.954310 32601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOAnHmO/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714614752658155-32601-0/minicluster-data/ts-0-root
uuid: "0e9e50bf3b4a4811af65a83150bacc54"
format_stamp: "Formatted at 2024-05-02 01:52:32 on dist-test-slave-jw2p"
I20240502 01:52:32.954460 32601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOAnHmO/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714614752658155-32601-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskOAnHmO/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714614752658155-32601-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskOAnHmO/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714614752658155-32601-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:32.997517 32601 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 01:52:32.998292 32601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:32.999311 32601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:33.000989 32601 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:33.001117 32601 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:33.001246 32601 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:33.001334 32601 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 01:52:33.022101 32601 rpc_server.cc:307] RPC server started. Bound to: 127.31.214.65:40491
I20240502 01:52:33.022140 32760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.214.65:40491 every 8 connection(s)
I20240502 01:52:33.042196 32761 heartbeater.cc:347] Connected to a master server at 127.31.214.126:36937
I20240502 01:52:33.042639 32761 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:33.043668 32761 heartbeater.cc:511] Master 127.31.214.126:36937 requested a full tablet report, sending...
I20240502 01:52:33.046027 32632 ts_manager.cc:194] Registered new tserver with Master: 0e9e50bf3b4a4811af65a83150bacc54 (127.31.214.65:40491)
I20240502 01:52:33.046209 32601 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.022502046s
I20240502 01:52:33.048070 32632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60490
I20240502 01:52:33.069535 32632 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60506:
name: "update-scan-delta-compact-tbl"
schema {
  columns {
    name: "key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20240502 01:52:33.098147 32726 tablet_service.cc:1467] Processing CreateTablet for tablet 98d308152131458cadc33411f7b54f22 (DEFAULT_TABLE table=update-scan-delta-compact-tbl [id=6453bdcef03f469084b39adbddf9f6f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:33.099021 32726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d308152131458cadc33411f7b54f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.106881   305 tablet_bootstrap.cc:492] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54: Bootstrap starting.
I20240502 01:52:33.108647   305 tablet_bootstrap.cc:654] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.110929   305 tablet_bootstrap.cc:492] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54: No bootstrap required, opened a new log
I20240502 01:52:33.111099   305 ts_tablet_manager.cc:1367] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.001s
I20240502 01:52:33.111774   305 raft_consensus.cc:348] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9e50bf3b4a4811af65a83150bacc54" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 40491 } }
I20240502 01:52:33.111972   305 raft_consensus.cc:374] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.112074   305 raft_consensus.cc:729] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e9e50bf3b4a4811af65a83150bacc54, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.112285   305 consensus_queue.cc:260] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9e50bf3b4a4811af65a83150bacc54" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 40491 } }
I20240502 01:52:33.112452   305 raft_consensus.cc:388] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:33.112538   305 raft_consensus.cc:482] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:33.112651   305 raft_consensus.cc:3037] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.114607   305 raft_consensus.cc:504] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9e50bf3b4a4811af65a83150bacc54" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 40491 } }
I20240502 01:52:33.114897   305 leader_election.cc:302] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e9e50bf3b4a4811af65a83150bacc54; no voters: 
I20240502 01:52:33.115425   305 leader_election.cc:288] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 01:52:33.115602   307 raft_consensus.cc:2781] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.116410   305 ts_tablet_manager.cc:1398] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.003s
I20240502 01:52:33.116514 32761 heartbeater.cc:503] Master 127.31.214.126:36937 was elected leader, sending a full tablet report...
I20240502 01:52:33.117496   307 raft_consensus.cc:686] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54 [term 1 LEADER]: Becoming Leader. State: Replica: 0e9e50bf3b4a4811af65a83150bacc54, State: Running, Role: LEADER
I20240502 01:52:33.117913   307 consensus_queue.cc:237] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9e50bf3b4a4811af65a83150bacc54" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 40491 } }
I20240502 01:52:33.122184 32632 catalog_manager.cc:5484] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e9e50bf3b4a4811af65a83150bacc54 (127.31.214.65). New cstate: current_term: 1 leader_uuid: "0e9e50bf3b4a4811af65a83150bacc54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e9e50bf3b4a4811af65a83150bacc54" member_type: VOTER last_known_addr { host: "127.31.214.65" port: 40491 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:37.024830 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling FlushMRSOp(98d308152131458cadc33411f7b54f22): perf score=1.000000
I20240502 01:52:39.473228 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: FlushMRSOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 2.447s	user 2.138s	sys 0.279s Metrics: {"bytes_written":164229,"cfile_init":1,"compiler_manager_pool.queue_time_us":434,"compiler_manager_pool.run_cpu_time_us":596426,"compiler_manager_pool.run_wall_time_us":600315,"dirs.queue_time_us":410,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":1022,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7110,"lbm_writes_lt_1ms":86,"peak_mem_usage":0,"rows_written":59001,"thread_start_us":599,"threads_started":2}
I20240502 01:52:39.475191 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22): 396269 bytes on disk
I20240502 01:52:39.475999 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240502 01:52:41.477146 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling FlushMRSOp(98d308152131458cadc33411f7b54f22): perf score=2.874778
I20240502 01:52:43.663153 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: FlushMRSOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 2.186s	user 2.092s	sys 0.085s Metrics: {"bytes_written":199818,"cfile_init":1,"dirs.queue_time_us":448,"dirs.run_cpu_time_us":283,"dirs.run_wall_time_us":1841,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8612,"lbm_writes_lt_1ms":106,"peak_mem_usage":0,"rows_written":75000,"spinlock_wait_cycles":2944,"thread_start_us":279,"threads_started":1}
I20240502 01:52:43.664572 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22): 503968 bytes on disk
I20240502 01:52:43.665176 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240502 01:52:43.665791 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling CompactRowSetsOp(98d308152131458cadc33411f7b54f22): perf score=0.014661
I20240502 01:52:45.914034 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: CompactRowSetsOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 2.248s	user 2.176s	sys 0.064s Metrics: {"bytes_written":370783,"cfile_cache_hit":10,"cfile_cache_hit_bytes":1746,"cfile_cache_miss":164,"cfile_cache_miss_bytes":1873530,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":2337,"dirs.run_cpu_time_us":817,"dirs.run_wall_time_us":4711,"drs_written":1,"lbm_read_time_us":7690,"lbm_reads_lt_1ms":184,"lbm_write_time_us":13697,"lbm_writes_lt_1ms":175,"num_input_rowsets":2,"peak_mem_usage":1148235,"rows_written":134001,"thread_start_us":1897,"threads_started":5}
I20240502 01:52:45.914942 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22): 898762 bytes on disk
I20240502 01:52:45.915627 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 01:52:45.916124 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling FlushMRSOp(98d308152131458cadc33411f7b54f22): perf score=1.882590
I20240502 01:52:48.034480 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: FlushMRSOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 2.118s	user 2.037s	sys 0.052s Metrics: {"bytes_written":186520,"cfile_init":1,"dirs.queue_time_us":418,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":936,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8186,"lbm_writes_lt_1ms":100,"peak_mem_usage":0,"rows_written":70000,"thread_start_us":277,"threads_started":1}
I20240502 01:52:48.035430 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling CompactRowSetsOp(98d308152131458cadc33411f7b54f22): perf score=0.014483
I20240502 01:52:51.715920 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: CompactRowSetsOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 3.680s	user 3.488s	sys 0.107s Metrics: {"bytes_written":563590,"cfile_cache_hit":10,"cfile_cache_hit_bytes":2483,"cfile_cache_miss":248,"cfile_cache_miss_bytes":2859517,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":4888,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1682,"drs_written":1,"lbm_read_time_us":13200,"lbm_reads_lt_1ms":272,"lbm_write_time_us":19228,"lbm_writes_lt_1ms":255,"num_input_rowsets":2,"peak_mem_usage":2139715,"rows_written":204001,"thread_start_us":4498,"threads_started":7}
I20240502 01:52:51.716742 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22): 1369287 bytes on disk
I20240502 01:52:51.717373 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240502 01:52:51.717890 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling FlushMRSOp(98d308152131458cadc33411f7b54f22): perf score=4.859153
W20240502 01:52:53.939741   310 log.cc:924] Time spent T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20240502 01:52:54.566831 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: FlushMRSOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 2.849s	user 2.670s	sys 0.116s Metrics: {"bytes_written":236466,"cfile_init":1,"dirs.queue_time_us":432,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":170640,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10755,"lbm_writes_lt_1ms":120,"peak_mem_usage":0,"rows_written":88000,"thread_start_us":299,"threads_started":1}
I20240502 01:52:54.568063 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22): 589092 bytes on disk
I20240502 01:52:54.568773 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240502 01:52:54.569550 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling CompactRowSetsOp(98d308152131458cadc33411f7b54f22): perf score=0.014260
I20240502 01:52:59.573801 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: CompactRowSetsOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 5.004s	user 4.783s	sys 0.200s Metrics: {"bytes_written":803464,"cfile_cache_hit":10,"cfile_cache_hit_bytes":3442,"cfile_cache_miss":344,"cfile_cache_miss_bytes":4092900,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":2502,"dirs.run_cpu_time_us":281,"dirs.run_wall_time_us":1789,"drs_written":1,"lbm_read_time_us":16933,"lbm_reads_lt_1ms":364,"lbm_write_time_us":27776,"lbm_writes_lt_1ms":355,"num_input_rowsets":2,"peak_mem_usage":3277136,"rows_written":292001,"thread_start_us":1757,"threads_started":7}
I20240502 01:52:59.574844 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling FlushMRSOp(98d308152131458cadc33411f7b54f22): perf score=8.827903
I20240502 01:53:03.250258 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: FlushMRSOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 3.675s	user 3.467s	sys 0.204s Metrics: {"bytes_written":324693,"cfile_init":1,"dirs.queue_time_us":458,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":950,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":13642,"lbm_writes_lt_1ms":157,"peak_mem_usage":0,"rows_written":121000,"thread_start_us":332,"threads_started":1}
I20240502 01:53:03.251138 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22): 2773684 bytes on disk
I20240502 01:53:03.251827 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":121,"lbm_reads_lt_1ms":8}
I20240502 01:53:03.252386 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling CompactRowSetsOp(98d308152131458cadc33411f7b54f22): perf score=0.013953
I20240502 01:53:10.127406 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: CompactRowSetsOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 6.875s	user 6.670s	sys 0.173s Metrics: {"bytes_written":1137427,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4794,"cfile_cache_miss":478,"cfile_cache_miss_bytes":5800892,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":2445,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":949,"drs_written":1,"lbm_read_time_us":23499,"lbm_reads_lt_1ms":498,"lbm_write_time_us":38433,"lbm_writes_lt_1ms":496,"num_input_rowsets":2,"peak_mem_usage":4707037,"rows_written":413001,"thread_start_us":1709,"threads_started":7}
I20240502 01:53:10.128271 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22): 2776582 bytes on disk
I20240502 01:53:10.128918 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240502 01:53:10.129480 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling FlushMRSOp(98d308152131458cadc33411f7b54f22): perf score=14.781028
I20240502 01:53:15.084900 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: FlushMRSOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 4.955s	user 4.757s	sys 0.194s Metrics: {"bytes_written":433634,"cfile_init":1,"dirs.queue_time_us":892,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":89,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":19107,"lbm_writes_lt_1ms":205,"peak_mem_usage":0,"rows_written":162000,"thread_start_us":782,"threads_started":1}
I20240502 01:53:15.086009 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling FlushMRSOp(98d308152131458cadc33411f7b54f22): perf score=2.874778
I20240502 01:53:16.999655 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: FlushMRSOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 1.913s	user 1.873s	sys 0.012s Metrics: {"bytes_written":201139,"cfile_init":1,"dirs.queue_time_us":428,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":869,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8244,"lbm_writes_lt_1ms":106,"peak_mem_usage":0,"rows_written":75000,"thread_start_us":304,"threads_started":1}
I20240502 01:53:17.000661 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling CompactRowSetsOp(98d308152131458cadc33411f7b54f22): perf score=0.022353
I20240502 01:53:27.607421 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: CompactRowSetsOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 10.606s	user 10.263s	sys 0.321s Metrics: {"bytes_written":1791133,"cfile_cache_hit":15,"cfile_cache_hit_bytes":7500,"cfile_cache_miss":759,"cfile_cache_miss_bytes":9129704,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":2517,"dirs.run_cpu_time_us":331,"dirs.run_wall_time_us":1936,"drs_written":1,"lbm_read_time_us":33049,"lbm_reads_lt_1ms":795,"lbm_write_time_us":59153,"lbm_writes_lt_1ms":770,"num_input_rowsets":3,"peak_mem_usage":6705931,"rows_written":650001,"thread_start_us":1815,"threads_started":7}
I20240502 01:53:27.608239 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling FlushMRSOp(98d308152131458cadc33411f7b54f22): perf score=19.741965
I20240502 01:53:33.521284 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: FlushMRSOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 5.913s	user 5.737s	sys 0.154s Metrics: {"bytes_written":532836,"cfile_init":1,"dirs.queue_time_us":399,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":989,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":21237,"lbm_writes_lt_1ms":250,"peak_mem_usage":0,"rows_written":200000,"thread_start_us":255,"threads_started":1}
I20240502 01:53:33.522234 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22): 5712589 bytes on disk
I20240502 01:53:33.522944 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":152,"lbm_reads_lt_1ms":8}
I20240502 01:53:33.523496 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling FlushMRSOp(98d308152131458cadc33411f7b54f22): perf score=4.859153
I20240502 01:53:36.477743 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: FlushMRSOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 2.954s	user 2.795s	sys 0.114s Metrics: {"bytes_written":246498,"cfile_init":1,"dirs.queue_time_us":462,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":1033,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10623,"lbm_writes_lt_1ms":124,"peak_mem_usage":0,"rows_written":92000,"thread_start_us":283,"threads_started":1}
I20240502 01:53:36.478760 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling LogGCOp(98d308152131458cadc33411f7b54f22): free 8349105 bytes of WAL
I20240502 01:53:36.479162 32703 log_reader.cc:386] T 98d308152131458cadc33411f7b54f22: removed 1 log segments from log reader
I20240502 01:53:36.479367 32703 log.cc:1076] T 98d308152131458cadc33411f7b54f22 P 0e9e50bf3b4a4811af65a83150bacc54: Deleting log segment in path: /tmp/dist-test-taskOAnHmO/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714614752658155-32601-0/minicluster-data/ts-0-root/wals/98d308152131458cadc33411f7b54f22/wal-000000001 (ops 1-863)
I20240502 01:53:36.481248 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: LogGCOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 0.002s	user 0.000s	sys 0.000s Metrics: {}
I20240502 01:53:36.481662 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22): 615518 bytes on disk
I20240502 01:53:36.482215 32703 maintenance_manager.cc:643] P 0e9e50bf3b4a4811af65a83150bacc54: UndoDeltaBlockGCOp(98d308152131458cadc33411f7b54f22) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240502 01:53:36.483106 32762 maintenance_manager.cc:419] P 0e9e50bf3b4a4811af65a83150bacc54: Scheduling CompactRowSetsOp(98d308152131458cadc33411f7b54f22): perf score=0.021613
I20240502 01:53:37.375771 32601 update_scan_delta_compact-test.cc:209] Time spent Insert: real 64.244s	user 58.902s	sys 4.703s
I20240502 01:53:37.396073   361 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:37.400825   361 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:37.404299   361 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:37.407824   361 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:37.412267   361 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:37.415964   361 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:37.419543   361 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:37.423285   361 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:37.426905   361 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:37.430423   361 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:37.434093   361 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:37.437490   361 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 01:53:37.441445   361 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240502 01:53:37.456993   361 monotime.cc:342] Check failed: t_beg.Initialized() 
*** Check failure stack trace: ***
*** Aborted at 1714614817 (unix time) try "date -d @1714614817" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGABRT (@0x3e800007f59) received by PID 32601 (TID 0x7f85ea661700) from PID 32601; stack trace: ***
    @     0x7f861ed80980 (unknown) at ??:0
    @     0x7f8617e2bfb7 gsignal at ??:0
    @     0x7f8617e2d921 abort at ??:0
    @     0x7f861ab54dcd google::LogMessage::Fail() at ??:0
    @     0x7f861ab58b93 google::LogMessage::SendToLog() at ??:0
    @     0x7f861ab547cc google::LogMessage::Flush() at ??:0
    @     0x7f861ab55f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f861c0b9c80 kudu::operator-() at ??:0
    @     0x7f86278da14a kudu::tablet::WriteOp::ToString() at ??:0
    @     0x7f86278a45de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
    @     0x7f86278a4336 kudu::tablet::OpDriver::ToString() at ??:0
    @     0x7f86278512d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
    @     0x7f862e6cfe47 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
    @     0x7f862e6de9fc kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
    @     0x7f862e6de862 std::_Function_handler<>::_M_invoke() at ??:0
    @     0x7f8624a3c02a std::function<>::operator()() at ??:0
    @     0x7f8624a347ba kudu::Webserver::RunPathHandler() at ??:0
    @     0x7f8624a2f7f1 kudu::Webserver::BeginRequestCallback() at ??:0
    @     0x7f8624a2d927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
    @     0x7f8624a6b089 handle_request at ??:0
    @     0x7f8624a6d76b worker_thread at ??:0
    @     0x7f861ed756db start_thread at ??:0
    @     0x7f8617f0e71f clone at ??:0