Diagnosed failure

UpdateScanDeltaCompactionTest.TestAll: Unrecognized error type. Please see the error log for more information.

Full log

[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from UpdateScanDeltaCompactionTest
[ RUN      ] UpdateScanDeltaCompactionTest.TestAll
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240426 09:57:41.753000  6632 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.6.122.62:39121
I20240426 09:57:41.754592  6632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240426 09:57:41.755849  6632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:57:41.766661  6638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:57:41.766911  6637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:57:41.767205  6640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:57:41.767616  6632 server_base.cc:1034] running on GCE node
I20240426 09:57:41.768293  6632 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 09:57:41.768441  6632 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:57:41.768505  6632 hybrid_clock.cc:648] HybridClock initialized: now 1714125461768490 us; error 0 us; skew 500 ppm
I20240426 09:57:41.768783  6632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:57:41.771157  6632 webserver.cc:460] Webserver started at http://127.6.122.62:37777/ using document root <none> and password file <none>
I20240426 09:57:41.771872  6632 fs_manager.cc:359] Metadata directory not provided
I20240426 09:57:41.771960  6632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:57:41.772243  6632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:57:41.775381  6632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBejrXv/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714125461681665-6632-0/minicluster-data/master-0-root/instance:
uuid: "eee90642c9f24159a89d06c361990dc8"
format_stamp: "Formatted at 2024-04-26 09:57:41 on dist-test-slave-3469"
I20240426 09:57:41.780638  6632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240426 09:57:41.783665  6645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:57:41.784977  6632 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240426 09:57:41.785140  6632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBejrXv/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714125461681665-6632-0/minicluster-data/master-0-root
uuid: "eee90642c9f24159a89d06c361990dc8"
format_stamp: "Formatted at 2024-04-26 09:57:41 on dist-test-slave-3469"
I20240426 09:57:41.785305  6632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBejrXv/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714125461681665-6632-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBejrXv/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714125461681665-6632-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBejrXv/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714125461681665-6632-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:57:41.799633  6632 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 09:57:41.800318  6632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240426 09:57:41.800577  6632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:57:41.816807  6632 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.62:39121
I20240426 09:57:41.816835  6696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.122.62:39121 every 8 connection(s)
I20240426 09:57:41.820030  6697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:57:41.831817  6697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8: Bootstrap starting.
I20240426 09:57:41.835007  6697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8: Neither blocks nor log segments found. Creating new log.
I20240426 09:57:41.835969  6697 log.cc:824] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8: Log is configured to *not* fsync() on all Append() calls
I20240426 09:57:41.838275  6697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8: No bootstrap required, opened a new log
I20240426 09:57:41.844650  6697 raft_consensus.cc:348] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee90642c9f24159a89d06c361990dc8" member_type: VOTER }
I20240426 09:57:41.844913  6697 raft_consensus.cc:374] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:57:41.845008  6697 raft_consensus.cc:729] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee90642c9f24159a89d06c361990dc8, State: Initialized, Role: FOLLOWER
I20240426 09:57:41.845675  6697 consensus_queue.cc:260] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee90642c9f24159a89d06c361990dc8" member_type: VOTER }
I20240426 09:57:41.845860  6697 raft_consensus.cc:388] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 09:57:41.845974  6697 raft_consensus.cc:482] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 09:57:41.846108  6697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:57:41.848784  6697 raft_consensus.cc:504] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee90642c9f24159a89d06c361990dc8" member_type: VOTER }
I20240426 09:57:41.849337  6697 leader_election.cc:302] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eee90642c9f24159a89d06c361990dc8; no voters: 
I20240426 09:57:41.853408  6697 leader_election.cc:288] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240426 09:57:41.853504  6700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:57:41.854910  6700 raft_consensus.cc:686] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [term 1 LEADER]: Becoming Leader. State: Replica: eee90642c9f24159a89d06c361990dc8, State: Running, Role: LEADER
I20240426 09:57:41.855558  6700 consensus_queue.cc:237] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee90642c9f24159a89d06c361990dc8" member_type: VOTER }
I20240426 09:57:41.855818  6697 sys_catalog.cc:564] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [sys.catalog]: configured and running, proceeding with master startup.
I20240426 09:57:41.858592  6702 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eee90642c9f24159a89d06c361990dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee90642c9f24159a89d06c361990dc8" member_type: VOTER } }
I20240426 09:57:41.858564  6701 sys_catalog.cc:455] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eee90642c9f24159a89d06c361990dc8. Latest consensus state: current_term: 1 leader_uuid: "eee90642c9f24159a89d06c361990dc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eee90642c9f24159a89d06c361990dc8" member_type: VOTER } }
I20240426 09:57:41.858876  6701 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [sys.catalog]: This master's current role is: LEADER
I20240426 09:57:41.858875  6702 sys_catalog.cc:458] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8 [sys.catalog]: This master's current role is: LEADER
I20240426 09:57:41.860752  6707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240426 09:57:41.866128  6707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240426 09:57:41.867164  6632 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240426 09:57:41.875397  6707 catalog_manager.cc:1347] Generated new cluster ID: 240fddcb7ad54b709c08640f958b6ed1
I20240426 09:57:41.875510  6707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240426 09:57:41.898322  6707 catalog_manager.cc:1370] Generated new certificate authority record
I20240426 09:57:41.899928  6707 catalog_manager.cc:1504] Loading token signing keys...
I20240426 09:57:41.909372  6707 catalog_manager.cc:5857] T 00000000000000000000000000000000 P eee90642c9f24159a89d06c361990dc8: Generated new TSK 0
I20240426 09:57:41.910413  6707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240426 09:57:41.932339  6632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240426 09:57:41.936589  6719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:57:41.937028  6720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240426 09:57:41.938320  6722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240426 09:57:41.938759  6632 server_base.cc:1034] running on GCE node
I20240426 09:57:41.939043  6632 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240426 09:57:41.939126  6632 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240426 09:57:41.939188  6632 hybrid_clock.cc:648] HybridClock initialized: now 1714125461939164 us; error 0 us; skew 500 ppm
I20240426 09:57:41.939363  6632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240426 09:57:41.940735  6632 webserver.cc:460] Webserver started at http://127.6.122.1:44277/ using document root <none> and password file <none>
I20240426 09:57:41.940984  6632 fs_manager.cc:359] Metadata directory not provided
I20240426 09:57:41.941066  6632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240426 09:57:41.941231  6632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240426 09:57:41.941967  6632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBejrXv/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714125461681665-6632-0/minicluster-data/ts-0-root/instance:
uuid: "d96d122635cc460c97a1f42d54ed32dc"
format_stamp: "Formatted at 2024-04-26 09:57:41 on dist-test-slave-3469"
I20240426 09:57:41.944649  6632 fs_manager.cc:691] Time spent creating directory manager: real 0.002s	user 0.003s	sys 0.000s
I20240426 09:57:41.946446  6727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:57:41.946854  6632 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
I20240426 09:57:41.946985  6632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBejrXv/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714125461681665-6632-0/minicluster-data/ts-0-root
uuid: "d96d122635cc460c97a1f42d54ed32dc"
format_stamp: "Formatted at 2024-04-26 09:57:41 on dist-test-slave-3469"
I20240426 09:57:41.947126  6632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBejrXv/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714125461681665-6632-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBejrXv/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714125461681665-6632-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBejrXv/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714125461681665-6632-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240426 09:57:41.963356  6632 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240426 09:57:41.964017  6632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240426 09:57:41.964867  6632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240426 09:57:41.966367  6632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240426 09:57:41.966498  6632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:57:41.966606  6632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240426 09:57:41.966676  6632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240426 09:57:41.987185  6632 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.1:38637
I20240426 09:57:41.987229  6789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.122.1:38637 every 8 connection(s)
I20240426 09:57:42.002972  6790 heartbeater.cc:347] Connected to a master server at 127.6.122.62:39121
I20240426 09:57:42.003269  6790 heartbeater.cc:464] Registering TS with master...
I20240426 09:57:42.003969  6790 heartbeater.cc:511] Master 127.6.122.62:39121 requested a full tablet report, sending...
I20240426 09:57:42.006220  6662 ts_manager.cc:194] Registered new tserver with Master: d96d122635cc460c97a1f42d54ed32dc (127.6.122.1:38637)
I20240426 09:57:42.006500  6632 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.017982311s
I20240426 09:57:42.008771  6662 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43978
I20240426 09:57:42.022516  6662 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43990:
name: "update-scan-delta-compact-tbl"
schema {
  columns {
    name: "key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20240426 09:57:42.050912  6755 tablet_service.cc:1467] Processing CreateTablet for tablet 07750a4a739f46488ca82c8041055cc0 (DEFAULT_TABLE table=update-scan-delta-compact-tbl [id=7cc995d457964668bcaf7be7a34560a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240426 09:57:42.051555  6755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07750a4a739f46488ca82c8041055cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240426 09:57:42.057080  6802 tablet_bootstrap.cc:492] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc: Bootstrap starting.
I20240426 09:57:42.059269  6802 tablet_bootstrap.cc:654] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc: Neither blocks nor log segments found. Creating new log.
I20240426 09:57:42.061441  6802 tablet_bootstrap.cc:492] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc: No bootstrap required, opened a new log
I20240426 09:57:42.061610  6802 ts_tablet_manager.cc:1367] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240426 09:57:42.062474  6802 raft_consensus.cc:348] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96d122635cc460c97a1f42d54ed32dc" member_type: VOTER last_known_addr { host: "127.6.122.1" port: 38637 } }
I20240426 09:57:42.062732  6802 raft_consensus.cc:374] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240426 09:57:42.062837  6802 raft_consensus.cc:729] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d96d122635cc460c97a1f42d54ed32dc, State: Initialized, Role: FOLLOWER
I20240426 09:57:42.063077  6802 consensus_queue.cc:260] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96d122635cc460c97a1f42d54ed32dc" member_type: VOTER last_known_addr { host: "127.6.122.1" port: 38637 } }
I20240426 09:57:42.063292  6802 raft_consensus.cc:388] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240426 09:57:42.063381  6802 raft_consensus.cc:482] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240426 09:57:42.063513  6802 raft_consensus.cc:3037] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc [term 0 FOLLOWER]: Advancing to term 1
I20240426 09:57:42.066462  6802 raft_consensus.cc:504] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96d122635cc460c97a1f42d54ed32dc" member_type: VOTER last_known_addr { host: "127.6.122.1" port: 38637 } }
I20240426 09:57:42.066735  6802 leader_election.cc:302] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d96d122635cc460c97a1f42d54ed32dc; no voters: 
I20240426 09:57:42.067214  6802 leader_election.cc:288] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240426 09:57:42.067369  6804 raft_consensus.cc:2781] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc [term 1 FOLLOWER]: Leader election won for term 1
I20240426 09:57:42.067855  6804 raft_consensus.cc:686] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc [term 1 LEADER]: Becoming Leader. State: Replica: d96d122635cc460c97a1f42d54ed32dc, State: Running, Role: LEADER
I20240426 09:57:42.068202  6804 consensus_queue.cc:237] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96d122635cc460c97a1f42d54ed32dc" member_type: VOTER last_known_addr { host: "127.6.122.1" port: 38637 } }
I20240426 09:57:42.068425  6790 heartbeater.cc:503] Master 127.6.122.62:39121 was elected leader, sending a full tablet report...
I20240426 09:57:42.072279  6662 catalog_manager.cc:5484] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc reported cstate change: term changed from 0 to 1, leader changed from <none> to d96d122635cc460c97a1f42d54ed32dc (127.6.122.1). New cstate: current_term: 1 leader_uuid: "d96d122635cc460c97a1f42d54ed32dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d96d122635cc460c97a1f42d54ed32dc" member_type: VOTER last_known_addr { host: "127.6.122.1" port: 38637 } health_report { overall_health: HEALTHY } } }
I20240426 09:57:42.074112  6802 ts_tablet_manager.cc:1398] T 07750a4a739f46488ca82c8041055cc0 P d96d122635cc460c97a1f42d54ed32dc: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.007s
I20240426 09:57:45.989557  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling FlushMRSOp(07750a4a739f46488ca82c8041055cc0): perf score=1.882590
I20240426 09:57:48.522778  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: FlushMRSOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 2.532s	user 2.223s	sys 0.273s Metrics: {"bytes_written":192365,"cfile_init":1,"compiler_manager_pool.queue_time_us":2189,"compiler_manager_pool.run_cpu_time_us":521683,"compiler_manager_pool.run_wall_time_us":524850,"dirs.queue_time_us":340,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":924,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6928,"lbm_writes_lt_1ms":100,"peak_mem_usage":0,"rows_written":69001,"thread_start_us":2320,"threads_started":2}
I20240426 09:57:48.526593  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0): 464214 bytes on disk
I20240426 09:57:48.527737  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240426 09:57:50.528882  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling FlushMRSOp(07750a4a739f46488ca82c8041055cc0): perf score=3.866965
I20240426 09:57:52.849920  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: FlushMRSOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 2.321s	user 2.203s	sys 0.067s Metrics: {"bytes_written":213509,"cfile_init":1,"dirs.queue_time_us":3244,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":32233,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8053,"lbm_writes_lt_1ms":111,"peak_mem_usage":0,"rows_written":80000,"thread_start_us":3130,"threads_started":1}
I20240426 09:57:52.851577  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0): 536529 bytes on disk
I20240426 09:57:52.852272  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240426 09:57:52.852989  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling CompactRowSetsOp(07750a4a739f46488ca82c8041055cc0): perf score=0.014623
I20240426 09:57:55.148439  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: CompactRowSetsOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 2.295s	user 2.215s	sys 0.076s Metrics: {"bytes_written":412907,"cfile_cache_hit":10,"cfile_cache_hit_bytes":1942,"cfile_cache_miss":184,"cfile_cache_miss_bytes":2083732,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":2601,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":89077,"drs_written":1,"lbm_read_time_us":6998,"lbm_reads_lt_1ms":204,"lbm_write_time_us":11190,"lbm_writes_lt_1ms":191,"num_input_rowsets":2,"peak_mem_usage":1229478,"rows_written":149001,"spinlock_wait_cycles":896,"thread_start_us":1908,"threads_started":6}
I20240426 09:57:55.149183  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0): 1000280 bytes on disk
I20240426 09:57:55.149876  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240426 09:57:55.150390  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling FlushMRSOp(07750a4a739f46488ca82c8041055cc0): perf score=2.874778
I20240426 09:57:57.419697  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: FlushMRSOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 2.269s	user 2.190s	sys 0.068s Metrics: {"bytes_written":200643,"cfile_init":1,"dirs.queue_time_us":372,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":656,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8043,"lbm_writes_lt_1ms":106,"peak_mem_usage":0,"rows_written":75000,"thread_start_us":247,"threads_started":1}
I20240426 09:57:57.420688  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling CompactRowSetsOp(07750a4a739f46488ca82c8041055cc0): perf score=0.014432
I20240426 09:58:00.883929  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: CompactRowSetsOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 3.463s	user 3.369s	sys 0.092s Metrics: {"bytes_written":615579,"cfile_cache_hit":10,"cfile_cache_hit_bytes":2695,"cfile_cache_miss":268,"cfile_cache_miss_bytes":3142239,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":2904,"dirs.run_cpu_time_us":794,"dirs.run_wall_time_us":5671,"drs_written":1,"lbm_read_time_us":10258,"lbm_reads_lt_1ms":292,"lbm_write_time_us":17599,"lbm_writes_lt_1ms":276,"mutex_wait_us":19,"num_input_rowsets":2,"peak_mem_usage":2383448,"rows_written":224001,"thread_start_us":2755,"threads_started":6}
I20240426 09:58:00.884687  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling FlushMRSOp(07750a4a739f46488ca82c8041055cc0): perf score=4.859153
I20240426 09:58:03.547724  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: FlushMRSOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 2.663s	user 2.553s	sys 0.108s Metrics: {"bytes_written":245010,"cfile_init":1,"dirs.queue_time_us":374,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":753,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8723,"lbm_writes_lt_1ms":124,"peak_mem_usage":0,"rows_written":92000,"thread_start_us":244,"threads_started":1}
I20240426 09:58:03.548611  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0): 2117293 bytes on disk
I20240426 09:58:03.549279  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":122,"lbm_reads_lt_1ms":8}
I20240426 09:58:03.549872  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling CompactRowSetsOp(07750a4a739f46488ca82c8041055cc0): perf score=0.014201
I20240426 09:58:08.380755  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: CompactRowSetsOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 4.831s	user 4.661s	sys 0.155s Metrics: {"bytes_written":869351,"cfile_cache_hit":10,"cfile_cache_hit_bytes":3697,"cfile_cache_miss":368,"cfile_cache_miss_bytes":4424315,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":2105,"dirs.run_cpu_time_us":347,"dirs.run_wall_time_us":2284,"drs_written":1,"lbm_read_time_us":13949,"lbm_reads_lt_1ms":388,"lbm_write_time_us":23585,"lbm_writes_lt_1ms":383,"num_input_rowsets":2,"peak_mem_usage":3602116,"rows_written":316001,"spinlock_wait_cycles":896,"thread_start_us":2156,"threads_started":6}
I20240426 09:58:08.381603  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0): 2122573 bytes on disk
I20240426 09:58:08.382292  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240426 09:58:08.382854  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling FlushMRSOp(07750a4a739f46488ca82c8041055cc0): perf score=9.820090
I20240426 09:58:12.016106  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: FlushMRSOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 3.633s	user 3.480s	sys 0.144s Metrics: {"bytes_written":337606,"cfile_init":1,"dirs.queue_time_us":409,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":883,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11799,"lbm_writes_lt_1ms":162,"peak_mem_usage":0,"rows_written":125000,"thread_start_us":251,"threads_started":1}
I20240426 09:58:12.017113  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0): 837471 bytes on disk
I20240426 09:58:12.017736  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
I20240426 09:58:12.018380  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling FlushMRSOp(07750a4a739f46488ca82c8041055cc0): perf score=1.000000
I20240426 09:58:13.715487  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: FlushMRSOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 1.697s	user 1.606s	sys 0.060s Metrics: {"bytes_written":164499,"cfile_init":1,"dirs.queue_time_us":426,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":789,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5936,"lbm_writes_lt_1ms":87,"peak_mem_usage":0,"rows_written":60000,"thread_start_us":278,"threads_started":1}
I20240426 09:58:13.716276  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0): 402943 bytes on disk
I20240426 09:58:13.716787  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240426 09:58:13.717598  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling CompactRowSetsOp(07750a4a739f46488ca82c8041055cc0): perf score=0.022730
I20240426 09:58:21.402120  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: CompactRowSetsOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 7.684s	user 7.440s	sys 0.228s Metrics: {"bytes_written":1377690,"cfile_cache_hit":15,"cfile_cache_hit_bytes":5746,"cfile_cache_miss":581,"cfile_cache_miss_bytes":7027130,"cfile_init":7,"delta_iterators_relevant":6,"dirs.queue_time_us":1650,"dirs.run_cpu_time_us":1176,"dirs.run_wall_time_us":1176,"drs_written":1,"lbm_read_time_us":21181,"lbm_reads_lt_1ms":609,"lbm_write_time_us":38010,"lbm_writes_lt_1ms":598,"num_input_rowsets":3,"peak_mem_usage":5129789,"rows_written":501001,"thread_start_us":2052,"threads_started":4}
I20240426 09:58:21.402906  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling FlushMRSOp(07750a4a739f46488ca82c8041055cc0): perf score=13.788840
I20240426 09:58:26.212013  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: FlushMRSOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 4.809s	user 4.560s	sys 0.160s Metrics: {"bytes_written":419318,"cfile_init":1,"dirs.queue_time_us":353,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1016,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":15183,"lbm_writes_lt_1ms":199,"peak_mem_usage":0,"rows_written":157000,"thread_start_us":232,"threads_started":1}
I20240426 09:58:26.212841  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0): 4420419 bytes on disk
I20240426 09:58:26.213543  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: UndoDeltaBlockGCOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":148,"lbm_reads_lt_1ms":8}
I20240426 09:58:26.214138  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling FlushMRSOp(07750a4a739f46488ca82c8041055cc0): perf score=2.874778
I20240426 09:58:28.523582  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: FlushMRSOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 2.309s	user 2.197s	sys 0.081s Metrics: {"bytes_written":210525,"cfile_init":1,"dirs.queue_time_us":626,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":800,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7689,"lbm_writes_lt_1ms":109,"peak_mem_usage":0,"rows_written":77000,"thread_start_us":525,"threads_started":1}
I20240426 09:58:28.524439  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling CompactRowSetsOp(07750a4a739f46488ca82c8041055cc0): perf score=0.022137
I20240426 09:58:39.950794  6732 maintenance_manager.cc:643] P d96d122635cc460c97a1f42d54ed32dc: CompactRowSetsOp(07750a4a739f46488ca82c8041055cc0) complete. Timing: real 11.426s	user 11.154s	sys 0.225s Metrics: {"bytes_written":2022380,"cfile_cache_hit":15,"cfile_cache_hit_bytes":8499,"cfile_cache_miss":857,"cfile_cache_miss_bytes":10323549,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":2137,"dirs.run_cpu_time_us":404,"dirs.run_wall_time_us":2148,"drs_written":1,"lbm_read_time_us":32313,"lbm_reads_lt_1ms":889,"lbm_write_time_us":59077,"lbm_writes_lt_1ms":869,"num_input_rowsets":3,"peak_mem_usage":8135832,"rows_written":735001,"thread_start_us":1889,"threads_started":7}
I20240426 09:58:39.951843  6791 maintenance_manager.cc:419] P d96d122635cc460c97a1f42d54ed32dc: Scheduling FlushMRSOp(07750a4a739f46488ca82c8041055cc0): perf score=24.702903
I20240426 09:58:41.947283  6632 update_scan_delta_compact-test.cc:209] Time spent Insert: real 59.860s	user 55.314s	sys 3.837s
I20240426 09:58:41.970693  6853 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:58:41.998266  6853 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240426 09:58:42.004366  6853 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240426 09:58:42.021689  6853 monotime.cc:342] Check failed: t_beg.Initialized() 
*** Check failure stack trace: ***
*** Aborted at 1714125522 (unix time) try "date -d @1714125522" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGABRT (@0x3e8000019e8) received by PID 6632 (TID 0x7f5da8084700) from PID 6632; stack trace: ***
    @     0x7f5dd8eb6980 (unknown) at ??:0
    @     0x7f5dd1f66fb7 gsignal at ??:0
    @     0x7f5dd1f68921 abort at ??:0
    @     0x7f5dd4c8edbd google::LogMessage::Fail() at ??:0
    @     0x7f5dd4c92b83 google::LogMessage::SendToLog() at ??:0
    @     0x7f5dd4c8e7bc google::LogMessage::Flush() at ??:0
    @     0x7f5dd4c8ff49 google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f5dd61f2880 kudu::operator-() at ??:0
    @     0x7f5de1a0739a kudu::tablet::WriteOp::ToString() at ??:0
    @     0x7f5de19d182e kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
    @     0x7f5de19d1586 kudu::tablet::OpDriver::ToString() at ??:0
    @     0x7f5de197e529 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
    @     0x7f5de87f60a7 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
    @     0x7f5de8804c5c kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
    @     0x7f5de8804ac2 std::_Function_handler<>::_M_invoke() at ??:0
    @     0x7f5ddeb6d6aa std::function<>::operator()() at ??:0
    @     0x7f5ddeb65e3a kudu::Webserver::RunPathHandler() at ??:0
    @     0x7f5ddeb60e71 kudu::Webserver::BeginRequestCallback() at ??:0
    @     0x7f5ddeb5efa7 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
    @     0x7f5ddeb9c709 handle_request at ??:0
    @     0x7f5ddeb9edeb worker_thread at ??:0
    @     0x7f5dd8eab6db start_thread at ??:0
    @     0x7f5dd204971f clone at ??:0