Diagnosed failure

UpdateScanDeltaCompactionTest.TestAll: Unrecognized error type. Please see the error log for more information.

Full log

[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from UpdateScanDeltaCompactionTest
[ RUN      ] UpdateScanDeltaCompactionTest.TestAll
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 20:45:26.941277  5300 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.5.45.62:35499
I20240502 20:45:26.942191  5300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 20:45:26.942798  5300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:26.949270  5308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:26.949282  5306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:26.949290  5305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:26.949465  5300 server_base.cc:1034] running on GCE node
I20240502 20:45:26.949981  5300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:26.950076  5300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:26.950125  5300 hybrid_clock.cc:648] HybridClock initialized: now 1714682726950123 us; error 0 us; skew 500 ppm
I20240502 20:45:26.951967  5300 webserver.cc:460] Webserver started at http://127.5.45.62:38233/ using document root <none> and password file <none>
I20240502 20:45:26.952476  5300 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:26.952545  5300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:26.952802  5300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:26.954633  5300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwqQDOV/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714682726931335-5300-0/minicluster-data/master-0-root/instance:
uuid: "e1440e42e48746b2a573956e2c5a25d1"
format_stamp: "Formatted at 2024-05-02 20:45:26 on dist-test-slave-cxdv"
I20240502 20:45:26.958201  5300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240502 20:45:26.960263  5313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:26.961309  5300 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20240502 20:45:26.961422  5300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwqQDOV/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714682726931335-5300-0/minicluster-data/master-0-root
uuid: "e1440e42e48746b2a573956e2c5a25d1"
format_stamp: "Formatted at 2024-05-02 20:45:26 on dist-test-slave-cxdv"
I20240502 20:45:26.961524  5300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwqQDOV/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714682726931335-5300-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskwqQDOV/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714682726931335-5300-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskwqQDOV/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714682726931335-5300-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:26.980746  5300 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:26.981458  5300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 20:45:26.981638  5300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:26.988544  5366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.45.62:35499 every 8 connection(s)
I20240502 20:45:26.988543  5300 rpc_server.cc:307] RPC server started. Bound to: 127.5.45.62:35499
I20240502 20:45:26.990862  5367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:26.997000  5367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1: Bootstrap starting.
I20240502 20:45:26.999383  5367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:27.000106  5367 log.cc:824] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1: Log is configured to *not* fsync() on all Append() calls
I20240502 20:45:27.001799  5367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1: No bootstrap required, opened a new log
I20240502 20:45:27.004853  5367 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1440e42e48746b2a573956e2c5a25d1" member_type: VOTER }
I20240502 20:45:27.005025  5367 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:27.005066  5367 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1440e42e48746b2a573956e2c5a25d1, State: Initialized, Role: FOLLOWER
I20240502 20:45:27.005703  5367 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1440e42e48746b2a573956e2c5a25d1" member_type: VOTER }
I20240502 20:45:27.005867  5367 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:27.005939  5367 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:27.006059  5367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:27.006856  5367 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1440e42e48746b2a573956e2c5a25d1" member_type: VOTER }
I20240502 20:45:27.007308  5367 leader_election.cc:302] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1440e42e48746b2a573956e2c5a25d1; no voters: 
I20240502 20:45:27.007624  5367 leader_election.cc:288] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 20:45:27.007747  5370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:27.008009  5370 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [term 1 LEADER]: Becoming Leader. State: Replica: e1440e42e48746b2a573956e2c5a25d1, State: Running, Role: LEADER
I20240502 20:45:27.008440  5370 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1440e42e48746b2a573956e2c5a25d1" member_type: VOTER }
I20240502 20:45:27.008663  5367 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 20:45:27.010314  5371 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1440e42e48746b2a573956e2c5a25d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1440e42e48746b2a573956e2c5a25d1" member_type: VOTER } }
I20240502 20:45:27.010371  5372 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1440e42e48746b2a573956e2c5a25d1. Latest consensus state: current_term: 1 leader_uuid: "e1440e42e48746b2a573956e2c5a25d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1440e42e48746b2a573956e2c5a25d1" member_type: VOTER } }
I20240502 20:45:27.010464  5371 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:27.010473  5372 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1 [sys.catalog]: This master's current role is: LEADER
I20240502 20:45:27.010826  5382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 20:45:27.011003  5300 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240502 20:45:27.013290  5382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 20:45:27.018437  5382 catalog_manager.cc:1347] Generated new cluster ID: ecd8ea0210b7431ea554261f80ed2858
I20240502 20:45:27.018528  5382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 20:45:27.025754  5382 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 20:45:27.027016  5382 catalog_manager.cc:1504] Loading token signing keys...
I20240502 20:45:27.048228  5382 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e1440e42e48746b2a573956e2c5a25d1: Generated new TSK 0
I20240502 20:45:27.048949  5382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 20:45:27.076241  5300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 20:45:27.078635  5389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:27.078774  5392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 20:45:27.078980  5390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 20:45:27.079142  5300 server_base.cc:1034] running on GCE node
I20240502 20:45:27.079341  5300 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240502 20:45:27.079382  5300 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240502 20:45:27.079398  5300 hybrid_clock.cc:648] HybridClock initialized: now 1714682727079398 us; error 0 us; skew 500 ppm
I20240502 20:45:27.080446  5300 webserver.cc:460] Webserver started at http://127.5.45.1:38077/ using document root <none> and password file <none>
I20240502 20:45:27.080621  5300 fs_manager.cc:359] Metadata directory not provided
I20240502 20:45:27.080667  5300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 20:45:27.080747  5300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 20:45:27.081163  5300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwqQDOV/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714682726931335-5300-0/minicluster-data/ts-0-root/instance:
uuid: "981390e24e074e0d8d9ecb23fe9af8b3"
format_stamp: "Formatted at 2024-05-02 20:45:27 on dist-test-slave-cxdv"
I20240502 20:45:27.082612  5300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s	user 0.002s	sys 0.000s
I20240502 20:45:27.083518  5397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:27.083760  5300 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
I20240502 20:45:27.083822  5300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwqQDOV/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714682726931335-5300-0/minicluster-data/ts-0-root
uuid: "981390e24e074e0d8d9ecb23fe9af8b3"
format_stamp: "Formatted at 2024-05-02 20:45:27 on dist-test-slave-cxdv"
I20240502 20:45:27.083894  5300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwqQDOV/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714682726931335-5300-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskwqQDOV/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714682726931335-5300-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskwqQDOV/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714682726931335-5300-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 20:45:27.106194  5300 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 20:45:27.106730  5300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 20:45:27.107184  5300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 20:45:27.108029  5300 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 20:45:27.108088  5300 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:27.108134  5300 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 20:45:27.108162  5300 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 20:45:27.119603  5300 rpc_server.cc:307] RPC server started. Bound to: 127.5.45.1:38207
I20240502 20:45:27.119691  5460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.45.1:38207 every 8 connection(s)
I20240502 20:45:27.130226  5461 heartbeater.cc:347] Connected to a master server at 127.5.45.62:35499
I20240502 20:45:27.130499  5461 heartbeater.cc:464] Registering TS with master...
I20240502 20:45:27.130975  5461 heartbeater.cc:511] Master 127.5.45.62:35499 requested a full tablet report, sending...
I20240502 20:45:27.132629  5330 ts_manager.cc:194] Registered new tserver with Master: 981390e24e074e0d8d9ecb23fe9af8b3 (127.5.45.1:38207)
I20240502 20:45:27.132745  5300 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.012448669s
I20240502 20:45:27.134352  5330 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54130
I20240502 20:45:27.143033  5330 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54144:
name: "update-scan-delta-compact-tbl"
schema {
  columns {
    name: "key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20240502 20:45:27.158411  5425 tablet_service.cc:1467] Processing CreateTablet for tablet af685b91f0594aa7977dbe4cc2b23032 (DEFAULT_TABLE table=update-scan-delta-compact-tbl [id=64cd4fd4b0884fae846c537302a79dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 20:45:27.159034  5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af685b91f0594aa7977dbe4cc2b23032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 20:45:27.161718  5473 tablet_bootstrap.cc:492] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3: Bootstrap starting.
I20240502 20:45:27.162963  5473 tablet_bootstrap.cc:654] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3: Neither blocks nor log segments found. Creating new log.
I20240502 20:45:27.164397  5473 tablet_bootstrap.cc:492] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3: No bootstrap required, opened a new log
I20240502 20:45:27.164506  5473 ts_tablet_manager.cc:1367] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 20:45:27.165115  5473 raft_consensus.cc:348] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981390e24e074e0d8d9ecb23fe9af8b3" member_type: VOTER last_known_addr { host: "127.5.45.1" port: 38207 } }
I20240502 20:45:27.165766  5473 raft_consensus.cc:374] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 20:45:27.165875  5473 raft_consensus.cc:729] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981390e24e074e0d8d9ecb23fe9af8b3, State: Initialized, Role: FOLLOWER
I20240502 20:45:27.166059  5473 consensus_queue.cc:260] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981390e24e074e0d8d9ecb23fe9af8b3" member_type: VOTER last_known_addr { host: "127.5.45.1" port: 38207 } }
I20240502 20:45:27.166169  5473 raft_consensus.cc:388] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 20:45:27.166252  5473 raft_consensus.cc:482] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 20:45:27.166316  5473 raft_consensus.cc:3037] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 20:45:27.167405  5473 raft_consensus.cc:504] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981390e24e074e0d8d9ecb23fe9af8b3" member_type: VOTER last_known_addr { host: "127.5.45.1" port: 38207 } }
I20240502 20:45:27.167562  5473 leader_election.cc:302] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 981390e24e074e0d8d9ecb23fe9af8b3; no voters: 
I20240502 20:45:27.167791  5473 leader_election.cc:288] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 20:45:27.167918  5475 raft_consensus.cc:2781] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 20:45:27.168165  5475 raft_consensus.cc:686] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3 [term 1 LEADER]: Becoming Leader. State: Replica: 981390e24e074e0d8d9ecb23fe9af8b3, State: Running, Role: LEADER
I20240502 20:45:27.168318  5461 heartbeater.cc:503] Master 127.5.45.62:35499 was elected leader, sending a full tablet report...
I20240502 20:45:27.168407  5475 consensus_queue.cc:237] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981390e24e074e0d8d9ecb23fe9af8b3" member_type: VOTER last_known_addr { host: "127.5.45.1" port: 38207 } }
I20240502 20:45:27.168159  5473 ts_tablet_manager.cc:1398] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20240502 20:45:27.171350  5330 catalog_manager.cc:5484] T af685b91f0594aa7977dbe4cc2b23032 P 981390e24e074e0d8d9ecb23fe9af8b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 981390e24e074e0d8d9ecb23fe9af8b3 (127.5.45.1). New cstate: current_term: 1 leader_uuid: "981390e24e074e0d8d9ecb23fe9af8b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981390e24e074e0d8d9ecb23fe9af8b3" member_type: VOTER last_known_addr { host: "127.5.45.1" port: 38207 } health_report { overall_health: HEALTHY } } }
I20240502 20:45:31.121062  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling FlushMRSOp(af685b91f0594aa7977dbe4cc2b23032): perf score=7.835715
I20240502 20:45:32.908303  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: FlushMRSOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 1.787s	user 1.748s	sys 0.012s Metrics: {"bytes_written":309855,"cfile_init":1,"compiler_manager_pool.queue_time_us":307,"compiler_manager_pool.run_cpu_time_us":210266,"compiler_manager_pool.run_wall_time_us":212293,"dirs.queue_time_us":213,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1276,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10482,"lbm_writes_lt_1ms":148,"peak_mem_usage":0,"rows_written":112001,"thread_start_us":214,"threads_started":2}
I20240502 20:45:32.909292  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling UndoDeltaBlockGCOp(af685b91f0594aa7977dbe4cc2b23032): 749555 bytes on disk
I20240502 20:45:32.909955  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: UndoDeltaBlockGCOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240502 20:45:34.910754  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling FlushMRSOp(af685b91f0594aa7977dbe4cc2b23032): perf score=6.843528
I20240502 20:45:36.360860  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: FlushMRSOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 1.450s	user 1.432s	sys 0.008s Metrics: {"bytes_written":281695,"cfile_init":1,"dirs.queue_time_us":211,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":31625,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8057,"lbm_writes_lt_1ms":141,"peak_mem_usage":0,"rows_written":106000,"thread_start_us":106,"threads_started":1}
I20240502 20:45:36.361874  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling CompactRowSetsOp(af685b91f0594aa7977dbe4cc2b23032): perf score=0.014450
I20240502 20:45:38.503490  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: CompactRowSetsOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 2.141s	user 2.066s	sys 0.012s Metrics: {"bytes_written":601158,"cfile_cache_hit":10,"cfile_cache_hit_bytes":2780,"cfile_cache_miss":260,"cfile_cache_miss_bytes":3039798,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":1128,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":165443,"drs_written":1,"lbm_read_time_us":7865,"lbm_reads_lt_1ms":284,"lbm_write_time_us":15221,"lbm_writes_lt_1ms":270,"num_input_rowsets":2,"peak_mem_usage":1782241,"rows_written":218001,"spinlock_wait_cycles":1408,"thread_start_us":389,"threads_started":6}
I20240502 20:45:38.504712  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling FlushMRSOp(af685b91f0594aa7977dbe4cc2b23032): perf score=4.859153
I20240502 20:45:39.738853  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: FlushMRSOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 1.234s	user 1.196s	sys 0.020s Metrics: {"bytes_written":240614,"cfile_init":1,"dirs.queue_time_us":179,"dirs.run_cpu_time_us":135,"dirs.run_wall_time_us":936,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7237,"lbm_writes_lt_1ms":122,"peak_mem_usage":0,"rows_written":90000,"thread_start_us":86,"threads_started":1}
I20240502 20:45:39.739482  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling UndoDeltaBlockGCOp(af685b91f0594aa7977dbe4cc2b23032): 2064000 bytes on disk
I20240502 20:45:39.739965  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: UndoDeltaBlockGCOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":85,"lbm_reads_lt_1ms":8}
I20240502 20:45:39.740358  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling CompactRowSetsOp(af685b91f0594aa7977dbe4cc2b23032): perf score=0.014221
I20240502 20:45:42.634061  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: CompactRowSetsOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 2.894s	user 2.838s	sys 0.036s Metrics: {"bytes_written":850127,"cfile_cache_hit":10,"cfile_cache_hit_bytes":3606,"cfile_cache_miss":360,"cfile_cache_miss_bytes":4313176,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":1275,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1894,"drs_written":1,"lbm_read_time_us":10770,"lbm_reads_lt_1ms":380,"lbm_write_time_us":21754,"lbm_writes_lt_1ms":375,"num_input_rowsets":2,"peak_mem_usage":3504624,"rows_written":308001,"thread_start_us":406,"threads_started":6}
I20240502 20:45:42.634855  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling FlushMRSOp(af685b91f0594aa7977dbe4cc2b23032): perf score=7.835715
I20240502 20:45:44.152329  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: FlushMRSOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 1.517s	user 1.482s	sys 0.004s Metrics: {"bytes_written":300562,"cfile_init":1,"dirs.queue_time_us":247,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":35807,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8689,"lbm_writes_lt_1ms":148,"peak_mem_usage":0,"rows_written":112000,"thread_start_us":111,"threads_started":1}
I20240502 20:45:44.153227  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling CompactRowSetsOp(af685b91f0594aa7977dbe4cc2b23032): perf score=0.013935
I20240502 20:45:48.073189  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: CompactRowSetsOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 3.920s	user 3.897s	sys 0.016s Metrics: {"bytes_written":1156326,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4767,"cfile_cache_miss":490,"cfile_cache_miss_bytes":5895971,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":1341,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1431,"drs_written":1,"lbm_read_time_us":14263,"lbm_reads_lt_1ms":518,"lbm_write_time_us":25226,"lbm_writes_lt_1ms":504,"mutex_wait_us":153,"num_input_rowsets":2,"peak_mem_usage":4967019,"rows_written":420001,"thread_start_us":547,"threads_started":7}
I20240502 20:45:48.074786  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling UndoDeltaBlockGCOp(af685b91f0594aa7977dbe4cc2b23032): 2823368 bytes on disk
I20240502 20:45:48.075322  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: UndoDeltaBlockGCOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 20:45:48.075773  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling FlushMRSOp(af685b91f0594aa7977dbe4cc2b23032): perf score=11.804465
I20240502 20:45:50.077754  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: FlushMRSOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 2.002s	user 1.976s	sys 0.014s Metrics: {"bytes_written":387900,"cfile_init":1,"dirs.queue_time_us":169,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":913,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12942,"lbm_writes_lt_1ms":186,"peak_mem_usage":0,"rows_written":144000,"spinlock_wait_cycles":1280,"thread_start_us":76,"threads_started":1}
I20240502 20:45:50.078545  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling UndoDeltaBlockGCOp(af685b91f0594aa7977dbe4cc2b23032): 966464 bytes on disk
I20240502 20:45:50.079094  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: UndoDeltaBlockGCOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240502 20:45:50.079517  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling CompactRowSetsOp(af685b91f0594aa7977dbe4cc2b23032): perf score=0.013569
I20240502 20:45:55.353185  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: CompactRowSetsOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 5.274s	user 5.236s	sys 0.028s Metrics: {"bytes_written":1551420,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6415,"cfile_cache_miss":656,"cfile_cache_miss_bytes":7929461,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":946,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":838,"drs_written":1,"lbm_read_time_us":17566,"lbm_reads_lt_1ms":676,"lbm_write_time_us":33594,"lbm_writes_lt_1ms":671,"num_input_rowsets":2,"peak_mem_usage":6786893,"rows_written":564001,"thread_start_us":502,"threads_started":7}
I20240502 20:45:55.354015  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling UndoDeltaBlockGCOp(af685b91f0594aa7977dbe4cc2b23032): 3788009 bytes on disk
I20240502 20:45:55.354386  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: UndoDeltaBlockGCOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
I20240502 20:45:55.354830  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling FlushMRSOp(af685b91f0594aa7977dbe4cc2b23032): perf score=18.749778
I20240502 20:45:57.897626  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: FlushMRSOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 2.543s	user 2.526s	sys 0.015s Metrics: {"bytes_written":524042,"cfile_init":1,"dirs.queue_time_us":200,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":975,"drs_written":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4,"lbm_write_time_us":17280,"lbm_writes_lt_1ms":244,"peak_mem_usage":0,"rows_written":196000,"thread_start_us":85,"threads_started":1}
I20240502 20:45:57.898298  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling UndoDeltaBlockGCOp(af685b91f0594aa7977dbe4cc2b23032): 1316479 bytes on disk
I20240502 20:45:57.898730  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: UndoDeltaBlockGCOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240502 20:45:57.899171  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling FlushMRSOp(af685b91f0594aa7977dbe4cc2b23032): perf score=1.000000
I20240502 20:45:58.845924  5402 maintenance_manager.cc:643] P 981390e24e074e0d8d9ecb23fe9af8b3: FlushMRSOp(af685b91f0594aa7977dbe4cc2b23032) complete. Timing: real 0.947s	user 0.910s	sys 0.000s Metrics: {"bytes_written":168314,"cfile_init":1,"dirs.queue_time_us":217,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1121,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5898,"lbm_writes_lt_1ms":90,"peak_mem_usage":0,"rows_written":63000,"thread_start_us":92,"threads_started":1}
I20240502 20:45:58.846619  5462 maintenance_manager.cc:419] P 981390e24e074e0d8d9ecb23fe9af8b3: Scheduling CompactRowSetsOp(af685b91f0594aa7977dbe4cc2b23032): perf score=0.021916
I20240502 20:46:04.366909  5300 update_scan_delta_compact-test.cc:209] Time spent Insert: real 37.188s	user 34.824s	sys 2.014s
I20240502 20:46:04.369143  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.372074  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.373749  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.375329  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.376896  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.378427  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.380478  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.382448  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.384342  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.386252  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.388087  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.389963  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.391753  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.393561  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.395341  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.397142  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.398941  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.400684  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.402457  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.404191  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240502 20:46:04.406226  5520 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240502 20:46:04.407150  5520 monotime.cc:342] Check failed: t_beg.Initialized() 
*** Check failure stack trace: ***
*** Aborted at 1714682764 (unix time) try "date -d @1714682764" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGABRT (@0x3e8000014b4) received by PID 5300 (TID 0x7f5b4422a700) from PID 5300; stack trace: ***
    @     0x7f5b70d20980 (unknown) at ??:0
    @     0x7f5b6ed80fb7 gsignal at ??:0
    @     0x7f5b6ed82921 abort at ??:0
    @     0x7f5b6f8efdcd google::LogMessage::Fail() at ??:0
    @     0x7f5b6f8f3b93 google::LogMessage::SendToLog() at ??:0
    @     0x7f5b6f8ef7cc google::LogMessage::Flush() at ??:0
    @     0x7f5b6f8f0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f5b705e45b1 kudu::operator-() at ??:0
    @     0x7f5b7148ffd2 kudu::tablet::WriteOp::ToString() at ??:0
    @     0x7f5b7147905b kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
    @     0x7f5b71478f90 kudu::tablet::OpDriver::ToString() at ??:0
    @     0x7f5b7144f9bd kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
    @     0x7f5b71f2f861 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
    @     0x7f5b71f2e93a _ZZN4kudu7tserver24TabletServerPathHandlers8RegisterEPNS_9WebserverEENKUlRKNS_19WebCallbackRegistry10WebRequestEPNS4_22PrerenderedWebResponseEE2_clES7_S9_ at ??:0
    @     0x7f5b71f37149 _ZNSt17_Function_handlerIFvRKN4kudu19WebCallbackRegistry10WebRequestEPNS1_22PrerenderedWebResponseEEZNS0_7tserver24TabletServerPathHandlers8RegisterEPNS0_9WebserverEEUlS4_S6_E2_E9_M_invokeERKSt9_Any_dataS4_OS6_ at ??:0
    @     0x7f5b6e508d09 std::function<>::operator()() at ??:0
    @     0x7f5b6e503143 kudu::Webserver::RunPathHandler() at ??:0
    @     0x7f5b6e502a38 kudu::Webserver::BeginRequestCallback() at ??:0
    @     0x7f5b6e502123 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
    @     0x7f5b6e52ab89 handle_request at ??:0
    @     0x7f5b6e52d26b worker_thread at ??:0
    @     0x7f5b70d156db start_thread at ??:0
    @     0x7f5b6ee6371f clone at ??:0