Diagnosed failure

UpdateScanDeltaCompactionTest.TestAll: Unrecognized error type. Please see the error log for more information.

Full log

[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from UpdateScanDeltaCompactionTest
[ RUN      ] UpdateScanDeltaCompactionTest.TestAll
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 08:12:43.861449  1834 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.1.202.190:42509
I20240501 08:12:43.863231  1834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:43.864673  1834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:43.877099  1840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:43.877678  1842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:43.878417  1839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:43.879336  1834 server_base.cc:1034] running on GCE node
I20240501 08:12:43.880221  1834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:12:43.880427  1834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:12:43.880523  1834 hybrid_clock.cc:648] HybridClock initialized: now 1714551163880500 us; error 0 us; skew 500 ppm
I20240501 08:12:43.880895  1834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:43.889735  1834 webserver.cc:460] Webserver started at http://127.1.202.190:37381/ using document root <none> and password file <none>
I20240501 08:12:43.890619  1834 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:43.890738  1834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:43.891127  1834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:43.894816  1834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLTCxBL/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714551163783598-1834-0/minicluster-data/master-0-root/instance:
uuid: "e3961bd8289441c5a8be0f48722a1336"
format_stamp: "Formatted at 2024-05-01 08:12:43 on dist-test-slave-5hmb"
I20240501 08:12:43.901379  1834 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.000s
I20240501 08:12:43.905045  1847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:43.906723  1834 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240501 08:12:43.906914  1834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLTCxBL/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714551163783598-1834-0/minicluster-data/master-0-root
uuid: "e3961bd8289441c5a8be0f48722a1336"
format_stamp: "Formatted at 2024-05-01 08:12:43 on dist-test-slave-5hmb"
I20240501 08:12:43.907106  1834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLTCxBL/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714551163783598-1834-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskLTCxBL/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714551163783598-1834-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskLTCxBL/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714551163783598-1834-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:43.929807  1834 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:43.930718  1834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:43.930990  1834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:43.950600  1834 rpc_server.cc:307] RPC server started. Bound to: 127.1.202.190:42509
I20240501 08:12:43.950618  1898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.202.190:42509 every 8 connection(s)
I20240501 08:12:43.954792  1899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:43.968911  1899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336: Bootstrap starting.
I20240501 08:12:43.972993  1899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:43.974201  1899 log.cc:824] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:43.977180  1899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336: No bootstrap required, opened a new log
I20240501 08:12:43.984547  1899 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3961bd8289441c5a8be0f48722a1336" member_type: VOTER }
I20240501 08:12:43.984915  1899 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:43.985059  1899 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3961bd8289441c5a8be0f48722a1336, State: Initialized, Role: FOLLOWER
I20240501 08:12:43.985828  1899 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3961bd8289441c5a8be0f48722a1336" member_type: VOTER }
I20240501 08:12:43.986047  1899 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:12:43.986202  1899 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:12:43.986364  1899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:43.988549  1899 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3961bd8289441c5a8be0f48722a1336" member_type: VOTER }
I20240501 08:12:43.989137  1899 leader_election.cc:302] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3961bd8289441c5a8be0f48722a1336; no voters: 
I20240501 08:12:43.989776  1899 leader_election.cc:288] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:12:43.989943  1902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:43.990615  1902 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [term 1 LEADER]: Becoming Leader. State: Replica: e3961bd8289441c5a8be0f48722a1336, State: Running, Role: LEADER
I20240501 08:12:43.991320  1902 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3961bd8289441c5a8be0f48722a1336" member_type: VOTER }
I20240501 08:12:43.992054  1899 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:12:43.994263  1903 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3961bd8289441c5a8be0f48722a1336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3961bd8289441c5a8be0f48722a1336" member_type: VOTER } }
I20240501 08:12:43.994477  1904 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3961bd8289441c5a8be0f48722a1336. Latest consensus state: current_term: 1 leader_uuid: "e3961bd8289441c5a8be0f48722a1336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3961bd8289441c5a8be0f48722a1336" member_type: VOTER } }
I20240501 08:12:43.994622  1903 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [sys.catalog]: This master's current role is: LEADER
I20240501 08:12:43.994711  1904 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336 [sys.catalog]: This master's current role is: LEADER
I20240501 08:12:43.995577  1909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:12:44.001931  1909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:12:44.002604  1834 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 08:12:44.012779  1909 catalog_manager.cc:1347] Generated new cluster ID: a750ff3f9c5a419aaaaa73b2eeba6fcc
I20240501 08:12:44.012917  1909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:12:44.034430  1909 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:12:44.035880  1909 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:12:44.051069  1909 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e3961bd8289441c5a8be0f48722a1336: Generated new TSK 0
I20240501 08:12:44.052263  1909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:12:44.068225  1834 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:44.073432  1921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:44.075711  1922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:44.076629  1924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:44.077308  1834 server_base.cc:1034] running on GCE node
I20240501 08:12:44.077627  1834 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 08:12:44.077714  1834 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 08:12:44.077781  1834 hybrid_clock.cc:648] HybridClock initialized: now 1714551164077755 us; error 0 us; skew 500 ppm
I20240501 08:12:44.078002  1834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:44.079538  1834 webserver.cc:460] Webserver started at http://127.1.202.129:33341/ using document root <none> and password file <none>
I20240501 08:12:44.079828  1834 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:44.079921  1834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:44.080101  1834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:12:44.080845  1834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLTCxBL/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714551163783598-1834-0/minicluster-data/ts-0-root/instance:
uuid: "a646ff4d66fd40539fbd5881bb499f81"
format_stamp: "Formatted at 2024-05-01 08:12:44 on dist-test-slave-5hmb"
I20240501 08:12:44.084162  1834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.000s	sys 0.004s
I20240501 08:12:44.086190  1929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:44.086669  1834 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.000s	sys 0.002s
I20240501 08:12:44.086815  1834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLTCxBL/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714551163783598-1834-0/minicluster-data/ts-0-root
uuid: "a646ff4d66fd40539fbd5881bb499f81"
format_stamp: "Formatted at 2024-05-01 08:12:44 on dist-test-slave-5hmb"
I20240501 08:12:44.086988  1834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLTCxBL/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714551163783598-1834-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskLTCxBL/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714551163783598-1834-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskLTCxBL/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714551163783598-1834-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:44.105541  1834 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 08:12:44.106464  1834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:44.107638  1834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:44.109700  1834 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:44.109872  1834 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:44.110005  1834 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:44.110090  1834 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 08:12:44.130532  1991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.202.129:43491 every 8 connection(s)
I20240501 08:12:44.130527  1834 rpc_server.cc:307] RPC server started. Bound to: 127.1.202.129:43491
I20240501 08:12:44.147498  1992 heartbeater.cc:347] Connected to a master server at 127.1.202.190:42509
I20240501 08:12:44.147902  1992 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:44.148787  1992 heartbeater.cc:511] Master 127.1.202.190:42509 requested a full tablet report, sending...
I20240501 08:12:44.151190  1864 ts_manager.cc:194] Registered new tserver with Master: a646ff4d66fd40539fbd5881bb499f81 (127.1.202.129:43491)
I20240501 08:12:44.151710  1834 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.019406564s
I20240501 08:12:44.153445  1864 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38730
I20240501 08:12:44.170553  1864 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38736:
name: "update-scan-delta-compact-tbl"
schema {
  columns {
    name: "key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20240501 08:12:44.200019  1957 tablet_service.cc:1467] Processing CreateTablet for tablet 7334b7b3bff94789be9b4ae942cb7503 (DEFAULT_TABLE table=update-scan-delta-compact-tbl [id=d7f3c9756ca240978de24500f9509e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:44.200959  1957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7334b7b3bff94789be9b4ae942cb7503. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:44.208953  2004 tablet_bootstrap.cc:492] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81: Bootstrap starting.
I20240501 08:12:44.210774  2004 tablet_bootstrap.cc:654] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:44.213299  2004 tablet_bootstrap.cc:492] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81: No bootstrap required, opened a new log
I20240501 08:12:44.213471  2004 ts_tablet_manager.cc:1367] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240501 08:12:44.214264  2004 raft_consensus.cc:348] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a646ff4d66fd40539fbd5881bb499f81" member_type: VOTER last_known_addr { host: "127.1.202.129" port: 43491 } }
I20240501 08:12:44.214488  2004 raft_consensus.cc:374] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:44.214607  2004 raft_consensus.cc:729] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a646ff4d66fd40539fbd5881bb499f81, State: Initialized, Role: FOLLOWER
I20240501 08:12:44.214839  2004 consensus_queue.cc:260] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a646ff4d66fd40539fbd5881bb499f81" member_type: VOTER last_known_addr { host: "127.1.202.129" port: 43491 } }
I20240501 08:12:44.215008  2004 raft_consensus.cc:388] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:12:44.215107  2004 raft_consensus.cc:482] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:12:44.215224  2004 raft_consensus.cc:3037] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:44.217334  2004 raft_consensus.cc:504] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a646ff4d66fd40539fbd5881bb499f81" member_type: VOTER last_known_addr { host: "127.1.202.129" port: 43491 } }
I20240501 08:12:44.217588  2004 leader_election.cc:302] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a646ff4d66fd40539fbd5881bb499f81; no voters: 
I20240501 08:12:44.218065  2004 leader_election.cc:288] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 08:12:44.218720  2006 raft_consensus.cc:2781] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:44.223636  2006 raft_consensus.cc:686] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81 [term 1 LEADER]: Becoming Leader. State: Replica: a646ff4d66fd40539fbd5881bb499f81, State: Running, Role: LEADER
I20240501 08:12:44.224130  2006 consensus_queue.cc:237] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a646ff4d66fd40539fbd5881bb499f81" member_type: VOTER last_known_addr { host: "127.1.202.129" port: 43491 } }
I20240501 08:12:44.224507  1992 heartbeater.cc:503] Master 127.1.202.190:42509 was elected leader, sending a full tablet report...
I20240501 08:12:44.225677  2004 ts_tablet_manager.cc:1398] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.006s
I20240501 08:12:44.228982  1864 catalog_manager.cc:5484] T 7334b7b3bff94789be9b4ae942cb7503 P a646ff4d66fd40539fbd5881bb499f81 reported cstate change: term changed from 0 to 1, leader changed from <none> to a646ff4d66fd40539fbd5881bb499f81 (127.1.202.129). New cstate: current_term: 1 leader_uuid: "a646ff4d66fd40539fbd5881bb499f81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a646ff4d66fd40539fbd5881bb499f81" member_type: VOTER last_known_addr { host: "127.1.202.129" port: 43491 } health_report { overall_health: HEALTHY } } }
I20240501 08:12:48.133410  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503): perf score=1.000000
I20240501 08:12:50.785035  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 2.651s	user 2.281s	sys 0.350s Metrics: {"bytes_written":178220,"cfile_init":1,"compiler_manager_pool.queue_time_us":1022,"compiler_manager_pool.run_cpu_time_us":522803,"compiler_manager_pool.run_wall_time_us":528028,"dirs.queue_time_us":410,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1043,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7462,"lbm_writes_lt_1ms":91,"peak_mem_usage":0,"rows_written":64001,"thread_start_us":1186,"threads_started":2}
I20240501 08:12:50.787190  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling UndoDeltaBlockGCOp(7334b7b3bff94789be9b4ae942cb7503): 430142 bytes on disk
I20240501 08:12:50.788174  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: UndoDeltaBlockGCOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240501 08:12:52.789268  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503): perf score=2.874778
I20240501 08:12:55.196460  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 2.407s	user 2.307s	sys 0.073s Metrics: {"bytes_written":208833,"cfile_init":1,"dirs.queue_time_us":436,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":1688,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9113,"lbm_writes_lt_1ms":109,"peak_mem_usage":0,"rows_written":77000,"thread_start_us":300,"threads_started":1}
I20240501 08:12:55.197888  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling CompactRowSetsOp(7334b7b3bff94789be9b4ae942cb7503): perf score=0.014642
I20240501 08:12:57.656500  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: CompactRowSetsOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 2.458s	user 2.356s	sys 0.081s Metrics: {"bytes_written":389581,"cfile_cache_hit":10,"cfile_cache_hit_bytes":1813,"cfile_cache_miss":170,"cfile_cache_miss_bytes":1972377,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":2046,"dirs.run_cpu_time_us":689,"dirs.run_wall_time_us":3851,"drs_written":1,"lbm_read_time_us":8324,"lbm_reads_lt_1ms":194,"lbm_write_time_us":15219,"lbm_writes_lt_1ms":182,"num_input_rowsets":2,"peak_mem_usage":1180734,"rows_written":141001,"thread_start_us":1467,"threads_started":6}
I20240501 08:12:57.657374  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling UndoDeltaBlockGCOp(7334b7b3bff94789be9b4ae942cb7503): 946237 bytes on disk
I20240501 08:12:57.658005  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: UndoDeltaBlockGCOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 08:12:57.658847  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503): perf score=2.874778
I20240501 08:12:59.937371  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 2.278s	user 2.193s	sys 0.081s Metrics: {"bytes_written":200707,"cfile_init":1,"dirs.queue_time_us":406,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":1091,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8653,"lbm_writes_lt_1ms":106,"peak_mem_usage":0,"rows_written":75000,"thread_start_us":283,"threads_started":1}
I20240501 08:12:59.938405  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling UndoDeltaBlockGCOp(7334b7b3bff94789be9b4ae942cb7503): 503965 bytes on disk
I20240501 08:12:59.939157  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: UndoDeltaBlockGCOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240501 08:12:59.939896  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling CompactRowSetsOp(7334b7b3bff94789be9b4ae942cb7503): perf score=0.014453
I20240501 08:13:03.809864  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: CompactRowSetsOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 3.870s	user 3.777s	sys 0.090s Metrics: {"bytes_written":596361,"cfile_cache_hit":10,"cfile_cache_hit_bytes":2613,"cfile_cache_miss":260,"cfile_cache_miss_bytes":3029069,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":3267,"dirs.run_cpu_time_us":401,"dirs.run_wall_time_us":2416,"drs_written":1,"lbm_read_time_us":13987,"lbm_reads_lt_1ms":280,"lbm_write_time_us":22188,"lbm_writes_lt_1ms":268,"num_input_rowsets":2,"peak_mem_usage":2253457,"rows_written":216001,"spinlock_wait_cycles":768,"thread_start_us":2970,"threads_started":6}
I20240501 08:13:03.810838  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503): perf score=4.859153
I20240501 08:13:06.854998  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 3.044s	user 2.855s	sys 0.133s Metrics: {"bytes_written":253663,"cfile_init":1,"dirs.queue_time_us":441,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":874,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12471,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":126,"peak_mem_usage":0,"rows_written":95000,"thread_start_us":295,"threads_started":1}
I20240501 08:13:06.856140  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling CompactRowSetsOp(7334b7b3bff94789be9b4ae942cb7503): perf score=0.014213
I20240501 08:13:12.457983  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: CompactRowSetsOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 5.602s	user 5.444s	sys 0.136s Metrics: {"bytes_written":855232,"cfile_cache_hit":10,"cfile_cache_hit_bytes":3637,"cfile_cache_miss":362,"cfile_cache_miss_bytes":4353351,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":5587,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":1685,"drs_written":1,"lbm_read_time_us":20078,"lbm_reads_lt_1ms":390,"lbm_write_time_us":33176,"lbm_writes_lt_1ms":378,"num_input_rowsets":2,"peak_mem_usage":3472125,"rows_written":311001,"thread_start_us":4886,"threads_started":7}
I20240501 08:13:12.459177  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling UndoDeltaBlockGCOp(7334b7b3bff94789be9b4ae942cb7503): 2090128 bytes on disk
I20240501 08:13:12.459908  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: UndoDeltaBlockGCOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240501 08:13:12.460471  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503): perf score=10.812278
I20240501 08:13:16.868860  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 4.408s	user 4.203s	sys 0.168s Metrics: {"bytes_written":369766,"cfile_init":1,"dirs.queue_time_us":384,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1119,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":17035,"lbm_writes_lt_1ms":178,"peak_mem_usage":0,"rows_written":137000,"thread_start_us":252,"threads_started":1}
I20240501 08:13:16.869972  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503): perf score=1.882590
I20240501 08:13:18.806079  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 1.936s	user 1.912s	sys 0.020s Metrics: {"bytes_written":181899,"cfile_init":1,"dirs.queue_time_us":397,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":865,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8326,"lbm_writes_lt_1ms":98,"peak_mem_usage":0,"rows_written":67000,"spinlock_wait_cycles":13696,"thread_start_us":281,"threads_started":1}
I20240501 08:13:18.807427  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling UndoDeltaBlockGCOp(7334b7b3bff94789be9b4ae942cb7503): 1369588 bytes on disk
I20240501 08:13:18.808332  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: UndoDeltaBlockGCOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":177,"lbm_reads_lt_1ms":8}
I20240501 08:13:18.809199  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling CompactRowSetsOp(7334b7b3bff94789be9b4ae942cb7503): perf score=0.022695
I20240501 08:13:28.108592  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: CompactRowSetsOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 9.297s	user 9.038s	sys 0.217s Metrics: {"bytes_written":1419392,"cfile_cache_hit":15,"cfile_cache_hit_bytes":5983,"cfile_cache_miss":609,"cfile_cache_miss_bytes":7225181,"cfile_init":7,"delta_iterators_relevant":6,"dirs.queue_time_us":1727,"dirs.run_cpu_time_us":1598,"dirs.run_wall_time_us":3204,"drs_written":1,"lbm_read_time_us":31186,"lbm_reads_lt_1ms":637,"lbm_write_time_us":54738,"lbm_writes_lt_1ms":614,"num_input_rowsets":3,"peak_mem_usage":5048546,"rows_written":515001,"thread_start_us":1763,"threads_started":5}
I20240501 08:13:28.109480  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling UndoDeltaBlockGCOp(7334b7b3bff94789be9b4ae942cb7503): 3461228 bytes on disk
I20240501 08:13:28.110002  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: UndoDeltaBlockGCOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240501 08:13:28.110504  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503): perf score=16.765403
I20240501 08:13:33.685026  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 5.574s	user 5.375s	sys 0.165s Metrics: {"bytes_written":471415,"cfile_init":1,"dirs.queue_time_us":379,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1160,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":21007,"lbm_writes_lt_1ms":223,"peak_mem_usage":0,"rows_written":177000,"spinlock_wait_cycles":18816,"thread_start_us":278,"threads_started":1}
I20240501 08:13:33.685902  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling UndoDeltaBlockGCOp(7334b7b3bff94789be9b4ae942cb7503): 1190135 bytes on disk
I20240501 08:13:33.686475  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: UndoDeltaBlockGCOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240501 08:13:33.687058  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503): perf score=3.866965
I20240501 08:13:36.324122  1934 maintenance_manager.cc:643] P a646ff4d66fd40539fbd5881bb499f81: FlushMRSOp(7334b7b3bff94789be9b4ae942cb7503) complete. Timing: real 2.637s	user 2.571s	sys 0.064s Metrics: {"bytes_written":232506,"cfile_init":1,"dirs.queue_time_us":414,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":956,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9670,"lbm_writes_lt_1ms":118,"peak_mem_usage":0,"rows_written":86000,"thread_start_us":295,"threads_started":1}
I20240501 08:13:36.325039  1993 maintenance_manager.cc:419] P a646ff4d66fd40539fbd5881bb499f81: Scheduling CompactRowSetsOp(7334b7b3bff94789be9b4ae942cb7503): perf score=0.022028
I20240501 08:13:47.008409  1834 update_scan_delta_compact-test.cc:209] Time spent Insert: real 62.766s	user 57.760s	sys 4.528s
I20240501 08:13:47.021690  2050 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:13:47.026026  2050 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:13:47.029011  2050 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:13:47.033077  2050 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:13:47.037016  2050 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:13:47.040519  2050 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:13:47.044623  2050 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:13:47.048177  2050 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:13:47.051712  2050 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:13:47.055356  2050 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:13:47.058903  2050 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:13:47.062515  2050 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:13:47.065964  2050 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 08:13:47.069504  2050 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240501 08:13:47.071926  2050 monotime.cc:342] Check failed: t_beg.Initialized() 
*** Check failure stack trace: ***
*** Aborted at 1714551227 (unix time) try "date -d @1714551227" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGABRT (@0x3e80000072a) received by PID 1834 (TID 0x7fdf7b3c2700) from PID 1834; stack trace: ***
    @     0x7fdfab9e1980 (unknown) at ??:0
    @     0x7fdfa4a8cfb7 gsignal at ??:0
    @     0x7fdfa4a8e921 abort at ??:0
    @     0x7fdfa77b5dcd google::LogMessage::Fail() at ??:0
    @     0x7fdfa77b9b93 google::LogMessage::SendToLog() at ??:0
    @     0x7fdfa77b57cc google::LogMessage::Flush() at ??:0
    @     0x7fdfa77b6f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7fdfa8d1ac80 kudu::operator-() at ??:0
    @     0x7fdfb453b14a kudu::tablet::WriteOp::ToString() at ??:0
    @     0x7fdfb45055de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
    @     0x7fdfb4505336 kudu::tablet::OpDriver::ToString() at ??:0
    @     0x7fdfb44b22d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
    @     0x7fdfbb330e47 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
    @     0x7fdfbb33f9fc kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
    @     0x7fdfbb33f862 std::_Function_handler<>::_M_invoke() at ??:0
    @     0x7fdfb169d02a std::function<>::operator()() at ??:0
    @     0x7fdfb16957ba kudu::Webserver::RunPathHandler() at ??:0
    @     0x7fdfb16907f1 kudu::Webserver::BeginRequestCallback() at ??:0
    @     0x7fdfb168e927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
    @     0x7fdfb16cc089 handle_request at ??:0
    @     0x7fdfb16ce76b worker_thread at ??:0
    @     0x7fdfab9d66db start_thread at ??:0
    @     0x7fdfa4b6f71f clone at ??:0