Diagnosed failure

UpdateScanDeltaCompactionTest.TestAll: Unrecognized error type. Please see the error log for more information.

Full log

[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from UpdateScanDeltaCompactionTest
[ RUN      ] UpdateScanDeltaCompactionTest.TestAll
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 13:59:15.418740 27862 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.27.53.190:35567
I20240501 13:59:15.420311 27862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:15.421541 27862 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:15.432032 27867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.432066 27870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.432117 27868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:15.432804 27862 server_base.cc:1034] running on GCE node
I20240501 13:59:15.433552 27862 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:15.433712 27862 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:15.433789 27862 hybrid_clock.cc:648] HybridClock initialized: now 1714571955433767 us; error 0 us; skew 500 ppm
I20240501 13:59:15.434113 27862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:15.441253 27862 webserver.cc:460] Webserver started at http://127.27.53.190:44841/ using document root <none> and password file <none>
I20240501 13:59:15.442027 27862 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:15.442127 27862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:15.442516 27862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:15.445786 27862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskutWn__/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714571955348389-27862-0/minicluster-data/master-0-root/instance:
uuid: "3c9dd59d519848db9303f5ebdb5f3642"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-1fqb"
I20240501 13:59:15.451764 27862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
I20240501 13:59:15.454941 27875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 13:59:15.456358 27862 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240501 13:59:15.456554 27862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskutWn__/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714571955348389-27862-0/minicluster-data/master-0-root
uuid: "3c9dd59d519848db9303f5ebdb5f3642"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-1fqb"
I20240501 13:59:15.456737 27862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskutWn__/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714571955348389-27862-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskutWn__/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714571955348389-27862-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskutWn__/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714571955348389-27862-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:15.497295 27862 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 13:59:15.498214 27862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:15.498535 27862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:15.517407 27862 rpc_server.cc:307] RPC server started. Bound to: 127.27.53.190:35567
I20240501 13:59:15.517429 27926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.53.190:35567 every 8 connection(s)
I20240501 13:59:15.521100 27927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:15.533316 27927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642: Bootstrap starting.
I20240501 13:59:15.537019 27927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:15.538218 27927 log.cc:824] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:15.540724 27927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642: No bootstrap required, opened a new log
I20240501 13:59:15.547516 27927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9dd59d519848db9303f5ebdb5f3642" member_type: VOTER }
I20240501 13:59:15.547799 27927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:15.547919 27927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9dd59d519848db9303f5ebdb5f3642, State: Initialized, Role: FOLLOWER
I20240501 13:59:15.548645 27927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9dd59d519848db9303f5ebdb5f3642" member_type: VOTER }
I20240501 13:59:15.548859 27927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:15.548995 27927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:15.549147 27927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:15.551046 27927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9dd59d519848db9303f5ebdb5f3642" member_type: VOTER }
I20240501 13:59:15.551527 27927 leader_election.cc:302] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c9dd59d519848db9303f5ebdb5f3642; no voters: 
I20240501 13:59:15.552129 27927 leader_election.cc:288] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 13:59:15.552253 27930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:15.552908 27930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [term 1 LEADER]: Becoming Leader. State: Replica: 3c9dd59d519848db9303f5ebdb5f3642, State: Running, Role: LEADER
I20240501 13:59:15.553488 27930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9dd59d519848db9303f5ebdb5f3642" member_type: VOTER }
I20240501 13:59:15.554198 27927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:15.556196 27931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c9dd59d519848db9303f5ebdb5f3642" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9dd59d519848db9303f5ebdb5f3642" member_type: VOTER } }
I20240501 13:59:15.556521 27931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:15.558745 27932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c9dd59d519848db9303f5ebdb5f3642. Latest consensus state: current_term: 1 leader_uuid: "3c9dd59d519848db9303f5ebdb5f3642" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9dd59d519848db9303f5ebdb5f3642" member_type: VOTER } }
I20240501 13:59:15.559144 27932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:15.559762 27938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:15.560402 27862 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 13:59:15.564504 27938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:15.572835 27938 catalog_manager.cc:1347] Generated new cluster ID: 827483185d384a029c0114e934ffa81d
I20240501 13:59:15.572961 27938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:15.587356 27938 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:15.588631 27938 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:15.603039 27938 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3c9dd59d519848db9303f5ebdb5f3642: Generated new TSK 0
I20240501 13:59:15.603874 27938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:15.625963 27862 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:15.630795 27950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.630774 27949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:15.632313 27952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:15.632668 27862 server_base.cc:1034] running on GCE node
I20240501 13:59:15.632926 27862 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:15.633013 27862 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:15.633077 27862 hybrid_clock.cc:648] HybridClock initialized: now 1714571955633054 us; error 0 us; skew 500 ppm
I20240501 13:59:15.633271 27862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:15.634780 27862 webserver.cc:460] Webserver started at http://127.27.53.129:38725/ using document root <none> and password file <none>
I20240501 13:59:15.635025 27862 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:15.635126 27862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:15.635282 27862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:15.636000 27862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskutWn__/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714571955348389-27862-0/minicluster-data/ts-0-root/instance:
uuid: "e42ab84f669342fb9c9ea77f1bf72495"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-1fqb"
I20240501 13:59:15.639024 27862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240501 13:59:15.640817 27957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240501 13:59:15.641206 27862 fs_manager.cc:725] Time spent opening block manager: real 0.001s	user 0.001s	sys 0.000s
I20240501 13:59:15.641320 27862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskutWn__/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714571955348389-27862-0/minicluster-data/ts-0-root
uuid: "e42ab84f669342fb9c9ea77f1bf72495"
format_stamp: "Formatted at 2024-05-01 13:59:15 on dist-test-slave-1fqb"
I20240501 13:59:15.641484 27862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskutWn__/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714571955348389-27862-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskutWn__/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714571955348389-27862-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskutWn__/test-tmp/update_scan_delta_compact-test.0.UpdateScanDeltaCompactionTest.TestAll.1714571955348389-27862-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:15.662336 27862 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240501 13:59:15.663030 27862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:15.663864 27862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:15.665306 27862 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:15.665427 27862 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240501 13:59:15.665565 27862 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:15.665647 27862 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240501 13:59:15.685954 27862 rpc_server.cc:307] RPC server started. Bound to: 127.27.53.129:35973
I20240501 13:59:15.685979 28019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.53.129:35973 every 8 connection(s)
I20240501 13:59:15.701699 28020 heartbeater.cc:347] Connected to a master server at 127.27.53.190:35567
I20240501 13:59:15.702047 28020 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:15.702929 28020 heartbeater.cc:511] Master 127.27.53.190:35567 requested a full tablet report, sending...
I20240501 13:59:15.705296 27892 ts_manager.cc:194] Registered new tserver with Master: e42ab84f669342fb9c9ea77f1bf72495 (127.27.53.129:35973)
I20240501 13:59:15.705401 27862 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.018076619s
I20240501 13:59:15.707808 27892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45970
I20240501 13:59:15.722313 27892 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45978:
name: "update-scan-delta-compact-tbl"
schema {
  columns {
    name: "key"
    type: INT64
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string"
    type: STRING
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int64"
    type: INT64
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20240501 13:59:15.755108 27985 tablet_service.cc:1467] Processing CreateTablet for tablet 4e323031a85a4ef6b259fd243bdecb78 (DEFAULT_TABLE table=update-scan-delta-compact-tbl [id=1b53e5906a8c4fb199f52280cb3b25fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 13:59:15.755868 27985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e323031a85a4ef6b259fd243bdecb78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:15.763329 28032 tablet_bootstrap.cc:492] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495: Bootstrap starting.
I20240501 13:59:15.764868 28032 tablet_bootstrap.cc:654] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:15.766691 28032 tablet_bootstrap.cc:492] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495: No bootstrap required, opened a new log
I20240501 13:59:15.766842 28032 ts_tablet_manager.cc:1367] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240501 13:59:15.767561 28032 raft_consensus.cc:348] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42ab84f669342fb9c9ea77f1bf72495" member_type: VOTER last_known_addr { host: "127.27.53.129" port: 35973 } }
I20240501 13:59:15.767807 28032 raft_consensus.cc:374] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:15.767927 28032 raft_consensus.cc:729] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42ab84f669342fb9c9ea77f1bf72495, State: Initialized, Role: FOLLOWER
I20240501 13:59:15.768203 28032 consensus_queue.cc:260] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42ab84f669342fb9c9ea77f1bf72495" member_type: VOTER last_known_addr { host: "127.27.53.129" port: 35973 } }
I20240501 13:59:15.768427 28032 raft_consensus.cc:388] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:15.768513 28032 raft_consensus.cc:482] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:15.768636 28032 raft_consensus.cc:3037] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:15.770516 28032 raft_consensus.cc:504] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42ab84f669342fb9c9ea77f1bf72495" member_type: VOTER last_known_addr { host: "127.27.53.129" port: 35973 } }
I20240501 13:59:15.770735 28032 leader_election.cc:302] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e42ab84f669342fb9c9ea77f1bf72495; no voters: 
I20240501 13:59:15.771245 28032 leader_election.cc:288] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240501 13:59:15.771417 28034 raft_consensus.cc:2781] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:15.771981 28034 raft_consensus.cc:686] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495 [term 1 LEADER]: Becoming Leader. State: Replica: e42ab84f669342fb9c9ea77f1bf72495, State: Running, Role: LEADER
I20240501 13:59:15.772608 28020 heartbeater.cc:503] Master 127.27.53.190:35567 was elected leader, sending a full tablet report...
I20240501 13:59:15.772363 28034 consensus_queue.cc:237] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42ab84f669342fb9c9ea77f1bf72495" member_type: VOTER last_known_addr { host: "127.27.53.129" port: 35973 } }
I20240501 13:59:15.778303 28032 ts_tablet_manager.cc:1398] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.005s
I20240501 13:59:15.781646 27892 catalog_manager.cc:5484] T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495 reported cstate change: term changed from 0 to 1, leader changed from <none> to e42ab84f669342fb9c9ea77f1bf72495 (127.27.53.129). New cstate: current_term: 1 leader_uuid: "e42ab84f669342fb9c9ea77f1bf72495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e42ab84f669342fb9c9ea77f1bf72495" member_type: VOTER last_known_addr { host: "127.27.53.129" port: 35973 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:19.688402 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78): perf score=1.882590
I20240501 13:59:22.346586 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 2.657s	user 2.290s	sys 0.340s Metrics: {"bytes_written":187587,"cfile_init":1,"compiler_manager_pool.queue_time_us":399,"compiler_manager_pool.run_cpu_time_us":432395,"compiler_manager_pool.run_wall_time_us":433593,"dirs.queue_time_us":3532,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1305,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6091,"lbm_writes_lt_1ms":98,"peak_mem_usage":0,"rows_written":67001,"thread_start_us":3702,"threads_started":2}
I20240501 13:59:22.348093 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling UndoDeltaBlockGCOp(4e323031a85a4ef6b259fd243bdecb78): 450505 bytes on disk
I20240501 13:59:22.348776 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: UndoDeltaBlockGCOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240501 13:59:24.349749 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78): perf score=2.874778
I20240501 13:59:26.699182 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 2.349s	user 2.287s	sys 0.060s Metrics: {"bytes_written":213165,"cfile_init":1,"dirs.queue_time_us":434,"dirs.run_cpu_time_us":311,"dirs.run_wall_time_us":1980,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":10108,"lbm_writes_lt_1ms":111,"peak_mem_usage":0,"rows_written":79000,"spinlock_wait_cycles":6016,"thread_start_us":293,"threads_started":1}
I20240501 13:59:26.700791 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling CompactRowSetsOp(4e323031a85a4ef6b259fd243bdecb78): perf score=0.014629
I20240501 13:59:29.382853 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: CompactRowSetsOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 2.682s	user 2.586s	sys 0.052s Metrics: {"bytes_written":403654,"cfile_cache_hit":10,"cfile_cache_hit_bytes":1918,"cfile_cache_miss":182,"cfile_cache_miss_bytes":2041868,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":2483,"dirs.run_cpu_time_us":1741,"dirs.run_wall_time_us":11256,"drs_written":1,"lbm_read_time_us":8444,"lbm_reads_lt_1ms":206,"lbm_write_time_us":14810,"lbm_writes_lt_1ms":188,"mutex_wait_us":44,"num_input_rowsets":2,"peak_mem_usage":1213228,"rows_written":146001,"thread_start_us":5204,"threads_started":4}
I20240501 13:59:29.383679 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78): perf score=3.866965
I20240501 13:59:31.721776 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 2.338s	user 2.262s	sys 0.072s Metrics: {"bytes_written":215039,"cfile_init":1,"dirs.queue_time_us":370,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1035,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9058,"lbm_writes_lt_1ms":111,"peak_mem_usage":0,"rows_written":80000,"spinlock_wait_cycles":18304,"thread_start_us":254,"threads_started":1}
I20240501 13:59:31.722879 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling UndoDeltaBlockGCOp(4e323031a85a4ef6b259fd243bdecb78): 1516768 bytes on disk
I20240501 13:59:31.723614 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: UndoDeltaBlockGCOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":117,"lbm_reads_lt_1ms":8}
I20240501 13:59:31.724335 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling CompactRowSetsOp(4e323031a85a4ef6b259fd243bdecb78): perf score=0.014427
I20240501 13:59:35.393671 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: CompactRowSetsOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 3.669s	user 3.615s	sys 0.052s Metrics: {"bytes_written":624522,"cfile_cache_hit":10,"cfile_cache_hit_bytes":2721,"cfile_cache_miss":270,"cfile_cache_miss_bytes":3169411,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":3184,"dirs.run_cpu_time_us":257,"dirs.run_wall_time_us":2507,"drs_written":1,"lbm_read_time_us":11548,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":289,"lbm_write_time_us":17485,"lbm_writes_lt_1ms":279,"num_input_rowsets":2,"peak_mem_usage":2334699,"rows_written":226001,"thread_start_us":2464,"threads_started":7}
I20240501 13:59:35.394670 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling UndoDeltaBlockGCOp(4e323031a85a4ef6b259fd243bdecb78): 1515547 bytes on disk
I20240501 13:59:35.395203 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: UndoDeltaBlockGCOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240501 13:59:35.395722 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78): perf score=5.851340
I20240501 13:59:38.153164 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 2.757s	user 2.595s	sys 0.148s Metrics: {"bytes_written":259542,"cfile_init":1,"dirs.queue_time_us":381,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":873,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9200,"lbm_writes_lt_1ms":129,"peak_mem_usage":0,"rows_written":96000,"thread_start_us":247,"threads_started":1}
I20240501 13:59:38.154134 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling CompactRowSetsOp(4e323031a85a4ef6b259fd243bdecb78): perf score=0.014184
W20240501 13:59:39.629701 28037 log.cc:924] Time spent T 4e323031a85a4ef6b259fd243bdecb78 P e42ab84f669342fb9c9ea77f1bf72495: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20240501 13:59:43.745182 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: CompactRowSetsOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 5.591s	user 5.251s	sys 0.184s Metrics: {"bytes_written":887881,"cfile_cache_hit":10,"cfile_cache_hit_bytes":3759,"cfile_cache_miss":374,"cfile_cache_miss_bytes":4510595,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":1532,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":85665,"drs_written":1,"lbm_read_time_us":75825,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":397,"lbm_write_time_us":28585,"lbm_writes_lt_1ms":390,"num_input_rowsets":2,"peak_mem_usage":3634615,"rows_written":322001,"thread_start_us":1402,"threads_started":1}
I20240501 13:59:43.746127 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78): perf score=10.812278
I20240501 13:59:47.889104 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 4.143s	user 3.928s	sys 0.204s Metrics: {"bytes_written":368695,"cfile_init":1,"dirs.queue_time_us":431,"dirs.run_cpu_time_us":270,"dirs.run_wall_time_us":1269,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":14357,"lbm_writes_lt_1ms":178,"peak_mem_usage":0,"rows_written":137000,"thread_start_us":274,"threads_started":1}
I20240501 13:59:47.890677 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78): perf score=1.000000
I20240501 13:59:49.777186 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 1.886s	user 1.843s	sys 0.016s Metrics: {"bytes_written":176494,"cfile_init":1,"dirs.queue_time_us":373,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":962,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6046,"lbm_writes_lt_1ms":93,"peak_mem_usage":0,"rows_written":65000,"thread_start_us":270,"threads_started":1}
I20240501 13:59:49.777984 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling UndoDeltaBlockGCOp(4e323031a85a4ef6b259fd243bdecb78): 3518100 bytes on disk
I20240501 13:59:49.778743 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: UndoDeltaBlockGCOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 0.001s	user 0.000s	sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":160,"lbm_reads_lt_1ms":12}
I20240501 13:59:49.779318 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling CompactRowSetsOp(4e323031a85a4ef6b259fd243bdecb78): perf score=0.022672
I20240501 13:59:58.391124 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: CompactRowSetsOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 8.612s	user 8.347s	sys 0.213s Metrics: {"bytes_written":1442961,"cfile_cache_hit":15,"cfile_cache_hit_bytes":6036,"cfile_cache_miss":611,"cfile_cache_miss_bytes":7344786,"cfile_init":7,"delta_iterators_relevant":6,"dirs.queue_time_us":1529,"dirs.run_cpu_time_us":1869,"dirs.run_wall_time_us":1884,"drs_written":1,"lbm_read_time_us":23483,"lbm_reads_lt_1ms":639,"lbm_write_time_us":42668,"lbm_writes_lt_1ms":623,"mutex_wait_us":310,"num_input_rowsets":3,"peak_mem_usage":5227286,"rows_written":524001,"thread_start_us":1731,"threads_started":5}
I20240501 13:59:58.391801 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling UndoDeltaBlockGCOp(4e323031a85a4ef6b259fd243bdecb78): 3522191 bytes on disk
I20240501 13:59:58.392325 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: UndoDeltaBlockGCOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240501 13:59:58.392812 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78): perf score=15.773215
I20240501 14:00:03.333449 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 4.940s	user 4.802s	sys 0.114s Metrics: {"bytes_written":452016,"cfile_init":1,"dirs.queue_time_us":343,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1123,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":16672,"lbm_writes_lt_1ms":214,"peak_mem_usage":0,"rows_written":169000,"thread_start_us":235,"threads_started":1}
I20240501 14:00:03.334352 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling UndoDeltaBlockGCOp(4e323031a85a4ef6b259fd243bdecb78): 1136073 bytes on disk
I20240501 14:00:03.334929 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: UndoDeltaBlockGCOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240501 14:00:03.335479 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78): perf score=2.874778
I20240501 14:00:05.619580 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: FlushMRSOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 2.284s	user 2.254s	sys 0.028s Metrics: {"bytes_written":213913,"cfile_init":1,"dirs.queue_time_us":384,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":988,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7202,"lbm_writes_lt_1ms":111,"peak_mem_usage":0,"rows_written":79000,"thread_start_us":271,"threads_started":1}
I20240501 14:00:05.620397 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling UndoDeltaBlockGCOp(4e323031a85a4ef6b259fd243bdecb78): 530290 bytes on disk
I20240501 14:00:05.621040 27962 maintenance_manager.cc:643] P e42ab84f669342fb9c9ea77f1bf72495: UndoDeltaBlockGCOp(4e323031a85a4ef6b259fd243bdecb78) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240501 14:00:05.621690 28021 maintenance_manager.cc:419] P e42ab84f669342fb9c9ea77f1bf72495: Scheduling CompactRowSetsOp(4e323031a85a4ef6b259fd243bdecb78): perf score=0.022043
I20240501 14:00:16.905941 27862 update_scan_delta_compact-test.cc:209] Time spent Insert: real 61.108s	user 55.960s	sys 4.515s
I20240501 14:00:16.940085 28078 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:00:16.944607 28078 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:00:16.947835 28078 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:00:16.950793 28078 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:00:16.953756 28078 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:00:16.956853 28078 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
I20240501 14:00:16.960335 28078 scanners.cc:360] Slow scans show is disabled. Set --show_slow_scans to enable it.
F20240501 14:00:16.961829 28078 monotime.cc:342] Check failed: t_beg.Initialized() 
*** Check failure stack trace: ***
*** Aborted at 1714572016 (unix time) try "date -d @1714572016" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGABRT (@0x3e800006cd6) received by PID 27862 (TID 0x7f500fd83700) from PID 27862; stack trace: ***
    @     0x7f5040399980 (unknown) at ??:0
    @     0x7f5039444fb7 gsignal at ??:0
    @     0x7f5039446921 abort at ??:0
    @     0x7f503c16ddcd google::LogMessage::Fail() at ??:0
    @     0x7f503c171b93 google::LogMessage::SendToLog() at ??:0
    @     0x7f503c16d7cc google::LogMessage::Flush() at ??:0
    @     0x7f503c16ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f503d6d2c80 kudu::operator-() at ??:0
    @     0x7f5048ef314a kudu::tablet::WriteOp::ToString() at ??:0
    @     0x7f5048ebd5de kudu::tablet::OpDriver::ToStringUnlocked() at ??:0
    @     0x7f5048ebd336 kudu::tablet::OpDriver::ToString() at ??:0
    @     0x7f5048e6a2d9 kudu::tablet::TabletReplica::GetInFlightOps() at ??:0
    @     0x7f504fce8e47 kudu::tserver::TabletServerPathHandlers::HandleTransactionsPage() at ??:0
    @     0x7f504fcf79fc kudu::tserver::TabletServerPathHandlers::Register()::$_3::operator()() at ??:0
    @     0x7f504fcf7862 std::_Function_handler<>::_M_invoke() at ??:0
    @     0x7f504605502a std::function<>::operator()() at ??:0
    @     0x7f504604d7ba kudu::Webserver::RunPathHandler() at ??:0
    @     0x7f50460487f1 kudu::Webserver::BeginRequestCallback() at ??:0
    @     0x7f5046046927 kudu::Webserver::BeginRequestCallbackStatic() at ??:0
    @     0x7f5046084089 handle_request at ??:0
    @     0x7f504608676b worker_thread at ??:0
    @     0x7f504038e6db start_thread at ??:0
    @     0x7f503952771f clone at ??:0