Note: This is test shard 1 of 6.
[==========] Running 5 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 4 tests from TabletCopyITest
[ RUN ] TabletCopyITest.TestRejectRogueLeader
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:172: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] TabletCopyITest.TestRejectRogueLeader (18 ms)
[ RUN ] TabletCopyITest.TestDeleteLeaderDuringTabletCopyStressTest
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:727: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] TabletCopyITest.TestDeleteLeaderDuringTabletCopyStressTest (9 ms)
[ RUN ] TabletCopyITest.TestTabletCopyThrottling
2026-05-05T13:57:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-05T13:57:15Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260505 13:57:15.600073 6116 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
/tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.249.62:43179
--webserver_interface=127.5.249.62
--webserver_port=0
--builtin_ntp_servers=127.5.249.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.249.62:43179
--master_tombstone_evicted_tablet_replicas=false with env {}
W20260505 13:57:16.096079 6125 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260505 13:57:16.096956 6125 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260505 13:57:16.097426 6125 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260505 13:57:16.134847 6125 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260505 13:57:16.135244 6125 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260505 13:57:16.135573 6125 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260505 13:57:16.135854 6125 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260505 13:57:16.178099 6125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.249.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/wal
--master_tombstone_evicted_tablet_replicas=false
--ipki_ca_key_size=768
--master_addresses=127.5.249.62:43179
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.249.62:43179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.249.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 05 May 2026 13:43:23 UTC on e7f111948823
build id 11759
TSAN enabled
I20260505 13:57:16.179904 6125 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260505 13:57:16.181928 6125 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 13:57:16.200228 6131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:17.603782 6130 debug-util.cc:398] Leaking SignalData structure 0x7b0800037b60 after lost signal to thread 6125
W20260505 13:57:16.204823 6132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:18.200956 6130 kernel_stack_watchdog.cc:198] Thread 6125 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 391ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260505 13:57:18.201660 6125 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.994s user 0.684s sys 1.305s
W20260505 13:57:18.202303 6125 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.996s user 0.684s sys 1.305s
W20260505 13:57:18.205096 6134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:18.207477 6133 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1993 milliseconds
I20260505 13:57:18.207484 6125 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260505 13:57:18.209096 6125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260505 13:57:18.212520 6125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260505 13:57:18.213986 6125 hybrid_clock.cc:648] HybridClock initialized: now 1777989438213839 us; error 187 us; skew 500 ppm
I20260505 13:57:18.214951 6125 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 13:57:18.226888 6125 webserver.cc:492] Webserver started at http://127.5.249.62:42493/ using document root <none> and password file <none>
I20260505 13:57:18.228008 6125 fs_manager.cc:362] Metadata directory not provided
I20260505 13:57:18.228278 6125 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 13:57:18.228825 6125 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 13:57:18.234344 6125 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data/instance:
uuid: "9cf7931f25dc4480bbdf8487d28ac369"
format_stamp: "Formatted at 2026-05-05 13:57:18 on dist-test-slave-fz8k"
I20260505 13:57:18.235595 6125 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/wal/instance:
uuid: "9cf7931f25dc4480bbdf8487d28ac369"
format_stamp: "Formatted at 2026-05-05 13:57:18 on dist-test-slave-fz8k"
I20260505 13:57:18.245923 6125 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.002s sys 0.006s
I20260505 13:57:18.255235 6141 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:18.256668 6125 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.001s sys 0.006s
I20260505 13:57:18.257033 6125 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data,/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/wal
uuid: "9cf7931f25dc4480bbdf8487d28ac369"
format_stamp: "Formatted at 2026-05-05 13:57:18 on dist-test-slave-fz8k"
I20260505 13:57:18.257409 6125 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 13:57:18.331871 6125 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 13:57:18.334183 6125 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260505 13:57:18.334800 6125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 13:57:18.534370 6125 rpc_server.cc:307] RPC server started. Bound to: 127.5.249.62:43179
I20260505 13:57:18.534535 6192 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.249.62:43179 every 8 connection(s)
I20260505 13:57:18.537663 6125 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data/info.pb
I20260505 13:57:18.543190 6116 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu as pid 6125
I20260505 13:57:18.544549 6116 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/wal/instance
I20260505 13:57:18.550516 6193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 13:57:18.598917 6193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369: Bootstrap starting.
I20260505 13:57:18.610703 6193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369: Neither blocks nor log segments found. Creating new log.
I20260505 13:57:18.614351 6193 log.cc:826] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369: Log is configured to *not* fsync() on all Append() calls
I20260505 13:57:18.624977 6193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369: No bootstrap required, opened a new log
I20260505 13:57:18.652185 6193 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf7931f25dc4480bbdf8487d28ac369" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 43179 } }
I20260505 13:57:18.653108 6193 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 13:57:18.653584 6193 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf7931f25dc4480bbdf8487d28ac369, State: Initialized, Role: FOLLOWER
I20260505 13:57:18.656431 6193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf7931f25dc4480bbdf8487d28ac369" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 43179 } }
I20260505 13:57:18.657192 6193 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 13:57:18.657596 6193 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 13:57:18.658044 6193 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 0 FOLLOWER]: Advancing to term 1
I20260505 13:57:18.665558 6193 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf7931f25dc4480bbdf8487d28ac369" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 43179 } }
I20260505 13:57:18.666575 6193 leader_election.cc:304] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cf7931f25dc4480bbdf8487d28ac369; no voters:
I20260505 13:57:18.670833 6193 leader_election.cc:290] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 13:57:18.671639 6198 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 13:57:18.679827 6198 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 1 LEADER]: Becoming Leader. State: Replica: 9cf7931f25dc4480bbdf8487d28ac369, State: Running, Role: LEADER
I20260505 13:57:18.681401 6198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf7931f25dc4480bbdf8487d28ac369" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 43179 } }
I20260505 13:57:18.683313 6193 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [sys.catalog]: configured and running, proceeding with master startup.
I20260505 13:57:18.701081 6200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cf7931f25dc4480bbdf8487d28ac369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf7931f25dc4480bbdf8487d28ac369" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 43179 } } }
I20260505 13:57:18.702409 6200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [sys.catalog]: This master's current role is: LEADER
I20260505 13:57:18.704788 6199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cf7931f25dc4480bbdf8487d28ac369. Latest consensus state: current_term: 1 leader_uuid: "9cf7931f25dc4480bbdf8487d28ac369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf7931f25dc4480bbdf8487d28ac369" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 43179 } } }
I20260505 13:57:18.708303 6199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [sys.catalog]: This master's current role is: LEADER
I20260505 13:57:18.715425 6206 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 13:57:18.739568 6206 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 13:57:18.777721 6206 catalog_manager.cc:1357] Generated new cluster ID: eb74a5f99a9640f28fa45501764abc75
I20260505 13:57:18.778106 6206 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 13:57:18.819491 6206 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 13:57:18.821753 6206 catalog_manager.cc:1514] Loading token signing keys...
I20260505 13:57:18.839049 6206 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369: Generated new TSK 0
I20260505 13:57:18.840884 6206 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 13:57:18.864158 6116 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
/tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.249.1:0
--local_ip_for_outbound_sockets=127.5.249.1
--webserver_interface=127.5.249.1
--webserver_port=0
--tserver_master_addrs=127.5.249.62:43179
--builtin_ntp_servers=127.5.249.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--num_tablets_to_copy_simultaneously=1 with env {}
W20260505 13:57:19.350700 6217 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260505 13:57:19.351413 6217 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260505 13:57:19.351939 6217 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260505 13:57:19.387616 6217 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260505 13:57:19.388640 6217 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.249.1
I20260505 13:57:19.428524 6217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.249.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.249.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.249.1
--webserver_port=0
--tserver_master_addrs=127.5.249.62:43179
--num_tablets_to_copy_simultaneously=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.249.1
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 05 May 2026 13:43:23 UTC on e7f111948823
build id 11759
TSAN enabled
I20260505 13:57:19.430186 6217 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260505 13:57:19.432089 6217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 13:57:19.451503 6223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:20.858036 6222 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ca0 after lost signal to thread 6217
W20260505 13:57:19.456524 6224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:21.151943 6217 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.694s user 0.473s sys 1.123s
W20260505 13:57:21.155987 6217 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.699s user 0.473s sys 1.123s
W20260505 13:57:21.156944 6226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:21.159641 6225 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1693 milliseconds
I20260505 13:57:21.159642 6217 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260505 13:57:21.161854 6217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260505 13:57:21.165324 6217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260505 13:57:21.166838 6217 hybrid_clock.cc:648] HybridClock initialized: now 1777989441166763 us; error 123 us; skew 500 ppm
I20260505 13:57:21.167817 6217 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 13:57:21.179379 6217 webserver.cc:492] Webserver started at http://127.5.249.1:42209/ using document root <none> and password file <none>
I20260505 13:57:21.180605 6217 fs_manager.cc:362] Metadata directory not provided
I20260505 13:57:21.180913 6217 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 13:57:21.181397 6217 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 13:57:21.189194 6217 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/data/instance:
uuid: "2e5321dafcdb4d33af8a8e1b636f0de7"
format_stamp: "Formatted at 2026-05-05 13:57:21 on dist-test-slave-fz8k"
I20260505 13:57:21.190444 6217 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/wal/instance:
uuid: "2e5321dafcdb4d33af8a8e1b636f0de7"
format_stamp: "Formatted at 2026-05-05 13:57:21 on dist-test-slave-fz8k"
I20260505 13:57:21.201882 6217 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.005s sys 0.006s
I20260505 13:57:21.211963 6233 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:21.213550 6217 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.002s sys 0.005s
I20260505 13:57:21.213928 6217 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/wal
uuid: "2e5321dafcdb4d33af8a8e1b636f0de7"
format_stamp: "Formatted at 2026-05-05 13:57:21 on dist-test-slave-fz8k"
I20260505 13:57:21.214299 6217 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 13:57:21.291047 6217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 13:57:21.293694 6217 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260505 13:57:21.294348 6217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 13:57:21.301362 6217 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 13:57:21.308837 6217 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 13:57:21.309180 6217 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:21.309549 6217 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 13:57:21.309765 6217 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:21.791708 6217 rpc_server.cc:307] RPC server started. Bound to: 127.5.249.1:40903
I20260505 13:57:21.792061 6345 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.249.1:40903 every 8 connection(s)
I20260505 13:57:21.795002 6217 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/data/info.pb
I20260505 13:57:21.801610 6116 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu as pid 6217
I20260505 13:57:21.802418 6116 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-0/wal/instance
I20260505 13:57:21.821288 6116 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
/tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.249.2:0
--local_ip_for_outbound_sockets=127.5.249.2
--webserver_interface=127.5.249.2
--webserver_port=0
--tserver_master_addrs=127.5.249.62:43179
--builtin_ntp_servers=127.5.249.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--num_tablets_to_copy_simultaneously=1 with env {}
I20260505 13:57:21.858492 6346 heartbeater.cc:344] Connected to a master server at 127.5.249.62:43179
I20260505 13:57:21.859187 6346 heartbeater.cc:461] Registering TS with master...
I20260505 13:57:21.861266 6346 heartbeater.cc:507] Master 127.5.249.62:43179 requested a full tablet report, sending...
I20260505 13:57:21.865672 6158 ts_manager.cc:194] Registered new tserver with Master: 2e5321dafcdb4d33af8a8e1b636f0de7 (127.5.249.1:40903)
I20260505 13:57:21.869865 6158 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.5.249.1:38209
W20260505 13:57:22.312125 6350 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260505 13:57:22.312857 6350 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260505 13:57:22.313418 6350 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260505 13:57:22.348193 6350 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260505 13:57:22.349155 6350 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.249.2
I20260505 13:57:22.389079 6350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.249.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.249.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.249.2
--webserver_port=0
--tserver_master_addrs=127.5.249.62:43179
--num_tablets_to_copy_simultaneously=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.249.2
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 05 May 2026 13:43:23 UTC on e7f111948823
build id 11759
TSAN enabled
I20260505 13:57:22.390767 6350 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260505 13:57:22.392784 6350 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 13:57:22.412673 6356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 13:57:22.875989 6346 heartbeater.cc:499] Master 127.5.249.62:43179 was elected leader, sending a full tablet report...
W20260505 13:57:23.814999 6355 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ca0 after lost signal to thread 6350
W20260505 13:57:24.322031 6355 kernel_stack_watchdog.cc:198] Thread 6350 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 393ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260505 13:57:22.415470 6357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:24.328081 6350 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.911s user 0.000s sys 0.005s
W20260505 13:57:24.328922 6350 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.912s user 0.001s sys 0.005s
W20260505 13:57:24.334945 6358 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1916 milliseconds
W20260505 13:57:24.336009 6359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 13:57:24.336062 6350 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260505 13:57:24.337534 6350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260505 13:57:24.340739 6350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260505 13:57:24.342103 6350 hybrid_clock.cc:648] HybridClock initialized: now 1777989444342056 us; error 55 us; skew 500 ppm
I20260505 13:57:24.343014 6350 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 13:57:24.354175 6350 webserver.cc:492] Webserver started at http://127.5.249.2:33479/ using document root <none> and password file <none>
I20260505 13:57:24.355288 6350 fs_manager.cc:362] Metadata directory not provided
I20260505 13:57:24.355527 6350 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 13:57:24.355995 6350 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 13:57:24.361593 6350 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data/instance:
uuid: "c5e39347ea38406fabb2ab795a53d363"
format_stamp: "Formatted at 2026-05-05 13:57:24 on dist-test-slave-fz8k"
I20260505 13:57:24.362978 6350 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/wal/instance:
uuid: "c5e39347ea38406fabb2ab795a53d363"
format_stamp: "Formatted at 2026-05-05 13:57:24 on dist-test-slave-fz8k"
I20260505 13:57:24.373160 6350 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s
I20260505 13:57:24.382458 6366 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:24.383658 6350 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.005s
I20260505 13:57:24.384059 6350 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/wal
uuid: "c5e39347ea38406fabb2ab795a53d363"
format_stamp: "Formatted at 2026-05-05 13:57:24 on dist-test-slave-fz8k"
I20260505 13:57:24.384555 6350 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 13:57:24.445093 6350 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 13:57:24.447496 6350 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260505 13:57:24.448091 6350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 13:57:24.453333 6350 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 13:57:24.460282 6350 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 13:57:24.460530 6350 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:24.460824 6350 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 13:57:24.461016 6350 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:24.882107 6350 rpc_server.cc:307] RPC server started. Bound to: 127.5.249.2:42423
I20260505 13:57:24.882232 6478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.249.2:42423 every 8 connection(s)
I20260505 13:57:24.885397 6350 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data/info.pb
I20260505 13:57:24.893421 6116 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu as pid 6350
I20260505 13:57:24.894214 6116 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/wal/instance
I20260505 13:57:24.933313 6479 heartbeater.cc:344] Connected to a master server at 127.5.249.62:43179
I20260505 13:57:24.934018 6479 heartbeater.cc:461] Registering TS with master...
I20260505 13:57:24.936138 6479 heartbeater.cc:507] Master 127.5.249.62:43179 requested a full tablet report, sending...
I20260505 13:57:24.939013 6158 ts_manager.cc:194] Registered new tserver with Master: c5e39347ea38406fabb2ab795a53d363 (127.5.249.2:42423)
I20260505 13:57:24.940855 6158 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.5.249.2:42745
I20260505 13:57:24.950909 6116 external_mini_cluster.cc:949] 2 TS(s) registered with all masters
I20260505 13:57:25.009701 6116 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu with pid 6350
I20260505 13:57:25.032768 6116 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu with pid 6125
I20260505 13:57:25.066337 6116 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
/tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.249.62:43179
--webserver_interface=127.5.249.62
--webserver_port=42493
--builtin_ntp_servers=127.5.249.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.249.62:43179
--master_tombstone_evicted_tablet_replicas=false with env {}
W20260505 13:57:25.903111 6346 heartbeater.cc:646] Failed to heartbeat to 127.5.249.62:43179 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.249.62:43179: connect: Connection refused (error 111)
W20260505 13:57:26.661428 6491 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260505 13:57:26.662163 6491 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260505 13:57:26.662698 6491 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260505 13:57:26.697929 6491 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260505 13:57:26.698310 6491 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260505 13:57:26.698599 6491 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260505 13:57:26.698880 6491 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260505 13:57:26.737830 6491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.249.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/wal
--master_tombstone_evicted_tablet_replicas=false
--ipki_ca_key_size=768
--master_addresses=127.5.249.62:43179
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.249.62:43179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.249.62
--webserver_port=42493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 05 May 2026 13:43:23 UTC on e7f111948823
build id 11759
TSAN enabled
I20260505 13:57:26.739485 6491 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260505 13:57:26.741535 6491 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 13:57:26.754727 6497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:28.158213 6496 debug-util.cc:398] Leaking SignalData structure 0x7b0800037b60 after lost signal to thread 6491
W20260505 13:57:28.571825 6491 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.817s user 0.623s sys 1.192s
W20260505 13:57:26.755606 6498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:28.572392 6491 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.818s user 0.624s sys 1.192s
W20260505 13:57:28.574658 6500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:28.577625 6499 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1818 milliseconds
I20260505 13:57:28.577647 6491 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260505 13:57:28.579103 6491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260505 13:57:28.582149 6491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260505 13:57:28.583559 6491 hybrid_clock.cc:648] HybridClock initialized: now 1777989448583479 us; error 75 us; skew 500 ppm
I20260505 13:57:28.584519 6491 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 13:57:28.591470 6491 webserver.cc:492] Webserver started at http://127.5.249.62:42493/ using document root <none> and password file <none>
I20260505 13:57:28.592626 6491 fs_manager.cc:362] Metadata directory not provided
I20260505 13:57:28.592923 6491 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 13:57:28.601704 6491 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.002s
I20260505 13:57:28.607033 6509 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:28.608235 6491 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.005s
I20260505 13:57:28.608596 6491 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data,/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/wal
uuid: "9cf7931f25dc4480bbdf8487d28ac369"
format_stamp: "Formatted at 2026-05-05 13:57:18 on dist-test-slave-fz8k"
I20260505 13:57:28.610788 6491 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 13:57:28.671494 6491 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 13:57:28.673355 6491 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260505 13:57:28.673887 6491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 13:57:28.757952 6491 rpc_server.cc:307] RPC server started. Bound to: 127.5.249.62:43179
I20260505 13:57:28.758041 6560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.249.62:43179 every 8 connection(s)
I20260505 13:57:28.761935 6491 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/master-0/data/info.pb
I20260505 13:57:28.769187 6116 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu as pid 6491
I20260505 13:57:28.776181 6561 sys_catalog.cc:263] Verifying existing consensus state
I20260505 13:57:28.784390 6561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369: Bootstrap starting.
I20260505 13:57:28.829962 6561 log.cc:826] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369: Log is configured to *not* fsync() on all Append() calls
I20260505 13:57:28.846401 6561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260505 13:57:28.847323 6561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369: Bootstrap complete.
I20260505 13:57:28.869165 6561 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf7931f25dc4480bbdf8487d28ac369" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 43179 } }
I20260505 13:57:28.871403 6561 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf7931f25dc4480bbdf8487d28ac369, State: Initialized, Role: FOLLOWER
I20260505 13:57:28.872341 6561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf7931f25dc4480bbdf8487d28ac369" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 43179 } }
I20260505 13:57:28.872885 6561 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 13:57:28.873195 6561 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 13:57:28.873559 6561 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 1 FOLLOWER]: Advancing to term 2
I20260505 13:57:28.880277 6561 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf7931f25dc4480bbdf8487d28ac369" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 43179 } }
I20260505 13:57:28.881194 6561 leader_election.cc:304] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cf7931f25dc4480bbdf8487d28ac369; no voters:
I20260505 13:57:28.883180 6561 leader_election.cc:290] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260505 13:57:28.883677 6566 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 2 FOLLOWER]: Leader election won for term 2
I20260505 13:57:28.886266 6566 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [term 2 LEADER]: Becoming Leader. State: Replica: 9cf7931f25dc4480bbdf8487d28ac369, State: Running, Role: LEADER
I20260505 13:57:28.887099 6566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf7931f25dc4480bbdf8487d28ac369" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 43179 } }
I20260505 13:57:28.888264 6561 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [sys.catalog]: configured and running, proceeding with master startup.
I20260505 13:57:28.896845 6567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9cf7931f25dc4480bbdf8487d28ac369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf7931f25dc4480bbdf8487d28ac369" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 43179 } } }
I20260505 13:57:28.896099 6568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cf7931f25dc4480bbdf8487d28ac369. Latest consensus state: current_term: 2 leader_uuid: "9cf7931f25dc4480bbdf8487d28ac369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cf7931f25dc4480bbdf8487d28ac369" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 43179 } } }
I20260505 13:57:28.897563 6567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [sys.catalog]: This master's current role is: LEADER
I20260505 13:57:28.897755 6568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369 [sys.catalog]: This master's current role is: LEADER
I20260505 13:57:28.901777 6572 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 13:57:28.917721 6572 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 13:57:28.922612 6572 catalog_manager.cc:1269] Loaded cluster ID: eb74a5f99a9640f28fa45501764abc75
I20260505 13:57:28.923007 6572 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 13:57:28.929673 6572 catalog_manager.cc:1514] Loading token signing keys...
I20260505 13:57:28.933101 6572 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 9cf7931f25dc4480bbdf8487d28ac369: Loaded TSK: 0
I20260505 13:57:28.934471 6572 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 13:57:28.953708 6346 heartbeater.cc:344] Connected to a master server at 127.5.249.62:43179
I20260505 13:57:28.960034 6525 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" instance_seqno: 1777989441690827) as {username='slave'} at 127.5.249.1:33849; Asking this server to re-register.
I20260505 13:57:28.961629 6346 heartbeater.cc:461] Registering TS with master...
I20260505 13:57:28.962325 6346 heartbeater.cc:507] Master 127.5.249.62:43179 requested a full tablet report, sending...
I20260505 13:57:28.965029 6525 ts_manager.cc:194] Registered new tserver with Master: 2e5321dafcdb4d33af8a8e1b636f0de7 (127.5.249.1:40903)
I20260505 13:57:28.968114 6116 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20260505 13:57:29.032454 6526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55252:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260505 13:57:29.092351 6278 tablet_service.cc:1511] Processing CreateTablet for tablet ce2dafafeb1444c0a0dea4f7f68b309f (DEFAULT_TABLE table=test-workload [id=a47285a7cf094aeb823e2cb0eb7787f7]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20260505 13:57:29.093329 6281 tablet_service.cc:1511] Processing CreateTablet for tablet d82caf3e1b104c32a3c0c0df1b2e157d (DEFAULT_TABLE table=test-workload [id=a47285a7cf094aeb823e2cb0eb7787f7]), partition=RANGE (key) PARTITION VALUES < 536870911
I20260505 13:57:29.092803 6280 tablet_service.cc:1511] Processing CreateTablet for tablet 31781fc9f40049f8a50ee8dbd4c879ec (DEFAULT_TABLE table=test-workload [id=a47285a7cf094aeb823e2cb0eb7787f7]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20260505 13:57:29.092365 6279 tablet_service.cc:1511] Processing CreateTablet for tablet 4abb2e163f484a7993f6054876f9e0aa (DEFAULT_TABLE table=test-workload [id=a47285a7cf094aeb823e2cb0eb7787f7]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20260505 13:57:29.095624 6278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce2dafafeb1444c0a0dea4f7f68b309f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 13:57:29.096822 6280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31781fc9f40049f8a50ee8dbd4c879ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 13:57:29.098577 6281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d82caf3e1b104c32a3c0c0df1b2e157d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 13:57:29.099982 6279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4abb2e163f484a7993f6054876f9e0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 13:57:29.133811 6593 tablet_bootstrap.cc:492] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7: Bootstrap starting.
I20260505 13:57:29.143241 6593 tablet_bootstrap.cc:654] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7: Neither blocks nor log segments found. Creating new log.
I20260505 13:57:29.146507 6593 log.cc:826] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7: Log is configured to *not* fsync() on all Append() calls
I20260505 13:57:29.152822 6593 tablet_bootstrap.cc:492] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7: No bootstrap required, opened a new log
I20260505 13:57:29.153329 6593 ts_tablet_manager.cc:1403] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.011s
I20260505 13:57:29.173250 6593 raft_consensus.cc:359] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.173919 6593 raft_consensus.cc:385] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 13:57:29.174176 6593 raft_consensus.cc:740] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5321dafcdb4d33af8a8e1b636f0de7, State: Initialized, Role: FOLLOWER
I20260505 13:57:29.174950 6593 consensus_queue.cc:260] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.175451 6593 raft_consensus.cc:399] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 13:57:29.175735 6593 raft_consensus.cc:493] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 13:57:29.176054 6593 raft_consensus.cc:3060] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Advancing to term 1
I20260505 13:57:29.180804 6593 raft_consensus.cc:515] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.181597 6593 leader_election.cc:304] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e5321dafcdb4d33af8a8e1b636f0de7; no voters:
I20260505 13:57:29.184176 6593 leader_election.cc:290] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 13:57:29.184644 6595 raft_consensus.cc:2804] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 13:57:29.188621 6595 raft_consensus.cc:697] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 1 LEADER]: Becoming Leader. State: Replica: 2e5321dafcdb4d33af8a8e1b636f0de7, State: Running, Role: LEADER
I20260505 13:57:29.188905 6593 ts_tablet_manager.cc:1434] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7: Time spent starting tablet: real 0.035s user 0.029s sys 0.007s
I20260505 13:57:29.189903 6593 tablet_bootstrap.cc:492] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7: Bootstrap starting.
I20260505 13:57:29.189528 6595 consensus_queue.cc:237] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.198076 6593 tablet_bootstrap.cc:654] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7: Neither blocks nor log segments found. Creating new log.
I20260505 13:57:29.208401 6526 catalog_manager.cc:5671] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e5321dafcdb4d33af8a8e1b636f0de7 (127.5.249.1). New cstate: current_term: 1 leader_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } health_report { overall_health: HEALTHY } } }
I20260505 13:57:29.241122 6593 tablet_bootstrap.cc:492] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7: No bootstrap required, opened a new log
I20260505 13:57:29.241597 6593 ts_tablet_manager.cc:1403] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7: Time spent bootstrapping tablet: real 0.052s user 0.015s sys 0.035s
I20260505 13:57:29.244598 6593 raft_consensus.cc:359] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.245180 6593 raft_consensus.cc:385] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 13:57:29.245468 6593 raft_consensus.cc:740] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5321dafcdb4d33af8a8e1b636f0de7, State: Initialized, Role: FOLLOWER
I20260505 13:57:29.246125 6593 consensus_queue.cc:260] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.246765 6593 raft_consensus.cc:399] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 13:57:29.247056 6593 raft_consensus.cc:493] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 13:57:29.247404 6593 raft_consensus.cc:3060] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Advancing to term 1
I20260505 13:57:29.254462 6593 raft_consensus.cc:515] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.255142 6593 leader_election.cc:304] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e5321dafcdb4d33af8a8e1b636f0de7; no voters:
I20260505 13:57:29.255740 6593 leader_election.cc:290] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 13:57:29.255869 6595 raft_consensus.cc:2804] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 13:57:29.256397 6595 raft_consensus.cc:697] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 1 LEADER]: Becoming Leader. State: Replica: 2e5321dafcdb4d33af8a8e1b636f0de7, State: Running, Role: LEADER
I20260505 13:57:29.257385 6593 ts_tablet_manager.cc:1434] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260505 13:57:29.257124 6595 consensus_queue.cc:237] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.258226 6593 tablet_bootstrap.cc:492] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7: Bootstrap starting.
I20260505 13:57:29.266415 6593 tablet_bootstrap.cc:654] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7: Neither blocks nor log segments found. Creating new log.
I20260505 13:57:29.266214 6526 catalog_manager.cc:5671] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e5321dafcdb4d33af8a8e1b636f0de7 (127.5.249.1). New cstate: current_term: 1 leader_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } health_report { overall_health: HEALTHY } } }
I20260505 13:57:29.273094 6593 tablet_bootstrap.cc:492] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7: No bootstrap required, opened a new log
I20260505 13:57:29.273669 6593 ts_tablet_manager.cc:1403] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20260505 13:57:29.276399 6593 raft_consensus.cc:359] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.276928 6593 raft_consensus.cc:385] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 13:57:29.277199 6593 raft_consensus.cc:740] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5321dafcdb4d33af8a8e1b636f0de7, State: Initialized, Role: FOLLOWER
I20260505 13:57:29.277817 6593 consensus_queue.cc:260] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.278427 6593 raft_consensus.cc:399] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 13:57:29.278709 6593 raft_consensus.cc:493] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 13:57:29.279024 6593 raft_consensus.cc:3060] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Advancing to term 1
I20260505 13:57:29.285609 6593 raft_consensus.cc:515] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.286239 6593 leader_election.cc:304] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e5321dafcdb4d33af8a8e1b636f0de7; no voters:
I20260505 13:57:29.286870 6593 leader_election.cc:290] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 13:57:29.287003 6595 raft_consensus.cc:2804] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 13:57:29.287499 6595 raft_consensus.cc:697] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 1 LEADER]: Becoming Leader. State: Replica: 2e5321dafcdb4d33af8a8e1b636f0de7, State: Running, Role: LEADER
I20260505 13:57:29.288231 6595 consensus_queue.cc:237] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.288578 6593 ts_tablet_manager.cc:1434] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260505 13:57:29.289417 6593 tablet_bootstrap.cc:492] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7: Bootstrap starting.
I20260505 13:57:29.295980 6593 tablet_bootstrap.cc:654] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7: Neither blocks nor log segments found. Creating new log.
I20260505 13:57:29.296095 6526 catalog_manager.cc:5671] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e5321dafcdb4d33af8a8e1b636f0de7 (127.5.249.1). New cstate: current_term: 1 leader_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } health_report { overall_health: HEALTHY } } }
I20260505 13:57:29.302640 6593 tablet_bootstrap.cc:492] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7: No bootstrap required, opened a new log
I20260505 13:57:29.303081 6593 ts_tablet_manager.cc:1403] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260505 13:57:29.305902 6593 raft_consensus.cc:359] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.306445 6593 raft_consensus.cc:385] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 13:57:29.306718 6593 raft_consensus.cc:740] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e5321dafcdb4d33af8a8e1b636f0de7, State: Initialized, Role: FOLLOWER
I20260505 13:57:29.307322 6593 consensus_queue.cc:260] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.307934 6593 raft_consensus.cc:399] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 13:57:29.308308 6593 raft_consensus.cc:493] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 13:57:29.308650 6593 raft_consensus.cc:3060] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 0 FOLLOWER]: Advancing to term 1
I20260505 13:57:29.315512 6593 raft_consensus.cc:515] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.316290 6593 leader_election.cc:304] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e5321dafcdb4d33af8a8e1b636f0de7; no voters:
I20260505 13:57:29.316947 6593 leader_election.cc:290] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 13:57:29.317123 6595 raft_consensus.cc:2804] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 13:57:29.317627 6595 raft_consensus.cc:697] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7 [term 1 LEADER]: Becoming Leader. State: Replica: 2e5321dafcdb4d33af8a8e1b636f0de7, State: Running, Role: LEADER
I20260505 13:57:29.318310 6595 consensus_queue.cc:237] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:29.318641 6593 ts_tablet_manager.cc:1434] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
W20260505 13:57:29.337220 6347 tablet.cc:2404] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260505 13:57:29.339895 6526 catalog_manager.cc:5671] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e5321dafcdb4d33af8a8e1b636f0de7 (127.5.249.1). New cstate: current_term: 1 leader_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } health_report { overall_health: HEALTHY } } }
W20260505 13:57:30.241786 6586 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:40903, user_credentials={real_user=slave}} blocked reactor thread for 47431.4us
W20260505 13:57:30.702697 6612 meta_cache.cc:1261] Time spent looking up entry by key: real 0.059s user 0.001s sys 0.000s
W20260505 13:57:31.704754 6610 meta_cache.cc:1261] Time spent looking up entry by key: real 0.054s user 0.002s sys 0.000s
W20260505 13:57:32.185158 6586 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:40903, user_credentials={real_user=slave}} blocked reactor thread for 43933.9us
W20260505 13:57:33.073738 6611 meta_cache.cc:1261] Time spent looking up entry by key: real 0.053s user 0.001s sys 0.000s
I20260505 13:57:35.381543 6116 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
/tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.249.2:42423
--local_ip_for_outbound_sockets=127.5.249.2
--tserver_master_addrs=127.5.249.62:43179
--webserver_port=33479
--webserver_interface=127.5.249.2
--builtin_ntp_servers=127.5.249.20:46127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--num_tablets_to_copy_simultaneously=1 with env {}
W20260505 13:57:35.834334 6631 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260505 13:57:35.835044 6631 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260505 13:57:35.835587 6631 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260505 13:57:35.872951 6631 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260505 13:57:35.874008 6631 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.249.2
I20260505 13:57:35.913578 6631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.249.20:46127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.249.2:42423
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.249.2
--webserver_port=33479
--tserver_master_addrs=127.5.249.62:43179
--num_tablets_to_copy_simultaneously=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.249.2
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 05 May 2026 13:43:23 UTC on e7f111948823
build id 11759
TSAN enabled
I20260505 13:57:35.915210 6631 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260505 13:57:35.917218 6631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 13:57:35.932009 6638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:37.336503 6637 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ca0 after lost signal to thread 6631
W20260505 13:57:37.406418 6637 kernel_stack_watchdog.cc:198] Thread 6631 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260505 13:57:37.406960 6631 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.474s user 0.471s sys 0.981s
W20260505 13:57:37.407398 6631 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.474s user 0.471s sys 0.981s
W20260505 13:57:35.933830 6639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:37.408619 6640 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1473 milliseconds
W20260505 13:57:37.409152 6641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 13:57:37.409197 6631 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260505 13:57:37.413734 6631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260505 13:57:37.416122 6631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260505 13:57:37.417550 6631 hybrid_clock.cc:648] HybridClock initialized: now 1777989457417542 us; error 85 us; skew 500 ppm
I20260505 13:57:37.418510 6631 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 13:57:37.425398 6631 webserver.cc:492] Webserver started at http://127.5.249.2:33479/ using document root <none> and password file <none>
I20260505 13:57:37.426487 6631 fs_manager.cc:362] Metadata directory not provided
I20260505 13:57:37.426767 6631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 13:57:37.436101 6631 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20260505 13:57:37.441726 6649 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:37.443164 6631 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260505 13:57:37.443518 6631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/wal
uuid: "c5e39347ea38406fabb2ab795a53d363"
format_stamp: "Formatted at 2026-05-05 13:57:24 on dist-test-slave-fz8k"
I20260505 13:57:37.445710 6631 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 13:57:37.517511 6631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 13:57:37.519446 6631 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260505 13:57:37.519973 6631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 13:57:37.523056 6631 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 13:57:37.527681 6631 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 13:57:37.527980 6631 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:37.528328 6631 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 13:57:37.528502 6631 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:37.815032 6631 rpc_server.cc:307] RPC server started. Bound to: 127.5.249.2:42423
I20260505 13:57:37.815208 6761 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.249.2:42423 every 8 connection(s)
I20260505 13:57:37.820273 6631 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1777989435273980-6116-0/minicluster-data/ts-1/data/info.pb
I20260505 13:57:37.825623 6116 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu as pid 6631
I20260505 13:57:37.872519 6762 heartbeater.cc:344] Connected to a master server at 127.5.249.62:43179
I20260505 13:57:37.873145 6762 heartbeater.cc:461] Registering TS with master...
I20260505 13:57:37.874771 6762 heartbeater.cc:507] Master 127.5.249.62:43179 requested a full tablet report, sending...
I20260505 13:57:37.878239 6526 ts_manager.cc:194] Registered new tserver with Master: c5e39347ea38406fabb2ab795a53d363 (127.5.249.2:42423)
I20260505 13:57:37.879715 6768 ts_tablet_manager.cc:933] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363: Initiating tablet copy from peer 2e5321dafcdb4d33af8a8e1b636f0de7 (127.5.249.1:40903)
I20260505 13:57:37.881174 6526 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.5.249.2:48805
I20260505 13:57:37.886955 6768 tablet_copy_client.cc:323] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363: tablet copy: Beginning tablet copy session from remote peer at address 127.5.249.1:40903
I20260505 13:57:37.900867 6321 tablet_copy_service.cc:140] P 2e5321dafcdb4d33af8a8e1b636f0de7: Received BeginTabletCopySession request for tablet 31781fc9f40049f8a50ee8dbd4c879ec from peer c5e39347ea38406fabb2ab795a53d363 ({username='slave'} at 127.5.249.2:37629)
I20260505 13:57:37.901554 6321 tablet_copy_service.cc:161] P 2e5321dafcdb4d33af8a8e1b636f0de7: Beginning new tablet copy session on tablet 31781fc9f40049f8a50ee8dbd4c879ec from peer c5e39347ea38406fabb2ab795a53d363 at {username='slave'} at 127.5.249.2:37629: session id = c5e39347ea38406fabb2ab795a53d363-31781fc9f40049f8a50ee8dbd4c879ec
I20260505 13:57:37.909633 6321 tablet_copy_source_session.cc:215] T 31781fc9f40049f8a50ee8dbd4c879ec P 2e5321dafcdb4d33af8a8e1b636f0de7: Tablet Copy: opened 0 blocks and 1 log segments
I20260505 13:57:37.917707 6768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31781fc9f40049f8a50ee8dbd4c879ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 13:57:37.945372 6768 tablet_copy_client.cc:806] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363: tablet copy: Starting download of 0 data blocks...
I20260505 13:57:37.946285 6768 tablet_copy_client.cc:670] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363: tablet copy: Starting download of 1 WAL segments...
I20260505 13:57:37.970849 6768 tablet_copy_client.cc:538] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260505 13:57:37.980923 6768 tablet_bootstrap.cc:492] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363: Bootstrap starting.
I20260505 13:57:38.167641 6768 log.cc:826] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363: Log is configured to *not* fsync() on all Append() calls
I20260505 13:57:38.885754 6762 heartbeater.cc:499] Master 127.5.249.62:43179 was elected leader, sending a full tablet report...
I20260505 13:57:39.245908 6768 tablet_bootstrap.cc:492] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363: Bootstrap replayed 1/1 log segments. Stats: ops{read=214 overwritten=0 applied=214 ignored=0} inserts{seen=2638 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260505 13:57:39.247153 6768 tablet_bootstrap.cc:492] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363: Bootstrap complete.
I20260505 13:57:39.248226 6768 ts_tablet_manager.cc:1403] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363: Time spent bootstrapping tablet: real 1.268s user 1.204s sys 0.060s
I20260505 13:57:39.264292 6768 raft_consensus.cc:359] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:39.265040 6768 raft_consensus.cc:740] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: c5e39347ea38406fabb2ab795a53d363, State: Initialized, Role: NON_PARTICIPANT
I20260505 13:57:39.265818 6768 consensus_queue.cc:260] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 214, Last appended: 1.214, Last appended by leader: 214, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:39.271299 6768 ts_tablet_manager.cc:1434] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363: Time spent starting tablet: real 0.023s user 0.017s sys 0.004s
I20260505 13:57:39.273598 6321 tablet_copy_service.cc:342] P 2e5321dafcdb4d33af8a8e1b636f0de7: Request end of tablet copy session c5e39347ea38406fabb2ab795a53d363-31781fc9f40049f8a50ee8dbd4c879ec received from {username='slave'} at 127.5.249.2:37629
I20260505 13:57:39.274206 6321 tablet_copy_service.cc:434] P 2e5321dafcdb4d33af8a8e1b636f0de7: ending tablet copy session c5e39347ea38406fabb2ab795a53d363-31781fc9f40049f8a50ee8dbd4c879ec on tablet 31781fc9f40049f8a50ee8dbd4c879ec with peer c5e39347ea38406fabb2ab795a53d363
W20260505 13:57:39.278877 6768 ts_tablet_manager.cc:732] T 31781fc9f40049f8a50ee8dbd4c879ec P c5e39347ea38406fabb2ab795a53d363: Tablet Copy: Invalid argument: Leader has replica of tablet 31781fc9f40049f8a50ee8dbd4c879ec with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
I20260505 13:57:39.288108 6768 ts_tablet_manager.cc:933] T 4abb2e163f484a7993f6054876f9e0aa P c5e39347ea38406fabb2ab795a53d363: Initiating tablet copy from peer 2e5321dafcdb4d33af8a8e1b636f0de7 (127.5.249.1:40903)
I20260505 13:57:39.291070 6768 tablet_copy_client.cc:323] T 4abb2e163f484a7993f6054876f9e0aa P c5e39347ea38406fabb2ab795a53d363: tablet copy: Beginning tablet copy session from remote peer at address 127.5.249.1:40903
I20260505 13:57:39.292681 6321 tablet_copy_service.cc:140] P 2e5321dafcdb4d33af8a8e1b636f0de7: Received BeginTabletCopySession request for tablet 4abb2e163f484a7993f6054876f9e0aa from peer c5e39347ea38406fabb2ab795a53d363 ({username='slave'} at 127.5.249.2:37629)
I20260505 13:57:39.293223 6321 tablet_copy_service.cc:161] P 2e5321dafcdb4d33af8a8e1b636f0de7: Beginning new tablet copy session on tablet 4abb2e163f484a7993f6054876f9e0aa from peer c5e39347ea38406fabb2ab795a53d363 at {username='slave'} at 127.5.249.2:37629: session id = c5e39347ea38406fabb2ab795a53d363-4abb2e163f484a7993f6054876f9e0aa
I20260505 13:57:39.299782 6321 tablet_copy_source_session.cc:215] T 4abb2e163f484a7993f6054876f9e0aa P 2e5321dafcdb4d33af8a8e1b636f0de7: Tablet Copy: opened 0 blocks and 1 log segments
I20260505 13:57:39.302402 6768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4abb2e163f484a7993f6054876f9e0aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 13:57:39.311667 6768 tablet_copy_client.cc:806] T 4abb2e163f484a7993f6054876f9e0aa P c5e39347ea38406fabb2ab795a53d363: tablet copy: Starting download of 0 data blocks...
I20260505 13:57:39.312260 6768 tablet_copy_client.cc:670] T 4abb2e163f484a7993f6054876f9e0aa P c5e39347ea38406fabb2ab795a53d363: tablet copy: Starting download of 1 WAL segments...
I20260505 13:57:39.333031 6768 tablet_copy_client.cc:538] T 4abb2e163f484a7993f6054876f9e0aa P c5e39347ea38406fabb2ab795a53d363: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260505 13:57:39.339883 6768 tablet_bootstrap.cc:492] T 4abb2e163f484a7993f6054876f9e0aa P c5e39347ea38406fabb2ab795a53d363: Bootstrap starting.
I20260505 13:57:40.452679 6768 tablet_bootstrap.cc:492] T 4abb2e163f484a7993f6054876f9e0aa P c5e39347ea38406fabb2ab795a53d363: Bootstrap replayed 1/1 log segments. Stats: ops{read=214 overwritten=0 applied=214 ignored=0} inserts{seen=2705 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260505 13:57:40.453611 6768 tablet_bootstrap.cc:492] T 4abb2e163f484a7993f6054876f9e0aa P c5e39347ea38406fabb2ab795a53d363: Bootstrap complete.
I20260505 13:57:40.454217 6768 ts_tablet_manager.cc:1403] T 4abb2e163f484a7993f6054876f9e0aa P c5e39347ea38406fabb2ab795a53d363: Time spent bootstrapping tablet: real 1.114s user 1.082s sys 0.032s
I20260505 13:57:40.456557 6768 raft_consensus.cc:359] T 4abb2e163f484a7993f6054876f9e0aa P c5e39347ea38406fabb2ab795a53d363 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:40.456972 6768 raft_consensus.cc:740] T 4abb2e163f484a7993f6054876f9e0aa P c5e39347ea38406fabb2ab795a53d363 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: c5e39347ea38406fabb2ab795a53d363, State: Initialized, Role: NON_PARTICIPANT
I20260505 13:57:40.457526 6768 consensus_queue.cc:260] T 4abb2e163f484a7993f6054876f9e0aa P c5e39347ea38406fabb2ab795a53d363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 214, Last appended: 1.214, Last appended by leader: 214, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:40.461272 6768 ts_tablet_manager.cc:1434] T 4abb2e163f484a7993f6054876f9e0aa P c5e39347ea38406fabb2ab795a53d363: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260505 13:57:40.463661 6321 tablet_copy_service.cc:342] P 2e5321dafcdb4d33af8a8e1b636f0de7: Request end of tablet copy session c5e39347ea38406fabb2ab795a53d363-4abb2e163f484a7993f6054876f9e0aa received from {username='slave'} at 127.5.249.2:37629
I20260505 13:57:40.464386 6321 tablet_copy_service.cc:434] P 2e5321dafcdb4d33af8a8e1b636f0de7: ending tablet copy session c5e39347ea38406fabb2ab795a53d363-4abb2e163f484a7993f6054876f9e0aa on tablet 4abb2e163f484a7993f6054876f9e0aa with peer c5e39347ea38406fabb2ab795a53d363
I20260505 13:57:40.468348 6768 ts_tablet_manager.cc:933] T ce2dafafeb1444c0a0dea4f7f68b309f P c5e39347ea38406fabb2ab795a53d363: Initiating tablet copy from peer 2e5321dafcdb4d33af8a8e1b636f0de7 (127.5.249.1:40903)
I20260505 13:57:40.474416 6768 tablet_copy_client.cc:323] T ce2dafafeb1444c0a0dea4f7f68b309f P c5e39347ea38406fabb2ab795a53d363: tablet copy: Beginning tablet copy session from remote peer at address 127.5.249.1:40903
I20260505 13:57:40.476115 6321 tablet_copy_service.cc:140] P 2e5321dafcdb4d33af8a8e1b636f0de7: Received BeginTabletCopySession request for tablet ce2dafafeb1444c0a0dea4f7f68b309f from peer c5e39347ea38406fabb2ab795a53d363 ({username='slave'} at 127.5.249.2:37629)
I20260505 13:57:40.476660 6321 tablet_copy_service.cc:161] P 2e5321dafcdb4d33af8a8e1b636f0de7: Beginning new tablet copy session on tablet ce2dafafeb1444c0a0dea4f7f68b309f from peer c5e39347ea38406fabb2ab795a53d363 at {username='slave'} at 127.5.249.2:37629: session id = c5e39347ea38406fabb2ab795a53d363-ce2dafafeb1444c0a0dea4f7f68b309f
I20260505 13:57:40.485303 6321 tablet_copy_source_session.cc:215] T ce2dafafeb1444c0a0dea4f7f68b309f P 2e5321dafcdb4d33af8a8e1b636f0de7: Tablet Copy: opened 0 blocks and 1 log segments
I20260505 13:57:40.488102 6768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce2dafafeb1444c0a0dea4f7f68b309f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 13:57:40.504487 6768 tablet_copy_client.cc:806] T ce2dafafeb1444c0a0dea4f7f68b309f P c5e39347ea38406fabb2ab795a53d363: tablet copy: Starting download of 0 data blocks...
I20260505 13:57:40.505152 6768 tablet_copy_client.cc:670] T ce2dafafeb1444c0a0dea4f7f68b309f P c5e39347ea38406fabb2ab795a53d363: tablet copy: Starting download of 1 WAL segments...
I20260505 13:57:40.519243 6768 tablet_copy_client.cc:538] T ce2dafafeb1444c0a0dea4f7f68b309f P c5e39347ea38406fabb2ab795a53d363: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260505 13:57:40.526249 6768 tablet_bootstrap.cc:492] T ce2dafafeb1444c0a0dea4f7f68b309f P c5e39347ea38406fabb2ab795a53d363: Bootstrap starting.
I20260505 13:57:41.632707 6768 tablet_bootstrap.cc:492] T ce2dafafeb1444c0a0dea4f7f68b309f P c5e39347ea38406fabb2ab795a53d363: Bootstrap replayed 1/1 log segments. Stats: ops{read=214 overwritten=0 applied=214 ignored=0} inserts{seen=2667 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260505 13:57:41.633626 6768 tablet_bootstrap.cc:492] T ce2dafafeb1444c0a0dea4f7f68b309f P c5e39347ea38406fabb2ab795a53d363: Bootstrap complete.
I20260505 13:57:41.634266 6768 ts_tablet_manager.cc:1403] T ce2dafafeb1444c0a0dea4f7f68b309f P c5e39347ea38406fabb2ab795a53d363: Time spent bootstrapping tablet: real 1.108s user 1.080s sys 0.020s
I20260505 13:57:41.636518 6768 raft_consensus.cc:359] T ce2dafafeb1444c0a0dea4f7f68b309f P c5e39347ea38406fabb2ab795a53d363 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:41.636930 6768 raft_consensus.cc:740] T ce2dafafeb1444c0a0dea4f7f68b309f P c5e39347ea38406fabb2ab795a53d363 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: c5e39347ea38406fabb2ab795a53d363, State: Initialized, Role: NON_PARTICIPANT
I20260505 13:57:41.637478 6768 consensus_queue.cc:260] T ce2dafafeb1444c0a0dea4f7f68b309f P c5e39347ea38406fabb2ab795a53d363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 214, Last appended: 1.214, Last appended by leader: 214, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:41.640209 6768 ts_tablet_manager.cc:1434] T ce2dafafeb1444c0a0dea4f7f68b309f P c5e39347ea38406fabb2ab795a53d363: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260505 13:57:41.641925 6321 tablet_copy_service.cc:342] P 2e5321dafcdb4d33af8a8e1b636f0de7: Request end of tablet copy session c5e39347ea38406fabb2ab795a53d363-ce2dafafeb1444c0a0dea4f7f68b309f received from {username='slave'} at 127.5.249.2:37629
I20260505 13:57:41.642387 6321 tablet_copy_service.cc:434] P 2e5321dafcdb4d33af8a8e1b636f0de7: ending tablet copy session c5e39347ea38406fabb2ab795a53d363-ce2dafafeb1444c0a0dea4f7f68b309f on tablet ce2dafafeb1444c0a0dea4f7f68b309f with peer c5e39347ea38406fabb2ab795a53d363
W20260505 13:57:41.645767 6768 ts_tablet_manager.cc:732] T 4abb2e163f484a7993f6054876f9e0aa P c5e39347ea38406fabb2ab795a53d363: Tablet Copy: Invalid argument: Leader has replica of tablet 4abb2e163f484a7993f6054876f9e0aa with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
W20260505 13:57:41.651111 6768 ts_tablet_manager.cc:732] T ce2dafafeb1444c0a0dea4f7f68b309f P c5e39347ea38406fabb2ab795a53d363: Tablet Copy: Invalid argument: Leader has replica of tablet ce2dafafeb1444c0a0dea4f7f68b309f with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
I20260505 13:57:41.654531 6768 ts_tablet_manager.cc:933] T d82caf3e1b104c32a3c0c0df1b2e157d P c5e39347ea38406fabb2ab795a53d363: Initiating tablet copy from peer 2e5321dafcdb4d33af8a8e1b636f0de7 (127.5.249.1:40903)
I20260505 13:57:41.655700 6768 tablet_copy_client.cc:323] T d82caf3e1b104c32a3c0c0df1b2e157d P c5e39347ea38406fabb2ab795a53d363: tablet copy: Beginning tablet copy session from remote peer at address 127.5.249.1:40903
I20260505 13:57:41.657071 6321 tablet_copy_service.cc:140] P 2e5321dafcdb4d33af8a8e1b636f0de7: Received BeginTabletCopySession request for tablet d82caf3e1b104c32a3c0c0df1b2e157d from peer c5e39347ea38406fabb2ab795a53d363 ({username='slave'} at 127.5.249.2:37629)
I20260505 13:57:41.657490 6321 tablet_copy_service.cc:161] P 2e5321dafcdb4d33af8a8e1b636f0de7: Beginning new tablet copy session on tablet d82caf3e1b104c32a3c0c0df1b2e157d from peer c5e39347ea38406fabb2ab795a53d363 at {username='slave'} at 127.5.249.2:37629: session id = c5e39347ea38406fabb2ab795a53d363-d82caf3e1b104c32a3c0c0df1b2e157d
I20260505 13:57:41.662704 6321 tablet_copy_source_session.cc:215] T d82caf3e1b104c32a3c0c0df1b2e157d P 2e5321dafcdb4d33af8a8e1b636f0de7: Tablet Copy: opened 0 blocks and 1 log segments
I20260505 13:57:41.665091 6768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d82caf3e1b104c32a3c0c0df1b2e157d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 13:57:41.673744 6768 tablet_copy_client.cc:806] T d82caf3e1b104c32a3c0c0df1b2e157d P c5e39347ea38406fabb2ab795a53d363: tablet copy: Starting download of 0 data blocks...
I20260505 13:57:41.674189 6768 tablet_copy_client.cc:670] T d82caf3e1b104c32a3c0c0df1b2e157d P c5e39347ea38406fabb2ab795a53d363: tablet copy: Starting download of 1 WAL segments...
I20260505 13:57:41.687532 6768 tablet_copy_client.cc:538] T d82caf3e1b104c32a3c0c0df1b2e157d P c5e39347ea38406fabb2ab795a53d363: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260505 13:57:41.693715 6768 tablet_bootstrap.cc:492] T d82caf3e1b104c32a3c0c0df1b2e157d P c5e39347ea38406fabb2ab795a53d363: Bootstrap starting.
I20260505 13:57:42.732070 6768 tablet_bootstrap.cc:492] T d82caf3e1b104c32a3c0c0df1b2e157d P c5e39347ea38406fabb2ab795a53d363: Bootstrap replayed 1/1 log segments. Stats: ops{read=214 overwritten=0 applied=214 ignored=0} inserts{seen=2640 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260505 13:57:42.732945 6768 tablet_bootstrap.cc:492] T d82caf3e1b104c32a3c0c0df1b2e157d P c5e39347ea38406fabb2ab795a53d363: Bootstrap complete.
I20260505 13:57:42.733566 6768 ts_tablet_manager.cc:1403] T d82caf3e1b104c32a3c0c0df1b2e157d P c5e39347ea38406fabb2ab795a53d363: Time spent bootstrapping tablet: real 1.040s user 1.010s sys 0.024s
I20260505 13:57:42.735728 6768 raft_consensus.cc:359] T d82caf3e1b104c32a3c0c0df1b2e157d P c5e39347ea38406fabb2ab795a53d363 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:42.736220 6768 raft_consensus.cc:740] T d82caf3e1b104c32a3c0c0df1b2e157d P c5e39347ea38406fabb2ab795a53d363 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: c5e39347ea38406fabb2ab795a53d363, State: Initialized, Role: NON_PARTICIPANT
I20260505 13:57:42.736640 6768 consensus_queue.cc:260] T d82caf3e1b104c32a3c0c0df1b2e157d P c5e39347ea38406fabb2ab795a53d363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 214, Last appended: 1.214, Last appended by leader: 214, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e5321dafcdb4d33af8a8e1b636f0de7" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 40903 } }
I20260505 13:57:42.739166 6768 ts_tablet_manager.cc:1434] T d82caf3e1b104c32a3c0c0df1b2e157d P c5e39347ea38406fabb2ab795a53d363: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260505 13:57:42.740851 6321 tablet_copy_service.cc:342] P 2e5321dafcdb4d33af8a8e1b636f0de7: Request end of tablet copy session c5e39347ea38406fabb2ab795a53d363-d82caf3e1b104c32a3c0c0df1b2e157d received from {username='slave'} at 127.5.249.2:37629
I20260505 13:57:42.741297 6321 tablet_copy_service.cc:434] P 2e5321dafcdb4d33af8a8e1b636f0de7: ending tablet copy session c5e39347ea38406fabb2ab795a53d363-d82caf3e1b104c32a3c0c0df1b2e157d on tablet d82caf3e1b104c32a3c0c0df1b2e157d with peer c5e39347ea38406fabb2ab795a53d363
W20260505 13:57:42.744601 6768 ts_tablet_manager.cc:732] T d82caf3e1b104c32a3c0c0df1b2e157d P c5e39347ea38406fabb2ab795a53d363: Tablet Copy: Invalid argument: Leader has replica of tablet d82caf3e1b104c32a3c0c0df1b2e157d with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
I20260505 13:57:42.748585 6116 tablet_copy-itest.cc:1252] Number of Service unavailable responses: 1305
I20260505 13:57:42.748806 6116 tablet_copy-itest.cc:1253] Number of in progress responses: 980
I20260505 13:57:42.751129 6116 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu with pid 6217
I20260505 13:57:42.804126 6116 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu with pid 6631
I20260505 13:57:42.835597 6116 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu with pid 6491
2026-05-05T13:57:42Z chronyd exiting
[ OK ] TabletCopyITest.TestTabletCopyThrottling (27407 ms)
[ RUN ] TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate
2026-05-05T13:57:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-05-05T13:57:42Z Disabled control of system clock
I20260505 13:57:42.962852 6116 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
/tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.249.62:45327
--webserver_interface=127.5.249.62
--webserver_port=0
--builtin_ntp_servers=127.5.249.20:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.249.62:45327 with env {}
W20260505 13:57:43.307180 6786 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260505 13:57:43.307957 6786 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260505 13:57:43.308487 6786 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260505 13:57:43.344058 6786 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260505 13:57:43.344449 6786 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260505 13:57:43.344769 6786 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260505 13:57:43.345049 6786 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260505 13:57:43.389461 6786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.249.20:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.249.62:45327
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.249.62:45327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.249.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 05 May 2026 13:43:23 UTC on e7f111948823
build id 11759
TSAN enabled
I20260505 13:57:43.391144 6786 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260505 13:57:43.393136 6786 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 13:57:43.406685 6792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:43.408444 6793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:43.411660 6795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:44.618338 6794 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1205 milliseconds
I20260505 13:57:44.618496 6786 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260505 13:57:44.619946 6786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260505 13:57:44.623510 6786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260505 13:57:44.624954 6786 hybrid_clock.cc:648] HybridClock initialized: now 1777989464624911 us; error 63 us; skew 500 ppm
I20260505 13:57:44.625772 6786 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 13:57:44.633313 6786 webserver.cc:492] Webserver started at http://127.5.249.62:37909/ using document root <none> and password file <none>
I20260505 13:57:44.634331 6786 fs_manager.cc:362] Metadata directory not provided
I20260505 13:57:44.634527 6786 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 13:57:44.634981 6786 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 13:57:44.639716 6786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/data/instance:
uuid: "972d2d5e1f784c78bc1f3799a13bc74f"
format_stamp: "Formatted at 2026-05-05 13:57:44 on dist-test-slave-fz8k"
I20260505 13:57:44.640983 6786 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/wal/instance:
uuid: "972d2d5e1f784c78bc1f3799a13bc74f"
format_stamp: "Formatted at 2026-05-05 13:57:44 on dist-test-slave-fz8k"
I20260505 13:57:44.649750 6786 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260505 13:57:44.656091 6802 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:44.657400 6786 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260505 13:57:44.657712 6786 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/data,/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/wal
uuid: "972d2d5e1f784c78bc1f3799a13bc74f"
format_stamp: "Formatted at 2026-05-05 13:57:44 on dist-test-slave-fz8k"
I20260505 13:57:44.658027 6786 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 13:57:44.708071 6786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 13:57:44.709679 6786 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260505 13:57:44.710140 6786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 13:57:44.793238 6786 rpc_server.cc:307] RPC server started. Bound to: 127.5.249.62:45327
I20260505 13:57:44.793318 6853 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.249.62:45327 every 8 connection(s)
I20260505 13:57:44.796398 6786 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/data/info.pb
I20260505 13:57:44.803284 6854 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 13:57:44.805869 6116 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu as pid 6786
I20260505 13:57:44.806418 6116 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/master-0/wal/instance
I20260505 13:57:44.826959 6854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f: Bootstrap starting.
I20260505 13:57:44.833828 6854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f: Neither blocks nor log segments found. Creating new log.
I20260505 13:57:44.836381 6854 log.cc:826] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f: Log is configured to *not* fsync() on all Append() calls
I20260505 13:57:44.841475 6854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f: No bootstrap required, opened a new log
I20260505 13:57:44.861598 6854 raft_consensus.cc:359] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972d2d5e1f784c78bc1f3799a13bc74f" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 45327 } }
I20260505 13:57:44.862341 6854 raft_consensus.cc:385] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 13:57:44.862589 6854 raft_consensus.cc:740] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 972d2d5e1f784c78bc1f3799a13bc74f, State: Initialized, Role: FOLLOWER
I20260505 13:57:44.863384 6854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972d2d5e1f784c78bc1f3799a13bc74f" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 45327 } }
I20260505 13:57:44.863940 6854 raft_consensus.cc:399] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260505 13:57:44.864253 6854 raft_consensus.cc:493] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260505 13:57:44.864611 6854 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [term 0 FOLLOWER]: Advancing to term 1
I20260505 13:57:44.868897 6854 raft_consensus.cc:515] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972d2d5e1f784c78bc1f3799a13bc74f" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 45327 } }
I20260505 13:57:44.869877 6854 leader_election.cc:304] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 972d2d5e1f784c78bc1f3799a13bc74f; no voters:
I20260505 13:57:44.871798 6854 leader_election.cc:290] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260505 13:57:44.872710 6859 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [term 1 FOLLOWER]: Leader election won for term 1
I20260505 13:57:44.875172 6859 raft_consensus.cc:697] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [term 1 LEADER]: Becoming Leader. State: Replica: 972d2d5e1f784c78bc1f3799a13bc74f, State: Running, Role: LEADER
I20260505 13:57:44.875974 6859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972d2d5e1f784c78bc1f3799a13bc74f" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 45327 } }
I20260505 13:57:44.877112 6854 sys_catalog.cc:565] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [sys.catalog]: configured and running, proceeding with master startup.
I20260505 13:57:44.888391 6860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "972d2d5e1f784c78bc1f3799a13bc74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972d2d5e1f784c78bc1f3799a13bc74f" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 45327 } } }
I20260505 13:57:44.888379 6861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 972d2d5e1f784c78bc1f3799a13bc74f. Latest consensus state: current_term: 1 leader_uuid: "972d2d5e1f784c78bc1f3799a13bc74f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "972d2d5e1f784c78bc1f3799a13bc74f" member_type: VOTER last_known_addr { host: "127.5.249.62" port: 45327 } } }
I20260505 13:57:44.889050 6860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [sys.catalog]: This master's current role is: LEADER
I20260505 13:57:44.889351 6861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f [sys.catalog]: This master's current role is: LEADER
I20260505 13:57:44.892294 6867 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260505 13:57:44.905578 6867 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260505 13:57:44.923758 6867 catalog_manager.cc:1357] Generated new cluster ID: 2d160953f27f41de883f20af462e4f12
I20260505 13:57:44.924050 6867 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260505 13:57:44.964566 6867 catalog_manager.cc:1380] Generated new certificate authority record
I20260505 13:57:44.966852 6867 catalog_manager.cc:1514] Loading token signing keys...
I20260505 13:57:44.983628 6867 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 972d2d5e1f784c78bc1f3799a13bc74f: Generated new TSK 0
I20260505 13:57:44.984997 6867 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260505 13:57:45.004376 6116 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
/tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.249.1:0
--local_ip_for_outbound_sockets=127.5.249.1
--webserver_interface=127.5.249.1
--webserver_port=0
--tserver_master_addrs=127.5.249.62:45327
--builtin_ntp_servers=127.5.249.20:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
W20260505 13:57:45.349459 6878 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260505 13:57:45.350107 6878 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260505 13:57:45.350533 6878 flags.cc:432] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20260505 13:57:45.350752 6878 flags.cc:432] Enabled unsafe flag: --enable_flush_memrowset=false
W20260505 13:57:45.351087 6878 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260505 13:57:45.385002 6878 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260505 13:57:45.385990 6878 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.249.1
I20260505 13:57:45.426003 6878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.249.20:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.249.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.249.1
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.5.249.62:45327
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.249.1
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 05 May 2026 13:43:23 UTC on e7f111948823
build id 11759
TSAN enabled
I20260505 13:57:45.427615 6878 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260505 13:57:45.429553 6878 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 13:57:45.450620 6884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:45.454001 6885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:46.671904 6886 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20260505 13:57:46.673599 6887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 13:57:46.673715 6878 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260505 13:57:46.677196 6878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260505 13:57:46.680676 6878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260505 13:57:46.682192 6878 hybrid_clock.cc:648] HybridClock initialized: now 1777989466682132 us; error 49 us; skew 500 ppm
I20260505 13:57:46.683370 6878 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 13:57:46.691898 6878 webserver.cc:492] Webserver started at http://127.5.249.1:38525/ using document root <none> and password file <none>
I20260505 13:57:46.693297 6878 fs_manager.cc:362] Metadata directory not provided
I20260505 13:57:46.693615 6878 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 13:57:46.694199 6878 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 13:57:46.699259 6878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/data/instance:
uuid: "05d24c1cf32a4a18bfb58c541a620c2f"
format_stamp: "Formatted at 2026-05-05 13:57:46 on dist-test-slave-fz8k"
I20260505 13:57:46.700543 6878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/wal/instance:
uuid: "05d24c1cf32a4a18bfb58c541a620c2f"
format_stamp: "Formatted at 2026-05-05 13:57:46 on dist-test-slave-fz8k"
I20260505 13:57:46.708837 6878 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260505 13:57:46.715579 6895 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:46.716881 6878 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20260505 13:57:46.717199 6878 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/wal
uuid: "05d24c1cf32a4a18bfb58c541a620c2f"
format_stamp: "Formatted at 2026-05-05 13:57:46 on dist-test-slave-fz8k"
I20260505 13:57:46.717605 6878 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 13:57:46.782097 6878 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 13:57:46.783803 6878 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260505 13:57:46.784305 6878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 13:57:46.787206 6878 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 13:57:46.792316 6878 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 13:57:46.792603 6878 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:46.792896 6878 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 13:57:46.793092 6878 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:46.952692 6878 rpc_server.cc:307] RPC server started. Bound to: 127.5.249.1:39501
I20260505 13:57:46.952812 7007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.249.1:39501 every 8 connection(s)
I20260505 13:57:46.955662 6878 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/data/info.pb
I20260505 13:57:46.962926 6116 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu as pid 6878
I20260505 13:57:46.963558 6116 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-0/wal/instance
I20260505 13:57:46.971378 6116 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
/tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.249.2:0
--local_ip_for_outbound_sockets=127.5.249.2
--webserver_interface=127.5.249.2
--webserver_port=0
--tserver_master_addrs=127.5.249.62:45327
--builtin_ntp_servers=127.5.249.20:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
I20260505 13:57:46.980428 7008 heartbeater.cc:344] Connected to a master server at 127.5.249.62:45327
I20260505 13:57:46.980929 7008 heartbeater.cc:461] Registering TS with master...
I20260505 13:57:46.982447 7008 heartbeater.cc:507] Master 127.5.249.62:45327 requested a full tablet report, sending...
I20260505 13:57:46.985671 6819 ts_manager.cc:194] Registered new tserver with Master: 05d24c1cf32a4a18bfb58c541a620c2f (127.5.249.1:39501)
I20260505 13:57:46.988734 6819 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.5.249.1:37601
W20260505 13:57:47.305526 7012 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260505 13:57:47.306162 7012 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260505 13:57:47.306607 7012 flags.cc:432] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20260505 13:57:47.306854 7012 flags.cc:432] Enabled unsafe flag: --enable_flush_memrowset=false
W20260505 13:57:47.307193 7012 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260505 13:57:47.341379 7012 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260505 13:57:47.342386 7012 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.249.2
I20260505 13:57:47.379655 7012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.249.20:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.249.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.249.2
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.5.249.62:45327
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.249.2
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 05 May 2026 13:43:23 UTC on e7f111948823
build id 11759
TSAN enabled
I20260505 13:57:47.381325 7012 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260505 13:57:47.383165 7012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 13:57:47.397524 7018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 13:57:47.994668 7008 heartbeater.cc:499] Master 127.5.249.62:45327 was elected leader, sending a full tablet report...
W20260505 13:57:47.398089 7019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:48.786082 7021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:48.793648 7012 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.395s user 0.450s sys 0.944s
W20260505 13:57:48.794018 7012 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.396s user 0.450s sys 0.944s
I20260505 13:57:48.797801 7012 server_base.cc:1061] running on GCE node
I20260505 13:57:48.799986 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260505 13:57:48.804975 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260505 13:57:48.806897 7012 hybrid_clock.cc:648] HybridClock initialized: now 1777989468806794 us; error 119 us; skew 500 ppm
I20260505 13:57:48.807907 7012 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 13:57:48.819949 7012 webserver.cc:492] Webserver started at http://127.5.249.2:45543/ using document root <none> and password file <none>
I20260505 13:57:48.821278 7012 fs_manager.cc:362] Metadata directory not provided
I20260505 13:57:48.821591 7012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 13:57:48.822217 7012 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 13:57:48.828337 7012 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/data/instance:
uuid: "b11284b44f424719ba754e92c1af3e33"
format_stamp: "Formatted at 2026-05-05 13:57:48 on dist-test-slave-fz8k"
I20260505 13:57:48.829828 7012 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/wal/instance:
uuid: "b11284b44f424719ba754e92c1af3e33"
format_stamp: "Formatted at 2026-05-05 13:57:48 on dist-test-slave-fz8k"
I20260505 13:57:48.841210 7012 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.008s sys 0.003s
I20260505 13:57:48.850431 7028 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:48.852372 7012 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.004s
I20260505 13:57:48.852787 7012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/wal
uuid: "b11284b44f424719ba754e92c1af3e33"
format_stamp: "Formatted at 2026-05-05 13:57:48 on dist-test-slave-fz8k"
I20260505 13:57:48.853240 7012 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260505 13:57:48.938452 7012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 13:57:48.940510 7012 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260505 13:57:48.941035 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 13:57:48.944510 7012 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 13:57:48.950353 7012 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 13:57:48.950680 7012 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:48.951169 7012 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 13:57:48.951426 7012 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:49.151440 7012 rpc_server.cc:307] RPC server started. Bound to: 127.5.249.2:41615
I20260505 13:57:49.151541 7140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.249.2:41615 every 8 connection(s)
I20260505 13:57:49.155159 7012 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/data/info.pb
I20260505 13:57:49.164247 6116 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu as pid 7012
I20260505 13:57:49.164932 6116 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-1/wal/instance
I20260505 13:57:49.174347 6116 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
/tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.249.3:0
--local_ip_for_outbound_sockets=127.5.249.3
--webserver_interface=127.5.249.3
--webserver_port=0
--tserver_master_addrs=127.5.249.62:45327
--builtin_ntp_servers=127.5.249.20:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
I20260505 13:57:49.184007 7141 heartbeater.cc:344] Connected to a master server at 127.5.249.62:45327
I20260505 13:57:49.184693 7141 heartbeater.cc:461] Registering TS with master...
I20260505 13:57:49.186215 7141 heartbeater.cc:507] Master 127.5.249.62:45327 requested a full tablet report, sending...
I20260505 13:57:49.190220 6819 ts_manager.cc:194] Registered new tserver with Master: b11284b44f424719ba754e92c1af3e33 (127.5.249.2:41615)
I20260505 13:57:49.193257 6819 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.5.249.2:48455
W20260505 13:57:49.717869 7145 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260505 13:57:49.719311 7145 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260505 13:57:49.719806 7145 flags.cc:432] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20260505 13:57:49.720057 7145 flags.cc:432] Enabled unsafe flag: --enable_flush_memrowset=false
W20260505 13:57:49.720414 7145 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260505 13:57:49.775585 7145 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260505 13:57:49.777302 7145 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.249.3
I20260505 13:57:49.863277 7145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.249.20:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.249.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.249.3
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.5.249.62:45327
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.249.3
--log_dir=/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 0ba7f54e15c9f673bf3e350f30b8d5f1a4037b79
build type FASTDEBUG
built by None at 05 May 2026 13:43:23 UTC on e7f111948823
build id 11759
TSAN enabled
I20260505 13:57:49.866364 7145 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260505 13:57:49.869275 7145 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260505 13:57:49.894474 7151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260505 13:57:50.200592 7141 heartbeater.cc:499] Master 127.5.249.62:45327 was elected leader, sending a full tablet report...
W20260505 13:57:49.895350 7152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:51.883589 7145 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.989s user 0.662s sys 1.295s
W20260505 13:57:51.311272 7150 debug-util.cc:398] Leaking SignalData structure 0x7b0800028f60 after lost signal to thread 7145
W20260505 13:57:51.884958 7145 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.991s user 0.664s sys 1.295s
W20260505 13:57:51.890525 7154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260505 13:57:51.899616 7153 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2000 milliseconds
I20260505 13:57:51.899595 7145 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260505 13:57:51.902940 7145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260505 13:57:51.908128 7145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260505 13:57:51.911728 7145 hybrid_clock.cc:648] HybridClock initialized: now 1777989471911663 us; error 68 us; skew 500 ppm
I20260505 13:57:51.913415 7145 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260505 13:57:51.924283 7145 webserver.cc:492] Webserver started at http://127.5.249.3:39853/ using document root <none> and password file <none>
I20260505 13:57:51.925737 7145 fs_manager.cc:362] Metadata directory not provided
I20260505 13:57:51.926013 7145 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260505 13:57:51.926855 7145 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260505 13:57:51.935384 7145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/data/instance:
uuid: "7f7d3baeb7b54d7fa07d485e506e97d2"
format_stamp: "Formatted at 2026-05-05 13:57:51 on dist-test-slave-fz8k"
I20260505 13:57:51.937357 7145 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/wal/instance:
uuid: "7f7d3baeb7b54d7fa07d485e506e97d2"
format_stamp: "Formatted at 2026-05-05 13:57:51 on dist-test-slave-fz8k"
I20260505 13:57:51.950687 7145 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.012s sys 0.000s
I20260505 13:57:51.963549 7161 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:51.965072 7145 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260505 13:57:51.965559 7145 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/wal
uuid: "7f7d3baeb7b54d7fa07d485e506e97d2"
format_stamp: "Formatted at 2026-05-05 13:57:51 on dist-test-slave-fz8k"
I20260505 13:57:51.966128 7145 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260505 13:57:51.980958 6850 debug-util.cc:398] Leaking SignalData structure 0x7b080009f8c0 after lost signal to thread 6787
W20260505 13:57:51.982146 6850 debug-util.cc:398] Leaking SignalData structure 0x7b08000abf20 after lost signal to thread 6853
I20260505 13:57:52.051568 7145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260505 13:57:52.055881 7145 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260505 13:57:52.056591 7145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260505 13:57:52.061838 7145 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260505 13:57:52.068514 7145 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260505 13:57:52.068797 7145 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:52.069087 7145 ts_tablet_manager.cc:616] Registered 0 tablets
I20260505 13:57:52.069517 7145 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260505 13:57:52.376037 7145 rpc_server.cc:307] RPC server started. Bound to: 127.5.249.3:43967
I20260505 13:57:52.376350 7273 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.249.3:43967 every 8 connection(s)
I20260505 13:57:52.379899 7145 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/data/info.pb
I20260505 13:57:52.390323 6116 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu as pid 7145
I20260505 13:57:52.390998 6116 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0/minicluster-data/ts-2/wal/instance
I20260505 13:57:52.443432 7274 heartbeater.cc:344] Connected to a master server at 127.5.249.62:45327
I20260505 13:57:52.444226 7274 heartbeater.cc:461] Registering TS with master...
I20260505 13:57:52.446259 7274 heartbeater.cc:507] Master 127.5.249.62:45327 requested a full tablet report, sending...
I20260505 13:57:52.449748 6819 ts_manager.cc:194] Registered new tserver with Master: 7f7d3baeb7b54d7fa07d485e506e97d2 (127.5.249.3:43967)
I20260505 13:57:52.452430 6819 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.5.249.3:47469
I20260505 13:57:52.457585 6116 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260505 13:57:52.604560 6819 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39152:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260505 13:57:52.608922 6819 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260505 13:57:52.743053 7076 tablet_service.cc:1511] Processing CreateTablet for tablet 88ffad060bde428e92d12021c59d85f7 (DEFAULT_TABLE table=test-workload [id=6b2e3f14651542838bc0097991be7cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260505 13:57:52.747296 7076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88ffad060bde428e92d12021c59d85f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 13:57:52.746732 6943 tablet_service.cc:1511] Processing CreateTablet for tablet 88ffad060bde428e92d12021c59d85f7 (DEFAULT_TABLE table=test-workload [id=6b2e3f14651542838bc0097991be7cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260505 13:57:52.750959 6943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88ffad060bde428e92d12021c59d85f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 13:57:52.767536 7209 tablet_service.cc:1511] Processing CreateTablet for tablet 88ffad060bde428e92d12021c59d85f7 (DEFAULT_TABLE table=test-workload [id=6b2e3f14651542838bc0097991be7cbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260505 13:57:52.772811 7209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88ffad060bde428e92d12021c59d85f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260505 13:57:52.816594 7299 tablet_bootstrap.cc:492] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33: Bootstrap starting.
I20260505 13:57:52.820012 7298 tablet_bootstrap.cc:492] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f: Bootstrap starting.
I20260505 13:57:52.831800 7299 tablet_bootstrap.cc:654] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33: Neither blocks nor log segments found. Creating new log.
I20260505 13:57:52.836133 7299 log.cc:826] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33: Log is configured to *not* fsync() on all Append() calls
I20260505 13:57:52.836539 7298 tablet_bootstrap.cc:654] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f: Neither blocks nor log segments found. Creating new log.
I20260505 13:57:52.841646 7298 log.cc:826] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f: Log is configured to *not* fsync() on all Append() calls
I20260505 13:57:52.852121 7299 tablet_bootstrap.cc:492] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33: No bootstrap required, opened a new log
I20260505 13:57:52.853276 7299 ts_tablet_manager.cc:1403] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33: Time spent bootstrapping tablet: real 0.038s user 0.019s sys 0.013s
I20260505 13:57:52.872674 7302 tablet_bootstrap.cc:492] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2: Bootstrap starting.
I20260505 13:57:52.873189 7298 tablet_bootstrap.cc:492] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f: No bootstrap required, opened a new log
I20260505 13:57:52.874671 7298 ts_tablet_manager.cc:1403] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f: Time spent bootstrapping tablet: real 0.055s user 0.014s sys 0.027s
I20260505 13:57:52.890311 7302 tablet_bootstrap.cc:654] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2: Neither blocks nor log segments found. Creating new log.
I20260505 13:57:52.906912 7302 log.cc:826] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2: Log is configured to *not* fsync() on all Append() calls
I20260505 13:57:52.928148 7299 raft_consensus.cc:359] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 } } peers { permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 } } peers { permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 } }
I20260505 13:57:52.930047 7299 raft_consensus.cc:385] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 13:57:52.931042 7299 raft_consensus.cc:740] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b11284b44f424719ba754e92c1af3e33, State: Initialized, Role: FOLLOWER
I20260505 13:57:52.934976 7302 tablet_bootstrap.cc:492] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2: No bootstrap required, opened a new log
I20260505 13:57:52.934288 7299 consensus_queue.cc:260] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 } } peers { permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 } } peers { permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 } }
I20260505 13:57:52.934505 7298 raft_consensus.cc:359] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 } } peers { permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 } } peers { permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 } }
I20260505 13:57:52.937117 7298 raft_consensus.cc:385] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 13:57:52.938129 7302 ts_tablet_manager.cc:1403] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2: Time spent bootstrapping tablet: real 0.068s user 0.033s sys 0.011s
I20260505 13:57:52.938055 7298 raft_consensus.cc:740] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05d24c1cf32a4a18bfb58c541a620c2f, State: Initialized, Role: FOLLOWER
I20260505 13:57:52.939491 7298 consensus_queue.cc:260] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 } } peers { permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 } } peers { permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 } }
I20260505 13:57:52.972955 7299 ts_tablet_manager.cc:1434] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33: Time spent starting tablet: real 0.119s user 0.058s sys 0.027s
I20260505 13:57:52.983197 7298 ts_tablet_manager.cc:1434] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f: Time spent starting tablet: real 0.108s user 0.054s sys 0.030s
I20260505 13:57:53.024580 7302 raft_consensus.cc:359] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 } } peers { permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 } } peers { permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 } }
I20260505 13:57:53.026660 7302 raft_consensus.cc:385] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260505 13:57:53.027076 7302 raft_consensus.cc:740] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7d3baeb7b54d7fa07d485e506e97d2, State: Initialized, Role: FOLLOWER
I20260505 13:57:53.029016 7302 consensus_queue.cc:260] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 } } peers { permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 } } peers { permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 } }
I20260505 13:57:53.040961 7274 heartbeater.cc:499] Master 127.5.249.62:45327 was elected leader, sending a full tablet report...
I20260505 13:57:53.042311 7302 ts_tablet_manager.cc:1434] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2: Time spent starting tablet: real 0.104s user 0.066s sys 0.008s
I20260505 13:57:53.103484 7306 raft_consensus.cc:493] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260505 13:57:53.105600 7306 raft_consensus.cc:515] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 } } peers { permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 } } peers { permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 } }
I20260505 13:57:53.114164 7306 leader_election.cc:290] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05d24c1cf32a4a18bfb58c541a620c2f (127.5.249.1:39501), b11284b44f424719ba754e92c1af3e33 (127.5.249.2:41615)
I20260505 13:57:53.139974 6963 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88ffad060bde428e92d12021c59d85f7" candidate_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" is_pre_election: true
I20260505 13:57:53.141602 7096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88ffad060bde428e92d12021c59d85f7" candidate_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b11284b44f424719ba754e92c1af3e33" is_pre_election: true
I20260505 13:57:53.142428 6963 raft_consensus.cc:2468] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f7d3baeb7b54d7fa07d485e506e97d2 in term 0.
W20260505 13:57:53.142850 7275 tablet.cc:2404] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260505 13:57:53.143201 7096 raft_consensus.cc:2468] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f7d3baeb7b54d7fa07d485e506e97d2 in term 0.
I20260505 13:57:53.145174 7165 leader_election.cc:304] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d24c1cf32a4a18bfb58c541a620c2f, 7f7d3baeb7b54d7fa07d485e506e97d2; no voters:
I20260505 13:57:53.147024 7306 raft_consensus.cc:2804] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260505 13:57:53.147737 7306 raft_consensus.cc:493] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260505 13:57:53.148172 7306 raft_consensus.cc:3060] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 0 FOLLOWER]: Advancing to term 1
I20260505 13:57:53.161422 7306 raft_consensus.cc:515] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 } } peers { permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 } } peers { permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 } }
I20260505 13:57:53.167062 7306 leader_election.cc:290] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [CANDIDATE]: Term 1 election: Requested vote from peers 05d24c1cf32a4a18bfb58c541a620c2f (127.5.249.1:39501), b11284b44f424719ba754e92c1af3e33 (127.5.249.2:41615)
I20260505 13:57:53.169442 6963 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88ffad060bde428e92d12021c59d85f7" candidate_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05d24c1cf32a4a18bfb58c541a620c2f"
I20260505 13:57:53.170204 6963 raft_consensus.cc:3060] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [term 0 FOLLOWER]: Advancing to term 1
W20260505 13:57:53.171252 7142 tablet.cc:2404] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260505 13:57:53.171936 7096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88ffad060bde428e92d12021c59d85f7" candidate_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b11284b44f424719ba754e92c1af3e33"
I20260505 13:57:53.172642 7096 raft_consensus.cc:3060] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33 [term 0 FOLLOWER]: Advancing to term 1
I20260505 13:57:53.180770 6963 raft_consensus.cc:2468] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f7d3baeb7b54d7fa07d485e506e97d2 in term 1.
I20260505 13:57:53.182003 7096 raft_consensus.cc:2468] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f7d3baeb7b54d7fa07d485e506e97d2 in term 1.
I20260505 13:57:53.182180 7165 leader_election.cc:304] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d24c1cf32a4a18bfb58c541a620c2f, 7f7d3baeb7b54d7fa07d485e506e97d2; no voters:
I20260505 13:57:53.183727 7306 raft_consensus.cc:2804] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260505 13:57:53.186565 7306 raft_consensus.cc:697] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 1 LEADER]: Becoming Leader. State: Replica: 7f7d3baeb7b54d7fa07d485e506e97d2, State: Running, Role: LEADER
I20260505 13:57:53.189075 7306 consensus_queue.cc:237] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 } } peers { permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 } } peers { permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 } }
I20260505 13:57:53.209990 6817 catalog_manager.cc:5671] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f7d3baeb7b54d7fa07d485e506e97d2 (127.5.249.3). New cstate: current_term: 1 leader_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 } health_report { overall_health: HEALTHY } } }
W20260505 13:57:53.224325 7009 tablet.cc:2404] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260505 13:57:53.367542 6963 raft_consensus.cc:1275] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [term 1 FOLLOWER]: Refusing update from remote peer 7f7d3baeb7b54d7fa07d485e506e97d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260505 13:57:53.367650 7096 raft_consensus.cc:1275] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33 [term 1 FOLLOWER]: Refusing update from remote peer 7f7d3baeb7b54d7fa07d485e506e97d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260505 13:57:53.369671 7309 consensus_queue.cc:1048] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260505 13:57:53.370584 7306 consensus_queue.cc:1048] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260505 13:57:53.454428 7319 mvcc.cc:204] Tried to move back new op lower bound from 7282644882893631488 to 7282644882199044096. Current Snapshot: MvccSnapshot[applied={T|T < 7282644882893631488}]
I20260505 13:57:53.516811 6963 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "88ffad060bde428e92d12021c59d85f7"
dest_uuid: "05d24c1cf32a4a18bfb58c541a620c2f"
from {username='slave'} at 127.0.0.1:47522
I20260505 13:57:53.517580 6963 raft_consensus.cc:493] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260505 13:57:53.518102 6963 raft_consensus.cc:3060] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [term 1 FOLLOWER]: Advancing to term 2
I20260505 13:57:53.526563 6963 raft_consensus.cc:515] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 } } peers { permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 } } peers { permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 } }
I20260505 13:57:53.530020 6963 leader_election.cc:290] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [CANDIDATE]: Term 2 election: Requested vote from peers b11284b44f424719ba754e92c1af3e33 (127.5.249.2:41615), 7f7d3baeb7b54d7fa07d485e506e97d2 (127.5.249.3:43967)
I20260505 13:57:53.550686 7096 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88ffad060bde428e92d12021c59d85f7" candidate_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "b11284b44f424719ba754e92c1af3e33"
I20260505 13:57:53.551400 7096 raft_consensus.cc:3060] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33 [term 1 FOLLOWER]: Advancing to term 2
I20260505 13:57:53.560660 7096 raft_consensus.cc:2468] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05d24c1cf32a4a18bfb58c541a620c2f in term 2.
I20260505 13:57:53.562511 6899 leader_election.cc:304] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05d24c1cf32a4a18bfb58c541a620c2f, b11284b44f424719ba754e92c1af3e33; no voters:
I20260505 13:57:53.563874 7305 raft_consensus.cc:2804] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [term 2 FOLLOWER]: Leader election won for term 2
I20260505 13:57:53.570998 7229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88ffad060bde428e92d12021c59d85f7" candidate_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2"
I20260505 13:57:53.571805 7229 raft_consensus.cc:3055] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 1 LEADER]: Stepping down as leader of term 1
I20260505 13:57:53.572124 7229 raft_consensus.cc:740] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f7d3baeb7b54d7fa07d485e506e97d2, State: Running, Role: LEADER
I20260505 13:57:53.573077 7229 consensus_queue.cc:260] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 } } peers { permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 } } peers { permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 } }
I20260505 13:57:53.574540 7229 raft_consensus.cc:3060] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 1 FOLLOWER]: Advancing to term 2
I20260505 13:57:53.576638 7305 raft_consensus.cc:697] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [term 2 LEADER]: Becoming Leader. State: Replica: 05d24c1cf32a4a18bfb58c541a620c2f, State: Running, Role: LEADER
I20260505 13:57:53.577770 7305 consensus_queue.cc:237] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 } } peers { permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 } } peers { permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 } }
I20260505 13:57:53.586890 7229 raft_consensus.cc:2468] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05d24c1cf32a4a18bfb58c541a620c2f in term 2.
I20260505 13:57:53.589717 6818 catalog_manager.cc:5671] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f reported cstate change: term changed from 1 to 2, leader changed from 7f7d3baeb7b54d7fa07d485e506e97d2 (127.5.249.3) to 05d24c1cf32a4a18bfb58c541a620c2f (127.5.249.1). New cstate: current_term: 2 leader_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05d24c1cf32a4a18bfb58c541a620c2f" member_type: VOTER last_known_addr { host: "127.5.249.1" port: 39501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 } health_report { overall_health: UNKNOWN } } }
W20260505 13:57:53.631541 7189 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58780: Illegal state: replica 7f7d3baeb7b54d7fa07d485e506e97d2 is not leader of this config: current role FOLLOWER
W20260505 13:57:53.651280 7189 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58780: Illegal state: replica 7f7d3baeb7b54d7fa07d485e506e97d2 is not leader of this config: current role FOLLOWER
I20260505 13:57:53.703362 7229 raft_consensus.cc:1275] T 88ffad060bde428e92d12021c59d85f7 P 7f7d3baeb7b54d7fa07d485e506e97d2 [term 2 FOLLOWER]: Refusing update from remote peer 05d24c1cf32a4a18bfb58c541a620c2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260505 13:57:53.705731 7333 consensus_queue.cc:1048] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f7d3baeb7b54d7fa07d485e506e97d2" member_type: VOTER last_known_addr { host: "127.5.249.3" port: 43967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260505 13:57:53.712182 7096 raft_consensus.cc:1275] T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33 [term 2 FOLLOWER]: Refusing update from remote peer 05d24c1cf32a4a18bfb58c541a620c2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260505 13:57:53.715358 7305 consensus_queue.cc:1048] T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b11284b44f424719ba754e92c1af3e33" member_type: VOTER last_known_addr { host: "127.5.249.2" port: 41615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260505 13:57:54.010519 7320 mvcc.cc:204] Tried to move back new op lower bound from 7282644885395472384 to 7282644883787964416. Current Snapshot: MvccSnapshot[applied={T|T < 7282644885395472384 or (T in {7282644885395472384})}]
W20260505 13:58:02.702742 7004 debug-util.cc:398] Leaking SignalData structure 0x7b08000b6e40 after lost signal to thread 6879
W20260505 13:58:02.704288 7004 debug-util.cc:398] Leaking SignalData structure 0x7b080005bbe0 after lost signal to thread 7007
W20260505 13:58:08.035732 6899 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.249.3:43967, user_credentials={real_user=slave}} blocked reactor thread for 52733.8us
W20260505 13:58:08.298185 7004 debug-util.cc:398] Leaking SignalData structure 0x7b08000d2840 after lost signal to thread 6879
W20260505 13:58:08.300557 7004 debug-util.cc:398] Leaking SignalData structure 0x7b08000b30e0 after lost signal to thread 7007
W20260505 13:58:09.434747 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 59449.1us
W20260505 13:58:09.631239 6899 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.249.2:41615, user_credentials={real_user=slave}} blocked reactor thread for 93624.6us
W20260505 13:58:17.909123 7322 log.cc:927] Time spent T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33: Append to log took a long time: real 0.073s user 0.008s sys 0.004s
W20260505 13:58:18.016292 7321 log.cc:927] Time spent T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f: Append to log took a long time: real 0.055s user 0.005s sys 0.001s
W20260505 13:58:21.499971 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 45756.8us
W20260505 13:58:22.494977 7331 meta_cache.cc:1261] Time spent looking up entry by key: real 0.056s user 0.001s sys 0.000s
W20260505 13:58:27.878901 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 77960.4us
I20260505 13:58:31.521814 7449 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=856, attempt_no=0}) took 1091 ms. Trace:
I20260505 13:58:31.524195 7449 rpcz_store.cc:276] 0505 13:58:30.399961 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:30.400870 (+ 909us) service_pool.cc:224] Handling call
0505 13:58:31.490389 (+1089519us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:31.121018 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:31.128044 (+ 7026us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:31.128130 (+ 86us) write_op.cc:435] Acquired schema lock
0505 13:58:31.128187 (+ 57us) tablet.cc:662] Decoding operations
0505 13:58:31.134497 (+ 6310us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:31.134558 (+ 61us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:31.134583 (+ 25us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:31.245120 (+110537us) lock_manager.cc:598] Waited 107360us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:31.245208 (+ 88us) tablet.cc:701] Row locks acquired
0505 13:58:31.245223 (+ 15us) write_op.cc:260] PREPARE: finished
0505 13:58:31.245565 (+ 342us) write_op.cc:270] Start()
0505 13:58:31.245672 (+ 107us) write_op.cc:276] Timestamp: P: 1777989511245360 usec, L: 0
0505 13:58:31.245700 (+ 28us) op_driver.cc:348] REPLICATION: starting
0505 13:58:31.246317 (+ 617us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:31.277242 (+ 30925us) op_driver.cc:464] REPLICATION: finished
0505 13:58:31.298210 (+ 20968us) write_op.cc:301] APPLY: starting
0505 13:58:31.298609 (+ 399us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:31.301223 (+ 2614us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:31.301333 (+ 110us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:31.409033 (+107700us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:31.414751 (+ 5718us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:31.414909 (+ 158us) write_op.cc:312] APPLY: finished
0505 13:58:31.419185 (+ 4276us) log.cc:844] Serialized 320 byte log entry
0505 13:58:31.421669 (+ 2484us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:31.488884 (+ 67215us) write_op.cc:454] Released schema lock
0505 13:58:31.489292 (+ 408us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":20352,"num_ops":50,"prepare.queue_time_us":698240,"prepare.run_cpu_time_us":10186,"prepare.run_wall_time_us":125793,"raft.queue_time_us":3201,"raft.run_cpu_time_us":2357,"raft.run_wall_time_us":2356,"replication_time_us":30974,"row_lock_wait_count":1,"row_lock_wait_us":107360}]]}
I20260505 13:58:31.752969 7452 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=858, attempt_no=0}) took 1003 ms. Trace:
I20260505 13:58:31.758881 7452 rpcz_store.cc:276] 0505 13:58:30.747011 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:30.747697 (+ 686us) service_pool.cc:224] Handling call
0505 13:58:31.750054 (+1002357us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:31.436468 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:31.437179 (+ 711us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:31.437215 (+ 36us) write_op.cc:435] Acquired schema lock
0505 13:58:31.437239 (+ 24us) tablet.cc:662] Decoding operations
0505 13:58:31.456724 (+ 19485us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:31.456792 (+ 68us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:31.456818 (+ 26us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:31.572632 (+115814us) lock_manager.cc:598] Waited 113296us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:31.572938 (+ 306us) tablet.cc:701] Row locks acquired
0505 13:58:31.572984 (+ 46us) write_op.cc:260] PREPARE: finished
0505 13:58:31.573601 (+ 617us) write_op.cc:270] Start()
0505 13:58:31.574025 (+ 424us) write_op.cc:276] Timestamp: P: 1777989511573468 usec, L: 0
0505 13:58:31.574068 (+ 43us) op_driver.cc:348] REPLICATION: starting
0505 13:58:31.576217 (+ 2149us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:31.615352 (+ 39135us) op_driver.cc:464] REPLICATION: finished
0505 13:58:31.616805 (+ 1453us) write_op.cc:301] APPLY: starting
0505 13:58:31.617473 (+ 668us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:31.629875 (+ 12402us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:31.630080 (+ 205us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:31.713309 (+ 83229us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:31.717773 (+ 4464us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:31.717962 (+ 189us) write_op.cc:312] APPLY: finished
0505 13:58:31.724802 (+ 6840us) log.cc:844] Serialized 320 byte log entry
0505 13:58:31.726890 (+ 2088us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:31.747798 (+ 20908us) write_op.cc:454] Released schema lock
0505 13:58:31.748284 (+ 486us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":734,"num_ops":50,"prepare.queue_time_us":680382,"prepare.run_cpu_time_us":12779,"prepare.run_wall_time_us":154157,"raft.queue_time_us":3002,"raft.run_cpu_time_us":6819,"raft.run_wall_time_us":50334,"replication_time_us":40550,"row_lock_wait_count":1,"row_lock_wait_us":113296}]]}
I20260505 13:58:32.088922 7452 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=861, attempt_no=0}) took 1031 ms. Trace:
I20260505 13:58:32.093649 7452 rpcz_store.cc:276] 0505 13:58:31.057917 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:31.058337 (+ 420us) service_pool.cc:224] Handling call
0505 13:58:32.088666 (+1030329us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:31.866873 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:31.868360 (+ 1487us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:31.868401 (+ 41us) write_op.cc:435] Acquired schema lock
0505 13:58:31.868414 (+ 13us) tablet.cc:662] Decoding operations
0505 13:58:31.875933 (+ 7519us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:31.876116 (+ 183us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:31.876291 (+ 175us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:31.965900 (+ 89609us) tablet.cc:701] Row locks acquired
0505 13:58:31.965931 (+ 31us) write_op.cc:260] PREPARE: finished
0505 13:58:31.966255 (+ 324us) write_op.cc:270] Start()
0505 13:58:31.966389 (+ 134us) write_op.cc:276] Timestamp: P: 1777989511966119 usec, L: 0
0505 13:58:31.966406 (+ 17us) op_driver.cc:348] REPLICATION: starting
0505 13:58:31.992819 (+ 26413us) spinlock_profiling.cc:243] Waited 5.96 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2fc900 00007f45cb385f22 00007f45cb37b347 00007f45cb37b184 00007f45cb7ed9f1 00007f45cb7ed1c8 00007f45cb7f1e61 00007f45cb7f1e19 00007f45cb7f1da9 00007f45cb7f1d71
0505 13:58:31.994720 (+ 1901us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:32.026168 (+ 31448us) op_driver.cc:464] REPLICATION: finished
0505 13:58:32.028044 (+ 1876us) write_op.cc:301] APPLY: starting
0505 13:58:32.028398 (+ 354us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:32.031668 (+ 3270us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:32.031727 (+ 59us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:32.073012 (+ 41285us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:32.074186 (+ 1174us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:32.074300 (+ 114us) write_op.cc:312] APPLY: finished
0505 13:58:32.076531 (+ 2231us) log.cc:844] Serialized 320 byte log entry
0505 13:58:32.077133 (+ 602us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:32.079842 (+ 2709us) write_op.cc:454] Released schema lock
0505 13:58:32.087169 (+ 7327us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1205,"num_ops":50,"prepare.queue_time_us":792099,"prepare.run_cpu_time_us":28978,"prepare.run_wall_time_us":133854,"raft.queue_time_us":829,"raft.run_cpu_time_us":7879,"raft.run_wall_time_us":21800,"replication_time_us":59410,"row_lock_wait_count":1,"row_lock_wait_us":83664,"spinlock_wait_cycles":13988480}]]}
I20260505 13:58:32.389349 7452 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=863, attempt_no=0}) took 1069 ms. Trace:
I20260505 13:58:32.390605 7452 rpcz_store.cc:276] 0505 13:58:31.320152 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:31.320404 (+ 252us) service_pool.cc:224] Handling call
0505 13:58:32.389213 (+1068809us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:32.091441 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:32.095683 (+ 4242us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:32.095768 (+ 85us) write_op.cc:435] Acquired schema lock
0505 13:58:32.095784 (+ 16us) tablet.cc:662] Decoding operations
0505 13:58:32.110643 (+ 14859us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:32.110802 (+ 159us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:32.110857 (+ 55us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:32.208719 (+ 97862us) tablet.cc:701] Row locks acquired
0505 13:58:32.208741 (+ 22us) write_op.cc:260] PREPARE: finished
0505 13:58:32.208917 (+ 176us) write_op.cc:270] Start()
0505 13:58:32.209008 (+ 91us) write_op.cc:276] Timestamp: P: 1777989512208759 usec, L: 0
0505 13:58:32.209025 (+ 17us) op_driver.cc:348] REPLICATION: starting
0505 13:58:32.209758 (+ 733us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:32.266061 (+ 56303us) op_driver.cc:464] REPLICATION: finished
0505 13:58:32.269561 (+ 3500us) write_op.cc:301] APPLY: starting
0505 13:58:32.269942 (+ 381us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:32.273357 (+ 3415us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:32.273828 (+ 471us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:32.355564 (+ 81736us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:32.359467 (+ 3903us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:32.359786 (+ 319us) write_op.cc:312] APPLY: finished
0505 13:58:32.363339 (+ 3553us) log.cc:844] Serialized 320 byte log entry
0505 13:58:32.364116 (+ 777us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:32.364816 (+ 700us) write_op.cc:454] Released schema lock
0505 13:58:32.365291 (+ 475us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2196,"num_ops":50,"prepare.queue_time_us":765236,"prepare.run_cpu_time_us":20838,"prepare.run_wall_time_us":118841,"raft.queue_time_us":11219,"raft.run_cpu_time_us":6715,"raft.run_wall_time_us":76044,"replication_time_us":56759,"row_lock_wait_count":1,"row_lock_wait_us":89654}]]}
W20260505 13:58:32.584878 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 62442us
W20260505 13:58:36.158941 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 50071.4us
W20260505 13:58:36.723254 7338 meta_cache.cc:1261] Time spent looking up entry by key: real 0.058s user 0.010s sys 0.000s
W20260505 13:58:37.182569 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 53381.5us
I20260505 13:58:41.616039 7487 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=943, attempt_no=0}) took 1012 ms. Trace:
I20260505 13:58:41.651250 7487 rpcz_store.cc:276] 0505 13:58:40.604027 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:40.604815 (+ 788us) service_pool.cc:224] Handling call
0505 13:58:41.615310 (+1010495us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:41.132390 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:41.133258 (+ 868us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:41.133296 (+ 38us) write_op.cc:435] Acquired schema lock
0505 13:58:41.133310 (+ 14us) tablet.cc:662] Decoding operations
0505 13:58:41.150498 (+ 17188us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:41.150791 (+ 293us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:41.150933 (+ 142us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:41.297632 (+146699us) lock_manager.cc:598] Waited 133983us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:41.297991 (+ 359us) tablet.cc:701] Row locks acquired
0505 13:58:41.298035 (+ 44us) write_op.cc:260] PREPARE: finished
0505 13:58:41.299251 (+ 1216us) write_op.cc:270] Start()
0505 13:58:41.299799 (+ 548us) write_op.cc:276] Timestamp: P: 1777989521298832 usec, L: 0
0505 13:58:41.299850 (+ 51us) op_driver.cc:348] REPLICATION: starting
0505 13:58:41.302231 (+ 2381us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:41.415072 (+112841us) op_driver.cc:464] REPLICATION: finished
0505 13:58:41.416153 (+ 1081us) write_op.cc:301] APPLY: starting
0505 13:58:41.417089 (+ 936us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:41.419680 (+ 2591us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:41.419873 (+ 193us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:41.483999 (+ 64126us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:41.506995 (+ 22996us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:41.507223 (+ 228us) write_op.cc:312] APPLY: finished
0505 13:58:41.512222 (+ 4999us) log.cc:844] Serialized 320 byte log entry
0505 13:58:41.517501 (+ 5279us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:41.613869 (+ 96368us) write_op.cc:454] Released schema lock
0505 13:58:41.614264 (+ 395us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":631,"num_ops":50,"prepare.queue_time_us":521656,"prepare.run_cpu_time_us":19643,"prepare.run_wall_time_us":190683,"raft.queue_time_us":10344,"raft.run_cpu_time_us":7712,"raft.run_wall_time_us":45962,"replication_time_us":114693,"row_lock_wait_count":1,"row_lock_wait_us":133983,"spinlock_wait_cycles":798080}]]}
I20260505 13:58:41.678467 7487 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=944, attempt_no=0}) took 1047 ms. Trace:
I20260505 13:58:41.679217 7487 rpcz_store.cc:276] 0505 13:58:40.631498 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:40.632609 (+ 1111us) service_pool.cc:224] Handling call
0505 13:58:41.678366 (+1045757us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:41.324046 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:41.327860 (+ 3814us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:41.328209 (+ 349us) write_op.cc:435] Acquired schema lock
0505 13:58:41.328351 (+ 142us) tablet.cc:662] Decoding operations
0505 13:58:41.353952 (+ 25601us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:41.354017 (+ 65us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:41.354062 (+ 45us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:41.518167 (+164105us) lock_manager.cc:598] Waited 161196us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:41.518421 (+ 254us) tablet.cc:701] Row locks acquired
0505 13:58:41.518456 (+ 35us) write_op.cc:260] PREPARE: finished
0505 13:58:41.519192 (+ 736us) write_op.cc:270] Start()
0505 13:58:41.519705 (+ 513us) write_op.cc:276] Timestamp: P: 1777989521518863 usec, L: 0
0505 13:58:41.519750 (+ 45us) op_driver.cc:348] REPLICATION: starting
0505 13:58:41.545502 (+ 25752us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:41.619825 (+ 74323us) op_driver.cc:464] REPLICATION: finished
0505 13:58:41.657088 (+ 37263us) write_op.cc:301] APPLY: starting
0505 13:58:41.657256 (+ 168us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:41.658146 (+ 890us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:41.658185 (+ 39us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:41.674033 (+ 15848us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:41.675503 (+ 1470us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:41.675639 (+ 136us) write_op.cc:312] APPLY: finished
0505 13:58:41.677118 (+ 1479us) log.cc:844] Serialized 320 byte log entry
0505 13:58:41.677477 (+ 359us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:41.677762 (+ 285us) write_op.cc:454] Released schema lock
0505 13:58:41.677840 (+ 78us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":36996,"num_ops":50,"prepare.queue_time_us":683219,"prepare.run_cpu_time_us":22608,"prepare.run_wall_time_us":239274,"raft.queue_time_us":12607,"raft.run_cpu_time_us":8532,"raft.run_wall_time_us":55147,"replication_time_us":99947,"row_lock_wait_count":1,"row_lock_wait_us":161196,"thread_start_us":40068,"threads_started":1}]]}
I20260505 13:58:42.809047 7487 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=952, attempt_no=0}) took 1101 ms. Trace:
I20260505 13:58:42.840209 7487 rpcz_store.cc:276] 0505 13:58:41.706879 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:41.707100 (+ 221us) service_pool.cc:224] Handling call
0505 13:58:42.807667 (+1100567us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:42.455118 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:42.489541 (+ 34423us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:42.489601 (+ 60us) write_op.cc:435] Acquired schema lock
0505 13:58:42.489728 (+ 127us) tablet.cc:662] Decoding operations
0505 13:58:42.498481 (+ 8753us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:42.498546 (+ 65us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:42.498569 (+ 23us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:42.625163 (+126594us) lock_manager.cc:598] Waited 122423us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:42.625257 (+ 94us) tablet.cc:701] Row locks acquired
0505 13:58:42.625270 (+ 13us) write_op.cc:260] PREPARE: finished
0505 13:58:42.626229 (+ 959us) write_op.cc:270] Start()
0505 13:58:42.626455 (+ 226us) write_op.cc:276] Timestamp: P: 1777989522625612 usec, L: 0
0505 13:58:42.626478 (+ 23us) op_driver.cc:348] REPLICATION: starting
0505 13:58:42.629138 (+ 2660us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:42.718840 (+ 89702us) op_driver.cc:464] REPLICATION: finished
0505 13:58:42.720008 (+ 1168us) write_op.cc:301] APPLY: starting
0505 13:58:42.720409 (+ 401us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:42.722902 (+ 2493us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:42.722956 (+ 54us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:42.769003 (+ 46047us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:42.769379 (+ 376us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:42.769421 (+ 42us) write_op.cc:312] APPLY: finished
0505 13:58:42.770081 (+ 660us) log.cc:844] Serialized 320 byte log entry
0505 13:58:42.770385 (+ 304us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:42.770532 (+ 147us) write_op.cc:454] Released schema lock
0505 13:58:42.770593 (+ 61us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":374,"mutex_wait_us":27421,"num_ops":50,"prepare.queue_time_us":745502,"prepare.run_cpu_time_us":20163,"prepare.run_wall_time_us":216368,"raft.queue_time_us":38771,"raft.run_cpu_time_us":7149,"raft.run_wall_time_us":43973,"replication_time_us":92166,"row_lock_wait_count":1,"row_lock_wait_us":122423,"spinlock_wait_cycles":968960}]]}
I20260505 13:58:42.928722 7494 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=953, attempt_no=0}) took 1145 ms. Trace:
I20260505 13:58:42.930371 7494 rpcz_store.cc:276] 0505 13:58:41.782398 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:41.782577 (+ 179us) service_pool.cc:224] Handling call
0505 13:58:42.928015 (+1145438us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:42.671847 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:42.677504 (+ 5657us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:42.677692 (+ 188us) write_op.cc:435] Acquired schema lock
0505 13:58:42.677843 (+ 151us) tablet.cc:662] Decoding operations
0505 13:58:42.683243 (+ 5400us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:42.683321 (+ 78us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:42.730881 (+ 47560us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:42.770563 (+ 39682us) tablet.cc:701] Row locks acquired
0505 13:58:42.770580 (+ 17us) write_op.cc:260] PREPARE: finished
0505 13:58:42.770805 (+ 225us) write_op.cc:270] Start()
0505 13:58:42.770907 (+ 102us) write_op.cc:276] Timestamp: P: 1777989522770748 usec, L: 0
0505 13:58:42.770950 (+ 43us) op_driver.cc:348] REPLICATION: starting
0505 13:58:42.773895 (+ 2945us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:42.816828 (+ 42933us) op_driver.cc:464] REPLICATION: finished
0505 13:58:42.864296 (+ 47468us) write_op.cc:301] APPLY: starting
0505 13:58:42.864435 (+ 139us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:42.865504 (+ 1069us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:42.865820 (+ 316us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:42.905841 (+ 40021us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:42.906788 (+ 947us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:42.906852 (+ 64us) write_op.cc:312] APPLY: finished
0505 13:58:42.908431 (+ 1579us) log.cc:844] Serialized 320 byte log entry
0505 13:58:42.908876 (+ 445us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:42.927459 (+ 18583us) write_op.cc:454] Released schema lock
0505 13:58:42.927584 (+ 125us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":38218,"mutex_wait_us":53,"num_ops":50,"prepare.queue_time_us":885772,"prepare.run_cpu_time_us":20128,"prepare.run_wall_time_us":110662,"raft.queue_time_us":772,"raft.run_cpu_time_us":6495,"raft.run_wall_time_us":16249,"replication_time_us":45478,"row_lock_wait_count":1,"row_lock_wait_us":31962,"spinlock_wait_cycles":467456,"thread_start_us":38453,"threads_started":1}]]}
W20260505 13:58:44.515462 7329 meta_cache.cc:1261] Time spent looking up entry by key: real 0.079s user 0.004s sys 0.000s
I20260505 13:58:45.218995 7498 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=973, attempt_no=0}) took 1006 ms. Trace:
I20260505 13:58:45.221271 7498 rpcz_store.cc:276] 0505 13:58:44.212025 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:44.212502 (+ 477us) service_pool.cc:224] Handling call
0505 13:58:45.217994 (+1005492us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:44.778533 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:44.779286 (+ 753us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:44.779345 (+ 59us) write_op.cc:435] Acquired schema lock
0505 13:58:44.779367 (+ 22us) tablet.cc:662] Decoding operations
0505 13:58:44.837673 (+ 58306us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:44.837779 (+ 106us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:44.837803 (+ 24us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:44.994327 (+156524us) lock_manager.cc:598] Waited 121303us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:44.995147 (+ 820us) tablet.cc:701] Row locks acquired
0505 13:58:44.995395 (+ 248us) write_op.cc:260] PREPARE: finished
0505 13:58:45.005620 (+ 10225us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b34000016d8. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45c8feef61 00007f45c902d2f4 00007f45cb7ed625 00007f45cb7ed1c8 00007f45cb7f1e61 00007f45cb7f1e19 00007f45cb7f1da9 00007f45cb7f1d71 00007f45cb7f0f9d 00000000005b3bc4 00000000005b39f9
0505 13:58:45.009243 (+ 3623us) write_op.cc:270] Start()
0505 13:58:45.009394 (+ 151us) write_op.cc:276] Timestamp: P: 1777989525008408 usec, L: 0
0505 13:58:45.009457 (+ 63us) op_driver.cc:348] REPLICATION: starting
0505 13:58:45.011879 (+ 2422us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:45.093772 (+ 81893us) op_driver.cc:464] REPLICATION: finished
0505 13:58:45.101019 (+ 7247us) write_op.cc:301] APPLY: starting
0505 13:58:45.101491 (+ 472us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:45.104074 (+ 2583us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:45.104435 (+ 361us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:45.198602 (+ 94167us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:45.205407 (+ 6805us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:45.205693 (+ 286us) write_op.cc:312] APPLY: finished
0505 13:58:45.209393 (+ 3700us) log.cc:844] Serialized 320 byte log entry
0505 13:58:45.214339 (+ 4946us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:45.216151 (+ 1812us) write_op.cc:454] Released schema lock
0505 13:58:45.216698 (+ 547us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6566,"num_ops":50,"prepare.queue_time_us":562521,"prepare.run_cpu_time_us":26388,"prepare.run_wall_time_us":252290,"raft.queue_time_us":410,"raft.run_cpu_time_us":4703,"raft.run_wall_time_us":30572,"replication_time_us":83513,"row_lock_wait_count":1,"row_lock_wait_us":121303,"spinlock_wait_cycles":2686848}]]}
I20260505 13:58:45.366405 7498 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=974, attempt_no=0}) took 1108 ms. Trace:
I20260505 13:58:45.375525 7498 rpcz_store.cc:276] 0505 13:58:44.258907 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:44.259485 (+ 578us) service_pool.cc:224] Handling call
0505 13:58:45.366273 (+1106788us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:45.030984 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:45.031547 (+ 563us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:45.031587 (+ 40us) write_op.cc:435] Acquired schema lock
0505 13:58:45.031612 (+ 25us) tablet.cc:662] Decoding operations
0505 13:58:45.081449 (+ 49837us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:45.081555 (+ 106us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:45.081599 (+ 44us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:45.215064 (+133465us) lock_manager.cc:598] Waited 105086us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:45.215242 (+ 178us) tablet.cc:701] Row locks acquired
0505 13:58:45.215281 (+ 39us) write_op.cc:260] PREPARE: finished
0505 13:58:45.216025 (+ 744us) write_op.cc:270] Start()
0505 13:58:45.216573 (+ 548us) write_op.cc:276] Timestamp: P: 1777989525215653 usec, L: 0
0505 13:58:45.216649 (+ 76us) op_driver.cc:348] REPLICATION: starting
0505 13:58:45.219596 (+ 2947us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:45.270679 (+ 51083us) op_driver.cc:464] REPLICATION: finished
0505 13:58:45.292310 (+ 21631us) write_op.cc:301] APPLY: starting
0505 13:58:45.292702 (+ 392us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:45.295187 (+ 2485us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:45.295325 (+ 138us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:45.361506 (+ 66181us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:45.362623 (+ 1117us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:45.362696 (+ 73us) write_op.cc:312] APPLY: finished
0505 13:58:45.364068 (+ 1372us) log.cc:844] Serialized 320 byte log entry
0505 13:58:45.365494 (+ 1426us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:45.365650 (+ 156us) write_op.cc:454] Released schema lock
0505 13:58:45.365724 (+ 74us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":20959,"num_ops":50,"prepare.queue_time_us":763155,"prepare.run_cpu_time_us":21022,"prepare.run_wall_time_us":209344,"raft.queue_time_us":3625,"raft.run_cpu_time_us":5204,"raft.run_wall_time_us":15874,"replication_time_us":53606,"row_lock_wait_count":1,"row_lock_wait_us":105086,"spinlock_wait_cycles":88320}]]}
I20260505 13:58:45.559193 7498 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=975, attempt_no=0}) took 1006 ms. Trace:
I20260505 13:58:45.584199 7498 rpcz_store.cc:276] 0505 13:58:44.532888 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:44.533090 (+ 202us) service_pool.cc:224] Handling call
0505 13:58:45.539050 (+1005960us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:45.241073 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:45.241805 (+ 732us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:45.241834 (+ 29us) write_op.cc:435] Acquired schema lock
0505 13:58:45.241859 (+ 25us) tablet.cc:662] Decoding operations
0505 13:58:45.248760 (+ 6901us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:45.248817 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:45.248853 (+ 36us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:45.365652 (+116799us) lock_manager.cc:598] Waited 113452us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:45.365726 (+ 74us) tablet.cc:701] Row locks acquired
0505 13:58:45.365740 (+ 14us) write_op.cc:260] PREPARE: finished
0505 13:58:45.365968 (+ 228us) write_op.cc:270] Start()
0505 13:58:45.366139 (+ 171us) write_op.cc:276] Timestamp: P: 1777989525365883 usec, L: 0
0505 13:58:45.366157 (+ 18us) op_driver.cc:348] REPLICATION: starting
0505 13:58:45.368781 (+ 2624us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:45.382586 (+ 13805us) spinlock_profiling.cc:243] Waited 7.07 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2f8522 00007f45cb2d928a 00007f45cb2e0133 00007f45cb2e00b9 00007f45cb2e0049 00007f45cb2e0011 00007f45cb2df1dd 00000000005b3bc4 00000000005b39f9 00007f45c42c2ab5
0505 13:58:45.434828 (+ 52242us) op_driver.cc:464] REPLICATION: finished
0505 13:58:45.436072 (+ 1244us) write_op.cc:301] APPLY: starting
0505 13:58:45.436481 (+ 409us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:45.439505 (+ 3024us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:45.439599 (+ 94us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:45.487509 (+ 47910us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:45.490131 (+ 2622us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:45.490210 (+ 79us) write_op.cc:312] APPLY: finished
0505 13:58:45.491973 (+ 1763us) log.cc:844] Serialized 320 byte log entry
0505 13:58:45.493707 (+ 1734us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:45.537078 (+ 43371us) write_op.cc:454] Released schema lock
0505 13:58:45.537521 (+ 443us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":586,"num_ops":50,"prepare.queue_time_us":705365,"prepare.run_cpu_time_us":9414,"prepare.run_wall_time_us":128486,"raft.queue_time_us":6030,"raft.run_cpu_time_us":9696,"raft.run_wall_time_us":46894,"replication_time_us":68281,"row_lock_wait_count":1,"row_lock_wait_us":113452,"spinlock_wait_cycles":16255872}]]}
I20260505 13:58:45.863288 7502 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=977, attempt_no=0}) took 1130 ms. Trace:
I20260505 13:58:45.865353 7502 rpcz_store.cc:276] 0505 13:58:44.731518 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:44.731686 (+ 168us) service_pool.cc:224] Handling call
0505 13:58:45.861624 (+1129938us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:45.496132 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:45.497039 (+ 907us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:45.497081 (+ 42us) write_op.cc:435] Acquired schema lock
0505 13:58:45.497112 (+ 31us) tablet.cc:662] Decoding operations
0505 13:58:45.533752 (+ 36640us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:45.533928 (+ 176us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:45.533984 (+ 56us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:45.661368 (+127384us) lock_manager.cc:598] Waited 102137us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:45.661500 (+ 132us) tablet.cc:701] Row locks acquired
0505 13:58:45.661523 (+ 23us) write_op.cc:260] PREPARE: finished
0505 13:58:45.662072 (+ 549us) write_op.cc:270] Start()
0505 13:58:45.662197 (+ 125us) write_op.cc:276] Timestamp: P: 1777989525661906 usec, L: 0
0505 13:58:45.662214 (+ 17us) op_driver.cc:348] REPLICATION: starting
0505 13:58:45.663104 (+ 890us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:45.769922 (+106818us) op_driver.cc:464] REPLICATION: finished
0505 13:58:45.775075 (+ 5153us) write_op.cc:301] APPLY: starting
0505 13:58:45.775639 (+ 564us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:45.776882 (+ 1243us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:45.776931 (+ 49us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:45.833048 (+ 56117us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:45.836009 (+ 2961us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:45.836287 (+ 278us) write_op.cc:312] APPLY: finished
0505 13:58:45.850728 (+ 14441us) log.cc:844] Serialized 320 byte log entry
0505 13:58:45.854871 (+ 4143us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:45.859095 (+ 4224us) write_op.cc:454] Released schema lock
0505 13:58:45.859198 (+ 103us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2680,"num_ops":50,"prepare.queue_time_us":762085,"prepare.run_cpu_time_us":13278,"prepare.run_wall_time_us":167423,"raft.queue_time_us":6657,"raft.run_cpu_time_us":6061,"raft.run_wall_time_us":43625,"replication_time_us":107140,"row_lock_wait_count":1,"row_lock_wait_us":102137,"spinlock_wait_cycles":593792}]]}
I20260505 13:58:46.034950 7502 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=978, attempt_no=0}) took 1223 ms. Trace:
I20260505 13:58:46.044811 7502 rpcz_store.cc:276] 0505 13:58:44.812808 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:44.813663 (+ 855us) service_pool.cc:224] Handling call
0505 13:58:46.034737 (+1221074us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:45.663860 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:45.684724 (+ 20864us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:45.684782 (+ 58us) write_op.cc:435] Acquired schema lock
0505 13:58:45.684833 (+ 51us) tablet.cc:662] Decoding operations
0505 13:58:45.717845 (+ 33012us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:45.717982 (+ 137us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:45.756339 (+ 38357us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:45.855369 (+ 99030us) tablet.cc:701] Row locks acquired
0505 13:58:45.855502 (+ 133us) write_op.cc:260] PREPARE: finished
0505 13:58:45.856542 (+ 1040us) write_op.cc:270] Start()
0505 13:58:45.856754 (+ 212us) write_op.cc:276] Timestamp: P: 1777989525856292 usec, L: 0
0505 13:58:45.856783 (+ 29us) op_driver.cc:348] REPLICATION: starting
0505 13:58:45.858802 (+ 2019us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:45.925379 (+ 66577us) op_driver.cc:464] REPLICATION: finished
0505 13:58:45.926155 (+ 776us) write_op.cc:301] APPLY: starting
0505 13:58:45.926222 (+ 67us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:45.926951 (+ 729us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:45.927051 (+ 100us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:45.972607 (+ 45556us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:45.975698 (+ 3091us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:45.976069 (+ 371us) write_op.cc:312] APPLY: finished
0505 13:58:45.983012 (+ 6943us) log.cc:844] Serialized 320 byte log entry
0505 13:58:45.986649 (+ 3637us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:46.033665 (+ 47016us) write_op.cc:454] Released schema lock
0505 13:58:46.033844 (+ 179us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":367,"num_ops":50,"prepare.queue_time_us":842576,"prepare.run_cpu_time_us":17005,"prepare.run_wall_time_us":215800,"raft.queue_time_us":16638,"raft.run_cpu_time_us":8131,"raft.run_wall_time_us":35176,"replication_time_us":68419,"row_lock_wait_count":1,"row_lock_wait_us":71830}]]}
I20260505 13:58:46.103181 7502 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=979, attempt_no=0}) took 1173 ms. Trace:
I20260505 13:58:46.103796 7502 rpcz_store.cc:276] 0505 13:58:44.930486 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:44.931096 (+ 610us) service_pool.cc:224] Handling call
0505 13:58:46.103084 (+1171988us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:45.880854 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:45.882497 (+ 1643us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:45.882527 (+ 30us) write_op.cc:435] Acquired schema lock
0505 13:58:45.882583 (+ 56us) tablet.cc:662] Decoding operations
0505 13:58:45.916863 (+ 34280us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:45.917109 (+ 246us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:45.917146 (+ 37us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:45.987485 (+ 70339us) tablet.cc:701] Row locks acquired
0505 13:58:45.987560 (+ 75us) write_op.cc:260] PREPARE: finished
0505 13:58:45.988248 (+ 688us) write_op.cc:270] Start()
0505 13:58:45.988765 (+ 517us) write_op.cc:276] Timestamp: P: 1777989525987942 usec, L: 0
0505 13:58:45.988822 (+ 57us) op_driver.cc:348] REPLICATION: starting
0505 13:58:45.993402 (+ 4580us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:46.012898 (+ 19496us) spinlock_profiling.cc:243] Waited 6.53 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e9470 00007f45cb2f6a98 00007f45cb2f66da 00007f45cb37ad45 00007f45cb37b1e9 00007f45cb7ed9f1 00007f45cb7ed1c8 00007f45cb7f1e61 00007f45cb7f1e19 00007f45cb7f1da9 00007f45cb7f1d71
0505 13:58:46.070927 (+ 58029us) op_driver.cc:464] REPLICATION: finished
0505 13:58:46.071611 (+ 684us) write_op.cc:301] APPLY: starting
0505 13:58:46.071720 (+ 109us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:46.072864 (+ 1144us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:46.072918 (+ 54us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:46.098884 (+ 25966us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:46.099901 (+ 1017us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:46.099992 (+ 91us) write_op.cc:312] APPLY: finished
0505 13:58:46.101713 (+ 1721us) log.cc:844] Serialized 320 byte log entry
0505 13:58:46.102053 (+ 340us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:46.102309 (+ 256us) write_op.cc:454] Released schema lock
0505 13:58:46.102499 (+ 190us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":421,"num_ops":50,"prepare.queue_time_us":945080,"prepare.run_cpu_time_us":24452,"prepare.run_wall_time_us":151951,"raft.queue_time_us":855,"raft.run_cpu_time_us":8372,"raft.run_wall_time_us":21543,"replication_time_us":81913,"row_lock_wait_count":1,"row_lock_wait_us":66142,"spinlock_wait_cycles":15023360}]]}
I20260505 13:58:46.231585 7502 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=980, attempt_no=0}) took 1064 ms. Trace:
I20260505 13:58:46.234704 7502 rpcz_store.cc:276] 0505 13:58:45.165873 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:45.166980 (+ 1107us) service_pool.cc:224] Handling call
0505 13:58:46.229473 (+1062493us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:46.037170 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:46.038610 (+ 1440us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:46.038648 (+ 38us) write_op.cc:435] Acquired schema lock
0505 13:58:46.038786 (+ 138us) tablet.cc:662] Decoding operations
0505 13:58:46.055616 (+ 16830us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:46.055658 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:46.055682 (+ 24us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:46.102920 (+ 47238us) tablet.cc:701] Row locks acquired
0505 13:58:46.102938 (+ 18us) write_op.cc:260] PREPARE: finished
0505 13:58:46.103148 (+ 210us) write_op.cc:270] Start()
0505 13:58:46.103275 (+ 127us) write_op.cc:276] Timestamp: P: 1777989526103037 usec, L: 0
0505 13:58:46.103292 (+ 17us) op_driver.cc:348] REPLICATION: starting
0505 13:58:46.104148 (+ 856us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:46.169873 (+ 65725us) op_driver.cc:464] REPLICATION: finished
0505 13:58:46.173621 (+ 3748us) write_op.cc:301] APPLY: starting
0505 13:58:46.174233 (+ 612us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:46.178169 (+ 3936us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:46.179526 (+ 1357us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:46.218159 (+ 38633us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:46.221312 (+ 3153us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:46.221444 (+ 132us) write_op.cc:312] APPLY: finished
0505 13:58:46.223302 (+ 1858us) log.cc:844] Serialized 320 byte log entry
0505 13:58:46.224640 (+ 1338us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:46.225357 (+ 717us) write_op.cc:454] Released schema lock
0505 13:58:46.225782 (+ 425us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1939,"num_ops":50,"prepare.queue_time_us":858195,"prepare.run_cpu_time_us":11259,"prepare.run_wall_time_us":71772,"raft.queue_time_us":4366,"raft.run_cpu_time_us":2062,"raft.run_wall_time_us":2053,"replication_time_us":65329,"row_lock_wait_count":1,"row_lock_wait_us":45305}]]}
I20260505 13:58:46.457825 7502 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=981, attempt_no=0}) took 1101 ms. Trace:
I20260505 13:58:46.459542 7502 rpcz_store.cc:276] 0505 13:58:45.356340 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:45.356577 (+ 237us) service_pool.cc:224] Handling call
0505 13:58:46.457277 (+1100700us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:46.104939 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:46.105584 (+ 645us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:46.105610 (+ 26us) write_op.cc:435] Acquired schema lock
0505 13:58:46.105635 (+ 25us) tablet.cc:662] Decoding operations
0505 13:58:46.116450 (+ 10815us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:46.116549 (+ 99us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:46.116578 (+ 29us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:46.228437 (+111859us) tablet.cc:701] Row locks acquired
0505 13:58:46.228606 (+ 169us) write_op.cc:260] PREPARE: finished
0505 13:58:46.229476 (+ 870us) write_op.cc:270] Start()
0505 13:58:46.229734 (+ 258us) write_op.cc:276] Timestamp: P: 1777989526229258 usec, L: 0
0505 13:58:46.229876 (+ 142us) op_driver.cc:348] REPLICATION: starting
0505 13:58:46.233822 (+ 3946us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:46.267205 (+ 33383us) op_driver.cc:464] REPLICATION: finished
0505 13:58:46.319571 (+ 52366us) write_op.cc:301] APPLY: starting
0505 13:58:46.319980 (+ 409us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:46.354638 (+ 34658us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:46.354941 (+ 303us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:46.420289 (+ 65348us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:46.424214 (+ 3925us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:46.424583 (+ 369us) write_op.cc:312] APPLY: finished
0505 13:58:46.430735 (+ 6152us) log.cc:844] Serialized 320 byte log entry
0505 13:58:46.432221 (+ 1486us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:46.432902 (+ 681us) write_op.cc:454] Released schema lock
0505 13:58:46.433419 (+ 517us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51748,"num_ops":50,"prepare.queue_time_us":746280,"prepare.run_cpu_time_us":14063,"prepare.run_wall_time_us":170944,"raft.queue_time_us":13553,"raft.run_cpu_time_us":3931,"raft.run_wall_time_us":3893,"replication_time_us":37136,"row_lock_wait_count":1,"row_lock_wait_us":63725,"thread_start_us":45800,"threads_started":1}]]}
W20260505 13:58:46.545644 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 53756.4us
I20260505 13:58:46.573793 7505 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=982, attempt_no=0}) took 1019 ms. Trace:
I20260505 13:58:46.574410 7505 rpcz_store.cc:276] 0505 13:58:45.554901 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:45.555631 (+ 730us) service_pool.cc:224] Handling call
0505 13:58:46.573112 (+1017481us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:46.279882 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:46.281889 (+ 2007us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:46.281970 (+ 81us) write_op.cc:435] Acquired schema lock
0505 13:58:46.282024 (+ 54us) tablet.cc:662] Decoding operations
0505 13:58:46.300596 (+ 18572us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:46.300828 (+ 232us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:46.301003 (+ 175us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:46.433945 (+132942us) lock_manager.cc:598] Waited 127922us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:46.434119 (+ 174us) tablet.cc:701] Row locks acquired
0505 13:58:46.434157 (+ 38us) write_op.cc:260] PREPARE: finished
0505 13:58:46.436086 (+ 1929us) write_op.cc:270] Start()
0505 13:58:46.436719 (+ 633us) write_op.cc:276] Timestamp: P: 1777989526435629 usec, L: 0
0505 13:58:46.436857 (+ 138us) op_driver.cc:348] REPLICATION: starting
0505 13:58:46.442329 (+ 5472us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:46.481471 (+ 39142us) op_driver.cc:464] REPLICATION: finished
0505 13:58:46.482986 (+ 1515us) write_op.cc:301] APPLY: starting
0505 13:58:46.483585 (+ 599us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:46.489026 (+ 5441us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:46.489114 (+ 88us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:46.569657 (+ 80543us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:46.570402 (+ 745us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:46.570487 (+ 85us) write_op.cc:312] APPLY: finished
0505 13:58:46.571835 (+ 1348us) log.cc:844] Serialized 320 byte log entry
0505 13:58:46.572239 (+ 404us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:46.572417 (+ 178us) write_op.cc:454] Released schema lock
0505 13:58:46.572519 (+ 102us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":842,"num_ops":50,"prepare.queue_time_us":716242,"prepare.run_cpu_time_us":21462,"prepare.run_wall_time_us":176292,"raft.queue_time_us":8318,"raft.run_cpu_time_us":2843,"raft.run_wall_time_us":2870,"replication_time_us":43512,"row_lock_wait_count":1,"row_lock_wait_us":127922,"spinlock_wait_cycles":1817984}]]}
I20260505 13:58:46.805804 7502 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=983, attempt_no=0}) took 1027 ms. Trace:
I20260505 13:58:46.807403 7502 rpcz_store.cc:276] 0505 13:58:45.765702 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:45.766112 (+ 410us) service_pool.cc:224] Handling call
0505 13:58:45.798919 (+ 32807us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b54000600a8. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb721671 00007f45cb7ab81f 00007f45cb7acf81 00007f45cc3af1de 00007f45cab39084 00007f45cab39012 00007f45cab38f41 00007f45cab38ebc 00007f45cab38172 00007f45c9072bcc 00007f45c9072006
0505 13:58:46.791483 (+992564us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:46.454025 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:46.455596 (+ 1571us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:46.455671 (+ 75us) write_op.cc:435] Acquired schema lock
0505 13:58:46.455752 (+ 81us) tablet.cc:662] Decoding operations
0505 13:58:46.520754 (+ 65002us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:46.521585 (+ 831us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:46.522009 (+ 424us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:46.572441 (+ 50432us) tablet.cc:701] Row locks acquired
0505 13:58:46.572469 (+ 28us) write_op.cc:260] PREPARE: finished
0505 13:58:46.572816 (+ 347us) write_op.cc:270] Start()
0505 13:58:46.572948 (+ 132us) write_op.cc:276] Timestamp: P: 1777989526572587 usec, L: 0
0505 13:58:46.572967 (+ 19us) op_driver.cc:348] REPLICATION: starting
0505 13:58:46.574302 (+ 1335us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:46.591326 (+ 17024us) op_driver.cc:464] REPLICATION: finished
0505 13:58:46.597989 (+ 6663us) write_op.cc:301] APPLY: starting
0505 13:58:46.598268 (+ 279us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:46.640105 (+ 41837us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:46.640591 (+ 486us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:46.649012 (+ 8421us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2fff06 00007f45cb2f987a 00007f45cb2f8af4 00007f45cb2d928a 00007f45cb2e0133 00007f45cb2e00b9 00007f45cb2e0049 00007f45cb2e0011 00007f45cb2df1dd 00000000005b3bc4
0505 13:58:46.753353 (+104341us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:46.768677 (+ 15324us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:46.768943 (+ 266us) write_op.cc:312] APPLY: finished
0505 13:58:46.774491 (+ 5548us) log.cc:844] Serialized 320 byte log entry
0505 13:58:46.789291 (+ 14800us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:46.789680 (+ 389us) write_op.cc:454] Released schema lock
0505 13:58:46.789962 (+ 282us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":27620096,"child_traces":[["op",{"apply.queue_time_us":6024,"mutex_wait_us":13335,"num_ops":50,"prepare.queue_time_us":647178,"prepare.run_cpu_time_us":23500,"prepare.run_wall_time_us":134426,"raft.queue_time_us":14970,"raft.run_cpu_time_us":11064,"raft.run_wall_time_us":97310,"replication_time_us":18151,"row_lock_wait_count":1,"row_lock_wait_us":43907,"spinlock_wait_cycles":44489088}]]}
I20260505 13:58:46.969475 7505 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=984, attempt_no=0}) took 1130 ms. Trace:
I20260505 13:58:46.970243 7505 rpcz_store.cc:276] 0505 13:58:45.840158 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:45.840963 (+ 805us) service_pool.cc:224] Handling call
0505 13:58:46.969341 (+1128378us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:46.588562 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:46.595023 (+ 6461us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:46.595358 (+ 335us) write_op.cc:435] Acquired schema lock
0505 13:58:46.595436 (+ 78us) tablet.cc:662] Decoding operations
0505 13:58:46.620436 (+ 25000us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:46.620515 (+ 79us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:46.620628 (+ 113us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:46.789593 (+168965us) lock_manager.cc:598] Waited 150515us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:46.789839 (+ 246us) tablet.cc:701] Row locks acquired
0505 13:58:46.789864 (+ 25us) write_op.cc:260] PREPARE: finished
0505 13:58:46.790412 (+ 548us) write_op.cc:270] Start()
0505 13:58:46.790817 (+ 405us) write_op.cc:276] Timestamp: P: 1777989526790205 usec, L: 0
0505 13:58:46.790953 (+ 136us) op_driver.cc:348] REPLICATION: starting
0505 13:58:46.793456 (+ 2503us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:46.843820 (+ 50364us) op_driver.cc:464] REPLICATION: finished
0505 13:58:46.845209 (+ 1389us) write_op.cc:301] APPLY: starting
0505 13:58:46.845866 (+ 657us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:46.849335 (+ 3469us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:46.849435 (+ 100us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:46.957695 (+108260us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:46.960459 (+ 2764us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:46.960565 (+ 106us) write_op.cc:312] APPLY: finished
0505 13:58:46.962833 (+ 2268us) log.cc:844] Serialized 320 byte log entry
0505 13:58:46.964127 (+ 1294us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:46.968567 (+ 4440us) write_op.cc:454] Released schema lock
0505 13:58:46.968763 (+ 196us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":636,"num_ops":50,"prepare.queue_time_us":740728,"prepare.run_cpu_time_us":22845,"prepare.run_wall_time_us":212884,"raft.queue_time_us":866,"raft.run_cpu_time_us":6676,"raft.run_wall_time_us":18179,"replication_time_us":52098,"row_lock_wait_count":1,"row_lock_wait_us":150515,"spinlock_wait_cycles":1290752}]]}
I20260505 13:58:47.133806 7505 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=985, attempt_no=0}) took 1049 ms. Trace:
I20260505 13:58:47.135555 7505 rpcz_store.cc:276] 0505 13:58:46.084794 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:46.085051 (+ 257us) service_pool.cc:224] Handling call
0505 13:58:47.132068 (+1047017us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:46.802156 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:46.803163 (+ 1007us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:46.803200 (+ 37us) write_op.cc:435] Acquired schema lock
0505 13:58:46.803252 (+ 52us) tablet.cc:662] Decoding operations
0505 13:58:46.821032 (+ 17780us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:46.821119 (+ 87us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:46.821163 (+ 44us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:46.964321 (+143158us) lock_manager.cc:598] Waited 140703us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:46.964413 (+ 92us) tablet.cc:701] Row locks acquired
0505 13:58:46.964426 (+ 13us) write_op.cc:260] PREPARE: finished
0505 13:58:46.964662 (+ 236us) write_op.cc:270] Start()
0505 13:58:46.964786 (+ 124us) write_op.cc:276] Timestamp: P: 1777989526964577 usec, L: 0
0505 13:58:46.964814 (+ 28us) op_driver.cc:348] REPLICATION: starting
0505 13:58:46.965806 (+ 992us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:47.035769 (+ 69963us) op_driver.cc:464] REPLICATION: finished
0505 13:58:47.036704 (+ 935us) write_op.cc:301] APPLY: starting
0505 13:58:47.037751 (+ 1047us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:47.042000 (+ 4249us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:47.042275 (+ 275us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:47.127043 (+ 84768us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:47.127992 (+ 949us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:47.128060 (+ 68us) write_op.cc:312] APPLY: finished
0505 13:58:47.129372 (+ 1312us) log.cc:844] Serialized 320 byte log entry
0505 13:58:47.130157 (+ 785us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:47.130650 (+ 493us) write_op.cc:454] Released schema lock
0505 13:58:47.130934 (+ 284us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":199,"mutex_wait_us":10217,"num_ops":50,"prepare.queue_time_us":713820,"prepare.run_cpu_time_us":12169,"prepare.run_wall_time_us":174538,"raft.queue_time_us":14612,"raft.run_cpu_time_us":3327,"raft.run_wall_time_us":12576,"replication_time_us":70442,"row_lock_wait_count":1,"row_lock_wait_us":140703}]]}
I20260505 13:58:47.342147 7505 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=986, attempt_no=0}) took 1194 ms. Trace:
I20260505 13:58:47.343607 7505 rpcz_store.cc:276] 0505 13:58:46.147877 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:46.150670 (+ 2793us) service_pool.cc:224] Handling call
0505 13:58:47.341907 (+1191237us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:46.976761 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:46.977650 (+ 889us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:46.977679 (+ 29us) write_op.cc:435] Acquired schema lock
0505 13:58:46.977691 (+ 12us) tablet.cc:662] Decoding operations
0505 13:58:47.011576 (+ 33885us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:47.011625 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:47.011652 (+ 27us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:47.130708 (+119056us) tablet.cc:701] Row locks acquired
0505 13:58:47.130726 (+ 18us) write_op.cc:260] PREPARE: finished
0505 13:58:47.131260 (+ 534us) write_op.cc:270] Start()
0505 13:58:47.131402 (+ 142us) write_op.cc:276] Timestamp: P: 1777989527131017 usec, L: 0
0505 13:58:47.131419 (+ 17us) op_driver.cc:348] REPLICATION: starting
0505 13:58:47.133211 (+ 1792us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:47.206850 (+ 73639us) op_driver.cc:464] REPLICATION: finished
0505 13:58:47.208738 (+ 1888us) write_op.cc:301] APPLY: starting
0505 13:58:47.208913 (+ 175us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:47.211160 (+ 2247us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:47.212200 (+ 1040us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:47.331787 (+119587us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:47.334455 (+ 2668us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:47.334595 (+ 140us) write_op.cc:312] APPLY: finished
0505 13:58:47.337763 (+ 3168us) log.cc:844] Serialized 320 byte log entry
0505 13:58:47.338599 (+ 836us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:47.338954 (+ 355us) write_op.cc:454] Released schema lock
0505 13:58:47.339430 (+ 476us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1105,"num_ops":50,"prepare.queue_time_us":822797,"prepare.run_cpu_time_us":13330,"prepare.run_wall_time_us":167604,"raft.queue_time_us":15413,"raft.run_cpu_time_us":7164,"raft.run_wall_time_us":11170,"replication_time_us":75099,"row_lock_wait_count":1,"row_lock_wait_us":41997}]]}
I20260505 13:58:47.464532 7505 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=987, attempt_no=0}) took 1122 ms. Trace:
I20260505 13:58:47.465924 7505 rpcz_store.cc:276] 0505 13:58:46.343507 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:46.344326 (+ 819us) service_pool.cc:224] Handling call
0505 13:58:47.464278 (+1119952us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:47.144883 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:47.145543 (+ 660us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:47.145591 (+ 48us) write_op.cc:435] Acquired schema lock
0505 13:58:47.145602 (+ 11us) tablet.cc:662] Decoding operations
0505 13:58:47.154529 (+ 8927us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:47.154605 (+ 76us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:47.154720 (+ 115us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:47.340453 (+185733us) lock_manager.cc:598] Waited 175385us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:47.340601 (+ 148us) tablet.cc:701] Row locks acquired
0505 13:58:47.340614 (+ 13us) write_op.cc:260] PREPARE: finished
0505 13:58:47.341807 (+ 1193us) write_op.cc:270] Start()
0505 13:58:47.341903 (+ 96us) write_op.cc:276] Timestamp: P: 1777989527339547 usec, L: 0
0505 13:58:47.341928 (+ 25us) op_driver.cc:348] REPLICATION: starting
0505 13:58:47.343954 (+ 2026us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:47.365342 (+ 21388us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2f8522 00007f45cb2d928a 00007f45cb2e0133 00007f45cb2e00b9 00007f45cb2e0049 00007f45cb2e0011 00007f45cb2df1dd 00000000005b3bc4 00000000005b39f9 00007f45c42c2ab5
0505 13:58:47.370641 (+ 5299us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2f8522 00007f45cb2d928a 00007f45cb2e0133 00007f45cb2e00b9 00007f45cb2e0049 00007f45cb2e0011 00007f45cb2df1dd 00000000005b3bc4 00000000005b39f9 00007f45c42c2ab5
0505 13:58:47.423416 (+ 52775us) op_driver.cc:464] REPLICATION: finished
0505 13:58:47.425704 (+ 2288us) write_op.cc:301] APPLY: starting
0505 13:58:47.425817 (+ 113us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:47.426777 (+ 960us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:47.426823 (+ 46us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:47.447830 (+ 21007us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:47.456037 (+ 8207us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:47.456087 (+ 50us) write_op.cc:312] APPLY: finished
0505 13:58:47.456954 (+ 867us) log.cc:844] Serialized 320 byte log entry
0505 13:58:47.457245 (+ 291us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:47.457461 (+ 216us) write_op.cc:454] Released schema lock
0505 13:58:47.457665 (+ 204us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2028,"mutex_wait_us":1013,"num_ops":50,"prepare.queue_time_us":792658,"prepare.run_cpu_time_us":13028,"prepare.run_wall_time_us":205895,"raft.queue_time_us":5286,"raft.run_cpu_time_us":13868,"raft.run_wall_time_us":62732,"replication_time_us":81379,"row_lock_wait_count":1,"row_lock_wait_us":175385,"spinlock_wait_cycles":75317760}]]}
I20260505 13:58:47.721794 7505 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=988, attempt_no=0}) took 1230 ms. Trace:
I20260505 13:58:47.725093 7505 rpcz_store.cc:276] 0505 13:58:46.518010 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:46.520753 (+ 2743us) service_pool.cc:224] Handling call
0505 13:58:47.721641 (+1200888us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:47.351174 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:47.352143 (+ 969us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:47.352221 (+ 78us) write_op.cc:435] Acquired schema lock
0505 13:58:47.352246 (+ 25us) tablet.cc:662] Decoding operations
0505 13:58:47.392960 (+ 40714us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:47.393015 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:47.393045 (+ 30us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:47.459159 (+ 66114us) tablet.cc:701] Row locks acquired
0505 13:58:47.459188 (+ 29us) write_op.cc:260] PREPARE: finished
0505 13:58:47.459685 (+ 497us) write_op.cc:270] Start()
0505 13:58:47.460270 (+ 585us) write_op.cc:276] Timestamp: P: 1777989527457360 usec, L: 0
0505 13:58:47.460305 (+ 35us) op_driver.cc:348] REPLICATION: starting
0505 13:58:47.463815 (+ 3510us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:47.533455 (+ 69640us) op_driver.cc:464] REPLICATION: finished
0505 13:58:47.534936 (+ 1481us) write_op.cc:301] APPLY: starting
0505 13:58:47.535021 (+ 85us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:47.538536 (+ 3515us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:47.539033 (+ 497us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:47.627873 (+ 88840us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:47.639562 (+ 11689us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:47.639836 (+ 274us) write_op.cc:312] APPLY: finished
0505 13:58:47.643080 (+ 3244us) log.cc:844] Serialized 320 byte log entry
0505 13:58:47.645686 (+ 2606us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:47.720737 (+ 75051us) write_op.cc:454] Released schema lock
0505 13:58:47.720950 (+ 213us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":873,"num_ops":50,"prepare.queue_time_us":816329,"prepare.run_cpu_time_us":18770,"prepare.run_wall_time_us":120407,"raft.queue_time_us":572,"raft.run_cpu_time_us":5891,"raft.run_wall_time_us":26116,"replication_time_us":72701,"row_lock_wait_count":1,"row_lock_wait_us":62988,"spinlock_wait_cycles":115968}]]}
W20260505 13:58:53.894506 7335 meta_cache.cc:1261] Time spent looking up entry by key: real 0.080s user 0.000s sys 0.000s
W20260505 13:58:57.168663 7332 meta_cache.cc:1261] Time spent looking up entry by key: real 0.054s user 0.004s sys 0.000s
I20260505 13:58:58.605844 7541 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1169, attempt_no=0}) took 1032 ms. Trace:
I20260505 13:58:58.610252 7541 rpcz_store.cc:276] 0505 13:58:57.571041 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:57.571621 (+ 580us) service_pool.cc:224] Handling call
0505 13:58:58.602757 (+1031136us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:58.339937 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:58.340915 (+ 978us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:58.340965 (+ 50us) write_op.cc:435] Acquired schema lock
0505 13:58:58.340990 (+ 25us) tablet.cc:662] Decoding operations
0505 13:58:58.387742 (+ 46752us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:58.387834 (+ 92us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:58.387893 (+ 59us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:58.449098 (+ 61205us) tablet.cc:701] Row locks acquired
0505 13:58:58.449115 (+ 17us) write_op.cc:260] PREPARE: finished
0505 13:58:58.449285 (+ 170us) write_op.cc:270] Start()
0505 13:58:58.449405 (+ 120us) write_op.cc:276] Timestamp: P: 1777989538449195 usec, L: 0
0505 13:58:58.449423 (+ 18us) op_driver.cc:348] REPLICATION: starting
0505 13:58:58.450649 (+ 1226us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:58.489855 (+ 39206us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b180005e1e0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cd0b9251 00007f45cd0c5ab1 00007f45c42c2a33 00007f45c42c4e01 00007f45c42c4db9 00007f45c42c4d49 00007f45c42c4d11 00007f45c42c400d 00000000005b3bc4 00000000005b39f9 00007f45c42a5fba
0505 13:58:58.493173 (+ 3318us) op_driver.cc:464] REPLICATION: finished
0505 13:58:58.499210 (+ 6037us) write_op.cc:301] APPLY: starting
0505 13:58:58.499530 (+ 320us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:58.502454 (+ 2924us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:58.502614 (+ 160us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:58.569057 (+ 66443us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:58.570220 (+ 1163us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:58.570306 (+ 86us) write_op.cc:312] APPLY: finished
0505 13:58:58.590203 (+ 19897us) log.cc:844] Serialized 320 byte log entry
0505 13:58:58.590803 (+ 600us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:58.591747 (+ 944us) write_op.cc:454] Released schema lock
0505 13:58:58.600487 (+ 8740us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5502,"num_ops":50,"prepare.queue_time_us":756315,"prepare.run_cpu_time_us":13198,"prepare.run_wall_time_us":117114,"raft.queue_time_us":4775,"raft.run_cpu_time_us":8100,"raft.run_wall_time_us":79229,"replication_time_us":43232,"row_lock_wait_count":1,"row_lock_wait_us":8559,"spinlock_wait_cycles":37163776}]]}
I20260505 13:58:58.919286 7541 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1171, attempt_no=0}) took 1053 ms. Trace:
I20260505 13:58:58.920509 7541 rpcz_store.cc:276] 0505 13:58:57.866376 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:57.869798 (+ 3422us) service_pool.cc:224] Handling call
0505 13:58:58.919135 (+1049337us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:58.700429 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:58.701257 (+ 828us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:58.701298 (+ 41us) write_op.cc:435] Acquired schema lock
0505 13:58:58.701326 (+ 28us) tablet.cc:662] Decoding operations
0505 13:58:58.729846 (+ 28520us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:58.730061 (+ 215us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:58.730139 (+ 78us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:58.782608 (+ 52469us) tablet.cc:701] Row locks acquired
0505 13:58:58.782747 (+ 139us) write_op.cc:260] PREPARE: finished
0505 13:58:58.783651 (+ 904us) write_op.cc:270] Start()
0505 13:58:58.784055 (+ 404us) write_op.cc:276] Timestamp: P: 1777989538783248 usec, L: 0
0505 13:58:58.784074 (+ 19us) op_driver.cc:348] REPLICATION: starting
0505 13:58:58.786957 (+ 2883us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:58.841563 (+ 54606us) op_driver.cc:464] REPLICATION: finished
0505 13:58:58.842357 (+ 794us) write_op.cc:301] APPLY: starting
0505 13:58:58.842531 (+ 174us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:58.843870 (+ 1339us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:58.844056 (+ 186us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:58.897924 (+ 53868us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:58.900481 (+ 2557us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:58.900528 (+ 47us) write_op.cc:312] APPLY: finished
0505 13:58:58.916682 (+ 16154us) log.cc:844] Serialized 320 byte log entry
0505 13:58:58.917246 (+ 564us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:58.917517 (+ 271us) write_op.cc:454] Released schema lock
0505 13:58:58.917826 (+ 309us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":247,"mutex_wait_us":7353,"num_ops":50,"prepare.queue_time_us":770475,"prepare.run_cpu_time_us":18219,"prepare.run_wall_time_us":132604,"raft.queue_time_us":7881,"raft.run_cpu_time_us":7856,"raft.run_wall_time_us":21930,"replication_time_us":57218}]]}
I20260505 13:58:59.153319 7541 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1172, attempt_no=0}) took 1023 ms. Trace:
I20260505 13:58:59.155138 7541 rpcz_store.cc:276] 0505 13:58:58.131017 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:58.131478 (+ 461us) service_pool.cc:224] Handling call
0505 13:58:59.152959 (+1021481us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:58.833571 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:58.835074 (+ 1503us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:58.835138 (+ 64us) write_op.cc:435] Acquired schema lock
0505 13:58:58.835164 (+ 26us) tablet.cc:662] Decoding operations
0505 13:58:58.877379 (+ 42215us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:58.877495 (+ 116us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:58.877594 (+ 99us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:58.918507 (+ 40913us) tablet.cc:701] Row locks acquired
0505 13:58:58.918588 (+ 81us) write_op.cc:260] PREPARE: finished
0505 13:58:58.919278 (+ 690us) write_op.cc:270] Start()
0505 13:58:58.919441 (+ 163us) write_op.cc:276] Timestamp: P: 1777989538919058 usec, L: 0
0505 13:58:58.919520 (+ 79us) op_driver.cc:348] REPLICATION: starting
0505 13:58:58.921868 (+ 2348us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:58.981424 (+ 59556us) op_driver.cc:464] REPLICATION: finished
0505 13:58:58.982655 (+ 1231us) write_op.cc:301] APPLY: starting
0505 13:58:58.982766 (+ 111us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:58.985030 (+ 2264us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:58.985396 (+ 366us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:59.077141 (+ 91745us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:59.080957 (+ 3816us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:59.081068 (+ 111us) write_op.cc:312] APPLY: finished
0505 13:58:59.084320 (+ 3252us) log.cc:844] Serialized 320 byte log entry
0505 13:58:59.085942 (+ 1622us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:59.088489 (+ 2547us) write_op.cc:454] Released schema lock
0505 13:58:59.089162 (+ 673us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":267,"num_ops":50,"prepare.queue_time_us":695135,"prepare.run_cpu_time_us":21126,"prepare.run_wall_time_us":115263,"raft.queue_time_us":9459,"raft.run_cpu_time_us":6440,"raft.run_wall_time_us":31234,"replication_time_us":61371,"row_lock_wait_count":1,"row_lock_wait_us":18367,"spinlock_wait_cycles":269568}]]}
I20260505 13:58:59.302366 7547 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1173, attempt_no=0}) took 1132 ms. Trace:
I20260505 13:58:59.305589 7547 rpcz_store.cc:276] 0505 13:58:58.168212 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:58.168596 (+ 384us) service_pool.cc:224] Handling call
0505 13:58:59.299523 (+1130927us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:58.949431 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:58.951198 (+ 1767us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:58.951255 (+ 57us) write_op.cc:435] Acquired schema lock
0505 13:58:58.951285 (+ 30us) tablet.cc:662] Decoding operations
0505 13:58:59.016158 (+ 64873us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:59.016234 (+ 76us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:59.016278 (+ 44us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:59.087009 (+ 70731us) tablet.cc:701] Row locks acquired
0505 13:58:59.087107 (+ 98us) write_op.cc:260] PREPARE: finished
0505 13:58:59.087933 (+ 826us) write_op.cc:270] Start()
0505 13:58:59.093121 (+ 5188us) write_op.cc:276] Timestamp: P: 1777989539087503 usec, L: 0
0505 13:58:59.093274 (+ 153us) op_driver.cc:348] REPLICATION: starting
0505 13:58:59.094906 (+ 1632us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:59.170883 (+ 75977us) op_driver.cc:464] REPLICATION: finished
0505 13:58:59.191754 (+ 20871us) write_op.cc:301] APPLY: starting
0505 13:58:59.200571 (+ 8817us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:59.204028 (+ 3457us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:59.205076 (+ 1048us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:59.280234 (+ 75158us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:59.283869 (+ 3635us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:59.284154 (+ 285us) write_op.cc:312] APPLY: finished
0505 13:58:59.289488 (+ 5334us) log.cc:844] Serialized 320 byte log entry
0505 13:58:59.295407 (+ 5919us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:59.297507 (+ 2100us) write_op.cc:454] Released schema lock
0505 13:58:59.297829 (+ 322us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":20087,"num_ops":50,"prepare.queue_time_us":775641,"prepare.run_cpu_time_us":19604,"prepare.run_wall_time_us":165415,"raft.queue_time_us":5527,"raft.run_cpu_time_us":6234,"raft.run_wall_time_us":36072,"replication_time_us":77303,"row_lock_wait_count":1,"row_lock_wait_us":68111,"thread_start_us":18883,"threads_started":1}]]}
I20260505 13:58:59.501288 7547 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1174, attempt_no=0}) took 1176 ms. Trace:
I20260505 13:58:59.502723 7547 rpcz_store.cc:276] 0505 13:58:58.325045 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:58.325192 (+ 147us) service_pool.cc:224] Handling call
0505 13:58:59.500733 (+1175541us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:59.114919 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:59.115366 (+ 447us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:59.115391 (+ 25us) write_op.cc:435] Acquired schema lock
0505 13:58:59.115404 (+ 13us) tablet.cc:662] Decoding operations
0505 13:58:59.118019 (+ 2615us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:59.118060 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:59.118084 (+ 24us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:59.296005 (+177921us) lock_manager.cc:598] Waited 176276us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:59.296300 (+ 295us) tablet.cc:701] Row locks acquired
0505 13:58:59.296315 (+ 15us) write_op.cc:260] PREPARE: finished
0505 13:58:59.296954 (+ 639us) write_op.cc:270] Start()
0505 13:58:59.297200 (+ 246us) write_op.cc:276] Timestamp: P: 1777989539296711 usec, L: 0
0505 13:58:59.297262 (+ 62us) op_driver.cc:348] REPLICATION: starting
0505 13:58:59.300647 (+ 3385us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:59.370174 (+ 69527us) op_driver.cc:464] REPLICATION: finished
0505 13:58:59.371569 (+ 1395us) write_op.cc:301] APPLY: starting
0505 13:58:59.371904 (+ 335us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:59.374583 (+ 2679us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:59.374744 (+ 161us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:59.446480 (+ 71736us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:59.450289 (+ 3809us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:59.450687 (+ 398us) write_op.cc:312] APPLY: finished
0505 13:58:59.456802 (+ 6115us) log.cc:844] Serialized 320 byte log entry
0505 13:58:59.459350 (+ 2548us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:59.460007 (+ 657us) write_op.cc:454] Released schema lock
0505 13:58:59.460671 (+ 664us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":557,"num_ops":50,"prepare.queue_time_us":786855,"prepare.run_cpu_time_us":11450,"prepare.run_wall_time_us":215134,"raft.queue_time_us":1053,"raft.run_cpu_time_us":5266,"raft.run_wall_time_us":23516,"replication_time_us":72368,"row_lock_wait_count":1,"row_lock_wait_us":176276,"spinlock_wait_cycles":241920}]]}
I20260505 13:58:59.692826 7541 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1175, attempt_no=0}) took 1277 ms. Trace:
I20260505 13:58:59.693403 7541 rpcz_store.cc:276] 0505 13:58:58.415746 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:58.416417 (+ 671us) service_pool.cc:224] Handling call
0505 13:58:59.692691 (+1276274us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:59.331916 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:59.333658 (+ 1742us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:59.333829 (+ 171us) write_op.cc:435] Acquired schema lock
0505 13:58:59.333856 (+ 27us) tablet.cc:662] Decoding operations
0505 13:58:59.344787 (+ 10931us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:59.344848 (+ 61us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:59.344909 (+ 61us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:59.459825 (+114916us) lock_manager.cc:598] Waited 107625us for lock on <redacted> (key hash 11785448468237383701)
0505 13:58:59.460286 (+ 461us) tablet.cc:701] Row locks acquired
0505 13:58:59.461115 (+ 829us) write_op.cc:260] PREPARE: finished
0505 13:58:59.462221 (+ 1106us) write_op.cc:270] Start()
0505 13:58:59.463008 (+ 787us) write_op.cc:276] Timestamp: P: 1777989539462025 usec, L: 0
0505 13:58:59.463168 (+ 160us) op_driver.cc:348] REPLICATION: starting
0505 13:58:59.466291 (+ 3123us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:59.516633 (+ 50342us) op_driver.cc:464] REPLICATION: finished
0505 13:58:59.517075 (+ 442us) write_op.cc:301] APPLY: starting
0505 13:58:59.517278 (+ 203us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:59.518083 (+ 805us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:59.518129 (+ 46us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:59.576479 (+ 58350us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:59.579628 (+ 3149us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:59.580112 (+ 484us) write_op.cc:312] APPLY: finished
0505 13:58:59.584738 (+ 4626us) log.cc:844] Serialized 320 byte log entry
0505 13:58:59.586957 (+ 2219us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:59.674183 (+ 87226us) write_op.cc:454] Released schema lock
0505 13:58:59.674773 (+ 590us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":166,"num_ops":50,"prepare.queue_time_us":906497,"prepare.run_cpu_time_us":25596,"prepare.run_wall_time_us":152735,"raft.queue_time_us":687,"raft.run_cpu_time_us":7771,"raft.run_wall_time_us":34250,"replication_time_us":53193,"row_lock_wait_count":1,"row_lock_wait_us":107625,"spinlock_wait_cycles":814976}]]}
I20260505 13:58:59.747022 7547 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1176, attempt_no=0}) took 1007 ms. Trace:
I20260505 13:58:59.748513 7547 rpcz_store.cc:276] 0505 13:58:58.739449 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:58.739789 (+ 340us) service_pool.cc:224] Handling call
0505 13:58:59.746709 (+1006920us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:59.485207 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:59.486600 (+ 1393us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:59.486643 (+ 43us) write_op.cc:435] Acquired schema lock
0505 13:58:59.486655 (+ 12us) tablet.cc:662] Decoding operations
0505 13:58:59.536613 (+ 49958us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:59.536824 (+ 211us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:59.536879 (+ 55us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:59.587399 (+ 50520us) tablet.cc:701] Row locks acquired
0505 13:58:59.587432 (+ 33us) write_op.cc:260] PREPARE: finished
0505 13:58:59.588019 (+ 587us) write_op.cc:270] Start()
0505 13:58:59.588256 (+ 237us) write_op.cc:276] Timestamp: P: 1777989539587855 usec, L: 0
0505 13:58:59.588289 (+ 33us) op_driver.cc:348] REPLICATION: starting
0505 13:58:59.591883 (+ 3594us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:59.649551 (+ 57668us) op_driver.cc:464] REPLICATION: finished
0505 13:58:59.650425 (+ 874us) write_op.cc:301] APPLY: starting
0505 13:58:59.650540 (+ 115us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:59.651583 (+ 1043us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:59.651875 (+ 292us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:58:59.714508 (+ 62633us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:58:59.733065 (+ 18557us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:58:59.733141 (+ 76us) write_op.cc:312] APPLY: finished
0505 13:58:59.735535 (+ 2394us) log.cc:844] Serialized 320 byte log entry
0505 13:58:59.737440 (+ 1905us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:58:59.738858 (+ 1418us) write_op.cc:454] Released schema lock
0505 13:58:59.745104 (+ 6246us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":346,"num_ops":50,"prepare.queue_time_us":739597,"prepare.run_cpu_time_us":19803,"prepare.run_wall_time_us":129598,"raft.queue_time_us":1949,"raft.run_cpu_time_us":6133,"raft.run_wall_time_us":71496,"replication_time_us":60838,"row_lock_wait_count":1,"row_lock_wait_us":23885}]]}
W20260505 13:58:59.924270 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 70032.7us
I20260505 13:59:00.058589 7547 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1178, attempt_no=0}) took 1037 ms. Trace:
I20260505 13:59:00.059216 7547 rpcz_store.cc:276] 0505 13:58:59.021963 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:59.025526 (+ 3563us) service_pool.cc:224] Handling call
0505 13:59:00.058486 (+1032960us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:58:59.842284 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:58:59.884493 (+ 42209us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:58:59.884524 (+ 31us) write_op.cc:435] Acquired schema lock
0505 13:58:59.884538 (+ 14us) tablet.cc:662] Decoding operations
0505 13:58:59.908341 (+ 23803us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:58:59.908437 (+ 96us) write_op.cc:641] Partition lock acquired for write op
0505 13:58:59.908477 (+ 40us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:58:59.910872 (+ 2395us) tablet.cc:701] Row locks acquired
0505 13:58:59.910889 (+ 17us) write_op.cc:260] PREPARE: finished
0505 13:58:59.911345 (+ 456us) write_op.cc:270] Start()
0505 13:58:59.911503 (+ 158us) write_op.cc:276] Timestamp: P: 1777989539911095 usec, L: 0
0505 13:58:59.911545 (+ 42us) op_driver.cc:348] REPLICATION: starting
0505 13:58:59.934123 (+ 22578us) log.cc:844] Serialized 2467 byte log entry
0505 13:58:59.972262 (+ 38139us) op_driver.cc:464] REPLICATION: finished
0505 13:58:59.972685 (+ 423us) write_op.cc:301] APPLY: starting
0505 13:58:59.972796 (+ 111us) tablet.cc:1367] starting BulkCheckPresence
0505 13:58:59.973939 (+ 1143us) tablet.cc:1370] finished BulkCheckPresence
0505 13:58:59.974091 (+ 152us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:00.027962 (+ 53871us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:00.031692 (+ 3730us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:00.031878 (+ 186us) write_op.cc:312] APPLY: finished
0505 13:59:00.056362 (+ 24484us) log.cc:844] Serialized 320 byte log entry
0505 13:59:00.057209 (+ 847us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:00.057851 (+ 642us) write_op.cc:454] Released schema lock
0505 13:59:00.058052 (+ 201us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":145,"num_ops":50,"prepare.queue_time_us":804514,"prepare.run_cpu_time_us":15253,"prepare.run_wall_time_us":168719,"raft.queue_time_us":244,"raft.run_cpu_time_us":7530,"raft.run_wall_time_us":47897,"replication_time_us":60531}]]}
I20260505 13:59:00.217823 7547 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1179, attempt_no=0}) took 1161 ms. Trace:
I20260505 13:59:00.218811 7547 rpcz_store.cc:276] 0505 13:58:59.056835 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:58:59.058417 (+ 1582us) service_pool.cc:224] Handling call
0505 13:59:00.217372 (+1158955us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:00.011362 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:00.012771 (+ 1409us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:00.012848 (+ 77us) write_op.cc:435] Acquired schema lock
0505 13:59:00.012888 (+ 40us) tablet.cc:662] Decoding operations
0505 13:59:00.059354 (+ 46466us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:00.059401 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:00.059423 (+ 22us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:00.069896 (+ 10473us) tablet.cc:701] Row locks acquired
0505 13:59:00.069930 (+ 34us) write_op.cc:260] PREPARE: finished
0505 13:59:00.070441 (+ 511us) write_op.cc:270] Start()
0505 13:59:00.070591 (+ 150us) write_op.cc:276] Timestamp: P: 1777989540070286 usec, L: 0
0505 13:59:00.070625 (+ 34us) op_driver.cc:348] REPLICATION: starting
0505 13:59:00.121560 (+ 50935us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:00.156500 (+ 34940us) op_driver.cc:464] REPLICATION: finished
0505 13:59:00.158504 (+ 2004us) write_op.cc:301] APPLY: starting
0505 13:59:00.158747 (+ 243us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:00.161896 (+ 3149us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:00.162290 (+ 394us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:00.210687 (+ 48397us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:00.212286 (+ 1599us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:00.212537 (+ 251us) write_op.cc:312] APPLY: finished
0505 13:59:00.214928 (+ 2391us) log.cc:844] Serialized 320 byte log entry
0505 13:59:00.215899 (+ 971us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:00.216135 (+ 236us) write_op.cc:454] Released schema lock
0505 13:59:00.216237 (+ 102us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":817,"num_ops":50,"prepare.queue_time_us":939314,"prepare.run_cpu_time_us":13829,"prepare.run_wall_time_us":171142,"raft.queue_time_us":3812,"raft.run_cpu_time_us":5938,"raft.run_wall_time_us":5921,"replication_time_us":85455}]]}
W20260505 13:59:00.222605 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 98470.4us
W20260505 13:59:13.587828 6899 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.249.3:43967, user_credentials={real_user=slave}} blocked reactor thread for 44434.2us
W20260505 13:59:14.080837 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 50491us
W20260505 13:59:14.206589 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 113219us
W20260505 13:59:14.202600 6850 debug-util.cc:398] Leaking SignalData structure 0x7b0800095d40 after lost signal to thread 6787
W20260505 13:59:14.209210 6850 debug-util.cc:398] Leaking SignalData structure 0x7b08000a4040 after lost signal to thread 6853
W20260505 13:59:16.840261 7332 meta_cache.cc:1261] Time spent looking up entry by key: real 0.064s user 0.003s sys 0.000s
W20260505 13:59:17.468039 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 61528.8us
I20260505 13:59:18.050148 7590 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1476, attempt_no=0}) took 1019 ms. Trace:
I20260505 13:59:18.069885 7590 rpcz_store.cc:276] 0505 13:59:17.029244 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:17.029887 (+ 643us) service_pool.cc:224] Handling call
0505 13:59:18.047735 (+1017848us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:17.586480 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:17.587023 (+ 543us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:17.587046 (+ 23us) write_op.cc:435] Acquired schema lock
0505 13:59:17.587058 (+ 12us) tablet.cc:662] Decoding operations
0505 13:59:17.600604 (+ 13546us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:17.600701 (+ 97us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:17.600744 (+ 43us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:17.740274 (+139530us) tablet.cc:701] Row locks acquired
0505 13:59:17.740314 (+ 40us) write_op.cc:260] PREPARE: finished
0505 13:59:17.740769 (+ 455us) write_op.cc:270] Start()
0505 13:59:17.740886 (+ 117us) write_op.cc:276] Timestamp: P: 1777989557740569 usec, L: 0
0505 13:59:17.740911 (+ 25us) op_driver.cc:348] REPLICATION: starting
0505 13:59:17.742920 (+ 2009us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:17.788825 (+ 45905us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2f8522 00007f45cb2d928a 00007f45cb2e0133 00007f45cb2e00b9 00007f45cb2e0049 00007f45cb2e0011 00007f45cb2df1dd 00000000005b3bc4 00000000005b39f9 00007f45c42c2ab5
0505 13:59:17.822627 (+ 33802us) op_driver.cc:464] REPLICATION: finished
0505 13:59:17.823616 (+ 989us) write_op.cc:301] APPLY: starting
0505 13:59:17.823818 (+ 202us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:17.825229 (+ 1411us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:17.825253 (+ 24us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:18.012684 (+187431us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:18.013747 (+ 1063us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:18.013826 (+ 79us) write_op.cc:312] APPLY: finished
0505 13:59:18.028528 (+ 14702us) log.cc:844] Serialized 320 byte log entry
0505 13:59:18.045731 (+ 17203us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:18.046287 (+ 556us) write_op.cc:454] Released schema lock
0505 13:59:18.046504 (+ 217us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":458,"mutex_wait_us":3276,"num_ops":50,"prepare.queue_time_us":537746,"prepare.run_cpu_time_us":18039,"prepare.run_wall_time_us":174623,"raft.queue_time_us":9980,"raft.run_cpu_time_us":12143,"raft.run_wall_time_us":100888,"replication_time_us":81248,"row_lock_wait_count":1,"row_lock_wait_us":86326,"spinlock_wait_cycles":39397120}]]}
I20260505 13:59:18.220806 7590 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1477, attempt_no=0}) took 1174 ms. Trace:
I20260505 13:59:18.222613 7590 rpcz_store.cc:276] 0505 13:59:17.046575 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:17.047146 (+ 571us) service_pool.cc:224] Handling call
0505 13:59:18.220567 (+1173421us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:17.761751 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:17.763506 (+ 1755us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:17.763667 (+ 161us) write_op.cc:435] Acquired schema lock
0505 13:59:17.763691 (+ 24us) tablet.cc:662] Decoding operations
0505 13:59:17.793991 (+ 30300us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:17.794046 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:17.794069 (+ 23us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:18.046921 (+252852us) lock_manager.cc:598] Waited 237315us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:18.047084 (+ 163us) tablet.cc:701] Row locks acquired
0505 13:59:18.047162 (+ 78us) write_op.cc:260] PREPARE: finished
0505 13:59:18.047951 (+ 789us) write_op.cc:270] Start()
0505 13:59:18.048478 (+ 527us) write_op.cc:276] Timestamp: P: 1777989558047670 usec, L: 0
0505 13:59:18.048739 (+ 261us) op_driver.cc:348] REPLICATION: starting
0505 13:59:18.051722 (+ 2983us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:18.120867 (+ 69145us) op_driver.cc:464] REPLICATION: finished
0505 13:59:18.125584 (+ 4717us) write_op.cc:301] APPLY: starting
0505 13:59:18.125738 (+ 154us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:18.126755 (+ 1017us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:18.126807 (+ 52us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:18.184465 (+ 57658us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:18.187069 (+ 2604us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:18.187347 (+ 278us) write_op.cc:312] APPLY: finished
0505 13:59:18.192526 (+ 5179us) log.cc:844] Serialized 320 byte log entry
0505 13:59:18.193302 (+ 776us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:18.218496 (+ 25194us) write_op.cc:454] Released schema lock
0505 13:59:18.219064 (+ 568us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":736,"mutex_wait_us":6273,"num_ops":50,"prepare.queue_time_us":707353,"prepare.run_cpu_time_us":17906,"prepare.run_wall_time_us":315061,"raft.queue_time_us":7085,"raft.run_cpu_time_us":7930,"raft.run_wall_time_us":32697,"replication_time_us":71955,"row_lock_wait_count":1,"row_lock_wait_us":237315,"spinlock_wait_cycles":264576}]]}
I20260505 13:59:18.314735 7590 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1478, attempt_no=0}) took 1064 ms. Trace:
I20260505 13:59:18.316133 7590 rpcz_store.cc:276] 0505 13:59:17.250790 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:17.251126 (+ 336us) service_pool.cc:224] Handling call
0505 13:59:18.314504 (+1063378us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:18.076806 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:18.077641 (+ 835us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:18.077723 (+ 82us) write_op.cc:435] Acquired schema lock
0505 13:59:18.077852 (+ 129us) tablet.cc:662] Decoding operations
0505 13:59:18.083397 (+ 5545us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:18.083510 (+ 113us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:18.083733 (+ 223us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:18.193740 (+110007us) lock_manager.cc:598] Waited 102696us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:18.194005 (+ 265us) tablet.cc:701] Row locks acquired
0505 13:59:18.194045 (+ 40us) write_op.cc:260] PREPARE: finished
0505 13:59:18.194546 (+ 501us) write_op.cc:270] Start()
0505 13:59:18.195005 (+ 459us) write_op.cc:276] Timestamp: P: 1777989558194374 usec, L: 0
0505 13:59:18.195111 (+ 106us) op_driver.cc:348] REPLICATION: starting
0505 13:59:18.197120 (+ 2009us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:18.211830 (+ 14710us) spinlock_profiling.cc:243] Waited 4.71 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2fff06 00007f45cb2f987a 00007f45cb2f8af4 00007f45cb2d928a 00007f45cb2e0133 00007f45cb2e00b9 00007f45cb2e0049 00007f45cb2e0011 00007f45cb2df1dd 00000000005b3bc4
0505 13:59:18.249344 (+ 37514us) op_driver.cc:464] REPLICATION: finished
0505 13:59:18.249795 (+ 451us) write_op.cc:301] APPLY: starting
0505 13:59:18.249880 (+ 85us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:18.250656 (+ 776us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:18.250686 (+ 30us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:18.276319 (+ 25633us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:18.279399 (+ 3080us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:18.279631 (+ 232us) write_op.cc:312] APPLY: finished
0505 13:59:18.282551 (+ 2920us) log.cc:844] Serialized 320 byte log entry
0505 13:59:18.283314 (+ 763us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:18.303840 (+ 20526us) write_op.cc:454] Released schema lock
0505 13:59:18.304369 (+ 529us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":236,"num_ops":50,"prepare.queue_time_us":813408,"prepare.run_cpu_time_us":18685,"prepare.run_wall_time_us":126280,"raft.queue_time_us":611,"raft.run_cpu_time_us":11250,"raft.run_wall_time_us":44781,"replication_time_us":53882,"row_lock_wait_count":1,"row_lock_wait_us":102696,"spinlock_wait_cycles":11588736}]]}
I20260505 13:59:18.372104 7590 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1479, attempt_no=0}) took 1057 ms. Trace:
I20260505 13:59:18.372943 7590 rpcz_store.cc:276] 0505 13:59:17.315315 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:17.318182 (+ 2867us) service_pool.cc:224] Handling call
0505 13:59:18.371980 (+1053798us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:18.203834 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:18.205599 (+ 1765us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:18.206035 (+ 436us) write_op.cc:435] Acquired schema lock
0505 13:59:18.206060 (+ 25us) tablet.cc:662] Decoding operations
0505 13:59:18.223794 (+ 17734us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:18.223853 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:18.223894 (+ 41us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:18.284066 (+ 60172us) tablet.cc:701] Row locks acquired
0505 13:59:18.284106 (+ 40us) write_op.cc:260] PREPARE: finished
0505 13:59:18.284718 (+ 612us) write_op.cc:270] Start()
0505 13:59:18.284922 (+ 204us) write_op.cc:276] Timestamp: P: 1777989558284412 usec, L: 0
0505 13:59:18.284942 (+ 20us) op_driver.cc:348] REPLICATION: starting
0505 13:59:18.286954 (+ 2012us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:18.354620 (+ 67666us) op_driver.cc:464] REPLICATION: finished
0505 13:59:18.356484 (+ 1864us) write_op.cc:301] APPLY: starting
0505 13:59:18.356543 (+ 59us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:18.357273 (+ 730us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:18.357394 (+ 121us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:18.367805 (+ 10411us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:18.368697 (+ 892us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:18.368772 (+ 75us) write_op.cc:312] APPLY: finished
0505 13:59:18.369887 (+ 1115us) log.cc:844] Serialized 320 byte log entry
0505 13:59:18.370952 (+ 1065us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:18.371160 (+ 208us) write_op.cc:454] Released schema lock
0505 13:59:18.371302 (+ 142us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1695,"num_ops":50,"prepare.queue_time_us":883023,"prepare.run_cpu_time_us":18244,"prepare.run_wall_time_us":92982,"raft.queue_time_us":3277,"raft.run_cpu_time_us":3504,"raft.run_wall_time_us":3563,"replication_time_us":69573,"row_lock_wait_count":1,"row_lock_wait_us":55751}]]}
W20260505 13:59:20.167155 7335 meta_cache.cc:1261] Time spent looking up entry by key: real 0.087s user 0.032s sys 0.031s
W20260505 13:59:20.171499 7329 meta_cache.cc:1261] Time spent looking up entry by key: real 0.102s user 0.004s sys 0.000s
W20260505 13:59:20.168565 7336 meta_cache.cc:1261] Time spent looking up entry by key: real 0.081s user 0.007s sys 0.000s
I20260505 13:59:23.034771 7609 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1522, attempt_no=0}) took 1079 ms. Trace:
I20260505 13:59:23.042868 7609 rpcz_store.cc:276] 0505 13:59:21.955255 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:21.955651 (+ 396us) service_pool.cc:224] Handling call
0505 13:59:23.033995 (+1078344us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:22.630379 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:22.631663 (+ 1284us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:22.631732 (+ 69us) write_op.cc:435] Acquired schema lock
0505 13:59:22.631761 (+ 29us) tablet.cc:662] Decoding operations
0505 13:59:22.645947 (+ 14186us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:22.646138 (+ 191us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:22.646199 (+ 61us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:22.801240 (+155041us) lock_manager.cc:598] Waited 147350us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:22.801322 (+ 82us) tablet.cc:701] Row locks acquired
0505 13:59:22.801350 (+ 28us) write_op.cc:260] PREPARE: finished
0505 13:59:22.801837 (+ 487us) write_op.cc:270] Start()
0505 13:59:22.802124 (+ 287us) write_op.cc:276] Timestamp: P: 1777989562801631 usec, L: 0
0505 13:59:22.802154 (+ 30us) op_driver.cc:348] REPLICATION: starting
0505 13:59:22.803286 (+ 1132us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:22.814115 (+ 10829us) spinlock_profiling.cc:243] Waited 4.58 ms on lock 0x7b18000495a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cd0b9251 00007f45cd0c5ab1 00007f45c4290be6 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cd0b9251 00007f45cd0c5ab1 00007f45c4290be6 00007f45c4290b80
0505 13:59:22.870833 (+ 56718us) op_driver.cc:464] REPLICATION: finished
0505 13:59:22.875466 (+ 4633us) write_op.cc:301] APPLY: starting
0505 13:59:22.875724 (+ 258us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:22.877731 (+ 2007us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:22.877781 (+ 50us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:23.023102 (+145321us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:23.028378 (+ 5276us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:23.028462 (+ 84us) write_op.cc:312] APPLY: finished
0505 13:59:23.029631 (+ 1169us) log.cc:844] Serialized 320 byte log entry
0505 13:59:23.029950 (+ 319us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:23.031703 (+ 1753us) write_op.cc:454] Released schema lock
0505 13:59:23.032092 (+ 389us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3927,"num_ops":50,"prepare.queue_time_us":670985,"prepare.run_cpu_time_us":14971,"prepare.run_wall_time_us":173747,"raft.queue_time_us":12307,"raft.run_cpu_time_us":2282,"raft.run_wall_time_us":2361,"replication_time_us":68396,"row_lock_wait_count":1,"row_lock_wait_us":147350,"spinlock_wait_cycles":10849792}]]}
I20260505 13:59:23.469496 7609 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1526, attempt_no=0}) took 1025 ms. Trace:
I20260505 13:59:23.471040 7609 rpcz_store.cc:276] 0505 13:59:22.444745 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:22.446062 (+ 1317us) service_pool.cc:224] Handling call
0505 13:59:23.469223 (+1023161us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:23.216698 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:23.217096 (+ 398us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:23.217120 (+ 24us) write_op.cc:435] Acquired schema lock
0505 13:59:23.217133 (+ 13us) tablet.cc:662] Decoding operations
0505 13:59:23.219453 (+ 2320us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:23.219496 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:23.219520 (+ 24us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:23.297576 (+ 78056us) tablet.cc:701] Row locks acquired
0505 13:59:23.297600 (+ 24us) write_op.cc:260] PREPARE: finished
0505 13:59:23.297876 (+ 276us) write_op.cc:270] Start()
0505 13:59:23.298001 (+ 125us) write_op.cc:276] Timestamp: P: 1777989563297754 usec, L: 0
0505 13:59:23.298021 (+ 20us) op_driver.cc:348] REPLICATION: starting
0505 13:59:23.300505 (+ 2484us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:23.375703 (+ 75198us) op_driver.cc:464] REPLICATION: finished
0505 13:59:23.377259 (+ 1556us) write_op.cc:301] APPLY: starting
0505 13:59:23.377565 (+ 306us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:23.380428 (+ 2863us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:23.380772 (+ 344us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:23.462578 (+ 81806us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:23.463449 (+ 871us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:23.463514 (+ 65us) write_op.cc:312] APPLY: finished
0505 13:59:23.465121 (+ 1607us) log.cc:844] Serialized 320 byte log entry
0505 13:59:23.465835 (+ 714us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:23.467765 (+ 1930us) write_op.cc:454] Released schema lock
0505 13:59:23.468235 (+ 470us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":871,"num_ops":50,"prepare.queue_time_us":765036,"prepare.run_cpu_time_us":7842,"prepare.run_wall_time_us":126789,"raft.queue_time_us":4341,"raft.run_cpu_time_us":5878,"raft.run_wall_time_us":50191,"replication_time_us":77206,"row_lock_wait_count":1,"row_lock_wait_us":59319}]]}
W20260505 13:59:23.973524 7338 meta_cache.cc:1261] Time spent looking up entry by key: real 0.080s user 0.004s sys 0.000s
W20260505 13:59:25.220947 7337 meta_cache.cc:1261] Time spent looking up entry by key: real 0.053s user 0.001s sys 0.000s
I20260505 13:59:27.369951 7621 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1557, attempt_no=0}) took 1132 ms. Trace:
I20260505 13:59:27.394150 7621 rpcz_store.cc:276] 0505 13:59:26.197253 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:26.198810 (+ 1557us) service_pool.cc:224] Handling call
0505 13:59:27.328943 (+1130133us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:26.875455 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:26.878318 (+ 2863us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:26.878390 (+ 72us) write_op.cc:435] Acquired schema lock
0505 13:59:26.878560 (+ 170us) tablet.cc:662] Decoding operations
0505 13:59:27.046812 (+168252us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:27.047123 (+ 311us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:27.047251 (+ 128us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:27.122413 (+ 75162us) tablet.cc:701] Row locks acquired
0505 13:59:27.122631 (+ 218us) write_op.cc:260] PREPARE: finished
0505 13:59:27.124277 (+ 1646us) write_op.cc:270] Start()
0505 13:59:27.124816 (+ 539us) write_op.cc:276] Timestamp: P: 1777989567124445 usec, L: 0
0505 13:59:27.125002 (+ 186us) op_driver.cc:348] REPLICATION: starting
0505 13:59:27.128887 (+ 3885us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:27.185159 (+ 56272us) op_driver.cc:464] REPLICATION: finished
0505 13:59:27.187778 (+ 2619us) write_op.cc:301] APPLY: starting
0505 13:59:27.187934 (+ 156us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:27.190964 (+ 3030us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:27.191472 (+ 508us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:27.255885 (+ 64413us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:27.259376 (+ 3491us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:27.259498 (+ 122us) write_op.cc:312] APPLY: finished
0505 13:59:27.263714 (+ 4216us) log.cc:844] Serialized 320 byte log entry
0505 13:59:27.264903 (+ 1189us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:27.265783 (+ 880us) write_op.cc:454] Released schema lock
0505 13:59:27.266434 (+ 651us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":3456,"child_traces":[["op",{"apply.queue_time_us":1547,"mutex_wait_us":25706,"num_ops":50,"prepare.queue_time_us":672266,"prepare.run_cpu_time_us":31241,"prepare.run_wall_time_us":280522,"raft.queue_time_us":26164,"raft.run_cpu_time_us":4547,"raft.run_wall_time_us":23149,"replication_time_us":59658,"row_lock_wait_count":1,"row_lock_wait_us":59014}]]}
I20260505 13:59:27.455466 7632 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1558, attempt_no=0}) took 1177 ms. Trace:
I20260505 13:59:27.456130 7632 rpcz_store.cc:276] 0505 13:59:26.277586 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:26.277806 (+ 220us) service_pool.cc:224] Handling call
0505 13:59:27.454640 (+1176834us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:27.159723 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:27.160358 (+ 635us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:27.160404 (+ 46us) write_op.cc:435] Acquired schema lock
0505 13:59:27.160452 (+ 48us) tablet.cc:662] Decoding operations
0505 13:59:27.163093 (+ 2641us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:27.163142 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:27.163164 (+ 22us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:27.267095 (+103931us) lock_manager.cc:598] Waited 102088us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:27.267175 (+ 80us) tablet.cc:701] Row locks acquired
0505 13:59:27.267188 (+ 13us) write_op.cc:260] PREPARE: finished
0505 13:59:27.268395 (+ 1207us) write_op.cc:270] Start()
0505 13:59:27.268511 (+ 116us) write_op.cc:276] Timestamp: P: 1777989567268843 usec, L: 0
0505 13:59:27.268527 (+ 16us) op_driver.cc:348] REPLICATION: starting
0505 13:59:27.270738 (+ 2211us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:27.323500 (+ 52762us) op_driver.cc:464] REPLICATION: finished
0505 13:59:27.406435 (+ 82935us) write_op.cc:301] APPLY: starting
0505 13:59:27.407442 (+ 1007us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:27.421000 (+ 13558us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:27.421073 (+ 73us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:27.450354 (+ 29281us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:27.451363 (+ 1009us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:27.451455 (+ 92us) write_op.cc:312] APPLY: finished
0505 13:59:27.452713 (+ 1258us) log.cc:844] Serialized 320 byte log entry
0505 13:59:27.453181 (+ 468us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:27.453461 (+ 280us) write_op.cc:454] Released schema lock
0505 13:59:27.454229 (+ 768us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":81632,"num_ops":50,"prepare.queue_time_us":873467,"prepare.run_cpu_time_us":9751,"prepare.run_wall_time_us":124065,"raft.queue_time_us":5699,"raft.run_cpu_time_us":6727,"raft.run_wall_time_us":39775,"replication_time_us":54669,"row_lock_wait_count":1,"row_lock_wait_us":102088,"spinlock_wait_cycles":1279872,"thread_start_us":66175,"threads_started":1}]]}
W20260505 13:59:27.689654 7004 debug-util.cc:398] Leaking SignalData structure 0x7b08000cffc0 after lost signal to thread 6879
W20260505 13:59:27.694406 7004 debug-util.cc:398] Leaking SignalData structure 0x7b08000c58e0 after lost signal to thread 7007
W20260505 13:59:28.028775 7321 log.cc:927] Time spent T 88ffad060bde428e92d12021c59d85f7 P 05d24c1cf32a4a18bfb58c541a620c2f: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260505 13:59:29.641403 7621 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1576, attempt_no=0}) took 1034 ms. Trace:
I20260505 13:59:29.645046 7621 rpcz_store.cc:276] 0505 13:59:28.606598 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:28.606751 (+ 153us) service_pool.cc:224] Handling call
0505 13:59:29.640842 (+1034091us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:29.385263 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:29.385873 (+ 610us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:29.385926 (+ 53us) write_op.cc:435] Acquired schema lock
0505 13:59:29.385966 (+ 40us) tablet.cc:662] Decoding operations
0505 13:59:29.389087 (+ 3121us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:29.389132 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:29.389156 (+ 24us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:29.525087 (+135931us) lock_manager.cc:598] Waited 133976us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:29.525308 (+ 221us) tablet.cc:701] Row locks acquired
0505 13:59:29.525337 (+ 29us) write_op.cc:260] PREPARE: finished
0505 13:59:29.525633 (+ 296us) write_op.cc:270] Start()
0505 13:59:29.525753 (+ 120us) write_op.cc:276] Timestamp: P: 1777989569525538 usec, L: 0
0505 13:59:29.525773 (+ 20us) op_driver.cc:348] REPLICATION: starting
0505 13:59:29.527799 (+ 2026us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:29.574749 (+ 46950us) op_driver.cc:464] REPLICATION: finished
0505 13:59:29.579232 (+ 4483us) write_op.cc:301] APPLY: starting
0505 13:59:29.579396 (+ 164us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:29.580255 (+ 859us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:29.580292 (+ 37us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:29.606079 (+ 25787us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:29.607169 (+ 1090us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:29.607283 (+ 114us) write_op.cc:312] APPLY: finished
0505 13:59:29.608817 (+ 1534us) log.cc:844] Serialized 320 byte log entry
0505 13:59:29.621473 (+ 12656us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:29.639969 (+ 18496us) write_op.cc:454] Released schema lock
0505 13:59:29.640133 (+ 164us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4183,"num_ops":50,"prepare.queue_time_us":776410,"prepare.run_cpu_time_us":9515,"prepare.run_wall_time_us":159605,"raft.queue_time_us":8340,"raft.run_cpu_time_us":5473,"raft.run_wall_time_us":17342,"replication_time_us":48735,"row_lock_wait_count":1,"row_lock_wait_us":133976}]]}
W20260505 13:59:29.727568 7331 meta_cache.cc:1261] Time spent looking up entry by key: real 0.055s user 0.004s sys 0.000s
I20260505 13:59:29.756934 7638 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1577, attempt_no=0}) took 1037 ms. Trace:
I20260505 13:59:29.757648 7638 rpcz_store.cc:276] 0505 13:59:28.719197 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:28.719346 (+ 149us) service_pool.cc:224] Handling call
0505 13:59:29.756408 (+1037062us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:29.544880 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:29.545841 (+ 961us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:29.545886 (+ 45us) write_op.cc:435] Acquired schema lock
0505 13:59:29.545908 (+ 22us) tablet.cc:662] Decoding operations
0505 13:59:29.569992 (+ 24084us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:29.570039 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:29.570065 (+ 26us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:29.621685 (+ 51620us) tablet.cc:701] Row locks acquired
0505 13:59:29.621711 (+ 26us) write_op.cc:260] PREPARE: finished
0505 13:59:29.621891 (+ 180us) write_op.cc:270] Start()
0505 13:59:29.621976 (+ 85us) write_op.cc:276] Timestamp: P: 1777989569621896 usec, L: 0
0505 13:59:29.621991 (+ 15us) op_driver.cc:348] REPLICATION: starting
0505 13:59:29.622873 (+ 882us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:29.638441 (+ 15568us) op_driver.cc:464] REPLICATION: finished
0505 13:59:29.638878 (+ 437us) write_op.cc:301] APPLY: starting
0505 13:59:29.638972 (+ 94us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:29.639953 (+ 981us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:29.640146 (+ 193us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:29.677081 (+ 36935us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:29.678050 (+ 969us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:29.678242 (+ 192us) write_op.cc:312] APPLY: finished
0505 13:59:29.688781 (+ 10539us) log.cc:844] Serialized 320 byte log entry
0505 13:59:29.690708 (+ 1927us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:29.748438 (+ 57730us) write_op.cc:454] Released schema lock
0505 13:59:29.748667 (+ 229us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":130,"num_ops":50,"prepare.queue_time_us":823824,"prepare.run_cpu_time_us":10613,"prepare.run_wall_time_us":78302,"raft.queue_time_us":4747,"raft.run_cpu_time_us":1312,"raft.run_wall_time_us":1311,"replication_time_us":16297,"row_lock_wait_count":1,"row_lock_wait_us":49896}]]}
I20260505 13:59:29.905119 7621 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1578, attempt_no=0}) took 1113 ms. Trace:
W20260505 13:59:29.920774 7331 meta_cache.cc:1261] Time spent looking up entry by key: real 0.061s user 0.002s sys 0.000s
I20260505 13:59:29.910969 7621 rpcz_store.cc:276] 0505 13:59:28.791858 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:28.792548 (+ 690us) service_pool.cc:224] Handling call
0505 13:59:29.904517 (+1111969us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:29.623363 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:29.623860 (+ 497us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:29.623884 (+ 24us) write_op.cc:435] Acquired schema lock
0505 13:59:29.623895 (+ 11us) tablet.cc:662] Decoding operations
0505 13:59:29.626016 (+ 2121us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:29.626053 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:29.626075 (+ 22us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:29.690988 (+ 64913us) tablet.cc:701] Row locks acquired
0505 13:59:29.691006 (+ 18us) write_op.cc:260] PREPARE: finished
0505 13:59:29.691421 (+ 415us) write_op.cc:270] Start()
0505 13:59:29.691704 (+ 283us) write_op.cc:276] Timestamp: P: 1777989569691398 usec, L: 0
0505 13:59:29.691756 (+ 52us) op_driver.cc:348] REPLICATION: starting
0505 13:59:29.694164 (+ 2408us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:29.731832 (+ 37668us) op_driver.cc:464] REPLICATION: finished
0505 13:59:29.732846 (+ 1014us) write_op.cc:301] APPLY: starting
0505 13:59:29.733029 (+ 183us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:29.735374 (+ 2345us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:29.735759 (+ 385us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:29.848328 (+112569us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:29.851590 (+ 3262us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:29.851808 (+ 218us) write_op.cc:312] APPLY: finished
0505 13:59:29.855525 (+ 3717us) log.cc:844] Serialized 320 byte log entry
0505 13:59:29.856362 (+ 837us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:29.883537 (+ 27175us) write_op.cc:454] Released schema lock
0505 13:59:29.884035 (+ 498us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":314,"num_ops":50,"prepare.queue_time_us":825821,"prepare.run_cpu_time_us":9012,"prepare.run_wall_time_us":85309,"raft.queue_time_us":4898,"raft.run_cpu_time_us":2059,"raft.run_wall_time_us":2056,"replication_time_us":39642,"row_lock_wait_count":1,"row_lock_wait_us":63110}]]}
W20260505 13:59:29.995028 7338 meta_cache.cc:1261] Time spent looking up entry by key: real 0.124s user 0.002s sys 0.000s
I20260505 13:59:30.170610 7638 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1579, attempt_no=0}) took 1094 ms. Trace:
I20260505 13:59:30.171411 7638 rpcz_store.cc:276] 0505 13:59:29.077189 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:29.081642 (+ 4453us) service_pool.cc:224] Handling call
0505 13:59:30.170396 (+1088754us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:29.709624 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:29.711793 (+ 2169us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:29.711857 (+ 64us) write_op.cc:435] Acquired schema lock
0505 13:59:29.711882 (+ 25us) tablet.cc:662] Decoding operations
0505 13:59:29.766122 (+ 54240us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:29.766328 (+ 206us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:29.766378 (+ 50us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:29.857067 (+ 90689us) tablet.cc:701] Row locks acquired
0505 13:59:29.857135 (+ 68us) write_op.cc:260] PREPARE: finished
0505 13:59:29.857572 (+ 437us) write_op.cc:270] Start()
0505 13:59:29.857858 (+ 286us) write_op.cc:276] Timestamp: P: 1777989569857488 usec, L: 0
0505 13:59:29.857892 (+ 34us) op_driver.cc:348] REPLICATION: starting
0505 13:59:29.859838 (+ 1946us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:29.931719 (+ 71881us) op_driver.cc:464] REPLICATION: finished
0505 13:59:29.932636 (+ 917us) write_op.cc:301] APPLY: starting
0505 13:59:29.932994 (+ 358us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:29.952278 (+ 19284us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:29.952935 (+ 657us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:30.092050 (+139115us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:30.149723 (+ 57673us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:30.149820 (+ 97us) write_op.cc:312] APPLY: finished
0505 13:59:30.167869 (+ 18049us) log.cc:844] Serialized 320 byte log entry
0505 13:59:30.169134 (+ 1265us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:30.169367 (+ 233us) write_op.cc:454] Released schema lock
0505 13:59:30.169522 (+ 155us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":447,"num_ops":50,"prepare.queue_time_us":622828,"prepare.run_cpu_time_us":20139,"prepare.run_wall_time_us":166434,"raft.queue_time_us":18218,"raft.run_cpu_time_us":6198,"raft.run_wall_time_us":7968,"replication_time_us":73412,"row_lock_wait_count":1,"row_lock_wait_us":83256}]]}
W20260505 13:59:31.827886 7338 meta_cache.cc:1261] Time spent looking up entry by key: real 0.054s user 0.004s sys 0.000s
I20260505 13:59:32.394363 7621 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1596, attempt_no=0}) took 1011 ms. Trace:
I20260505 13:59:32.395460 7621 rpcz_store.cc:276] 0505 13:59:31.379583 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:31.380593 (+ 1010us) service_pool.cc:224] Handling call
0505 13:59:32.389911 (+1009318us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:32.112033 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:32.112827 (+ 794us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:32.112861 (+ 34us) write_op.cc:435] Acquired schema lock
0505 13:59:32.112876 (+ 15us) tablet.cc:662] Decoding operations
0505 13:59:32.123049 (+ 10173us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:32.123104 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:32.123131 (+ 27us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:32.190296 (+ 67165us) tablet.cc:701] Row locks acquired
0505 13:59:32.190314 (+ 18us) write_op.cc:260] PREPARE: finished
0505 13:59:32.190591 (+ 277us) write_op.cc:270] Start()
0505 13:59:32.190722 (+ 131us) write_op.cc:276] Timestamp: P: 1777989572190515 usec, L: 0
0505 13:59:32.190740 (+ 18us) op_driver.cc:348] REPLICATION: starting
0505 13:59:32.191563 (+ 823us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:32.216492 (+ 24929us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2f8522 00007f45cb2d928a 00007f45cb2e0133 00007f45cb2e00b9 00007f45cb2e0049 00007f45cb2e0011 00007f45cb2df1dd 00000000005b3bc4 00000000005b39f9 00007f45c42c2ab5
0505 13:59:32.257481 (+ 40989us) op_driver.cc:464] REPLICATION: finished
0505 13:59:32.265174 (+ 7693us) write_op.cc:301] APPLY: starting
0505 13:59:32.265435 (+ 261us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:32.266350 (+ 915us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:32.266414 (+ 64us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:32.363259 (+ 96845us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:32.379590 (+ 16331us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:32.379822 (+ 232us) write_op.cc:312] APPLY: finished
0505 13:59:32.384284 (+ 4462us) log.cc:844] Serialized 320 byte log entry
0505 13:59:32.385647 (+ 1363us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:32.386342 (+ 695us) write_op.cc:454] Released schema lock
0505 13:59:32.388019 (+ 1677us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6814,"num_ops":50,"prepare.queue_time_us":722041,"prepare.run_cpu_time_us":12096,"prepare.run_wall_time_us":80098,"raft.queue_time_us":8650,"raft.run_cpu_time_us":7013,"raft.run_wall_time_us":48172,"replication_time_us":66183,"row_lock_wait_count":1,"row_lock_wait_us":49133,"spinlock_wait_cycles":31039744}]]}
I20260505 13:59:32.597616 7645 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1597, attempt_no=0}) took 1023 ms. Trace:
I20260505 13:59:32.599819 7645 rpcz_store.cc:276] 0505 13:59:31.571998 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:31.572355 (+ 357us) service_pool.cc:224] Handling call
0505 13:59:32.595581 (+1023226us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:32.192110 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:32.192636 (+ 526us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:32.192661 (+ 25us) write_op.cc:435] Acquired schema lock
0505 13:59:32.192674 (+ 13us) tablet.cc:662] Decoding operations
0505 13:59:32.195021 (+ 2347us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:32.195064 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:32.195087 (+ 23us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:32.388015 (+192928us) lock_manager.cc:598] Waited 164062us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:32.388414 (+ 399us) tablet.cc:701] Row locks acquired
0505 13:59:32.388443 (+ 29us) write_op.cc:260] PREPARE: finished
0505 13:59:32.390214 (+ 1771us) write_op.cc:270] Start()
0505 13:59:32.390558 (+ 344us) write_op.cc:276] Timestamp: P: 1777989572389976 usec, L: 0
0505 13:59:32.390712 (+ 154us) op_driver.cc:348] REPLICATION: starting
0505 13:59:32.392415 (+ 1703us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:32.425024 (+ 32609us) op_driver.cc:464] REPLICATION: finished
0505 13:59:32.426737 (+ 1713us) write_op.cc:301] APPLY: starting
0505 13:59:32.427328 (+ 591us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:32.429113 (+ 1785us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:32.429162 (+ 49us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:32.580013 (+150851us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:32.582686 (+ 2673us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:32.582920 (+ 234us) write_op.cc:312] APPLY: finished
0505 13:59:32.587202 (+ 4282us) log.cc:844] Serialized 320 byte log entry
0505 13:59:32.593016 (+ 5814us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:32.593582 (+ 566us) write_op.cc:454] Released schema lock
0505 13:59:32.593838 (+ 256us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1027,"num_ops":50,"prepare.queue_time_us":610564,"prepare.run_cpu_time_us":12536,"prepare.run_wall_time_us":206995,"raft.queue_time_us":7671,"raft.run_cpu_time_us":4575,"raft.run_wall_time_us":4566,"replication_time_us":33886,"row_lock_wait_count":1,"row_lock_wait_us":164062,"spinlock_wait_cycles":1914624}]]}
I20260505 13:59:32.777370 7621 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1598, attempt_no=0}) took 1080 ms. Trace:
I20260505 13:59:32.777923 7621 rpcz_store.cc:276] 0505 13:59:31.697650 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:31.699685 (+ 2035us) service_pool.cc:224] Handling call
0505 13:59:32.777215 (+1077530us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:32.399236 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:32.399894 (+ 658us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:32.399932 (+ 38us) write_op.cc:435] Acquired schema lock
0505 13:59:32.399947 (+ 15us) tablet.cc:662] Decoding operations
0505 13:59:32.402950 (+ 3003us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:32.403007 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:32.403041 (+ 34us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:32.597682 (+194641us) lock_manager.cc:598] Waited 129506us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:32.597849 (+ 167us) tablet.cc:701] Row locks acquired
0505 13:59:32.597997 (+ 148us) write_op.cc:260] PREPARE: finished
0505 13:59:32.598409 (+ 412us) write_op.cc:270] Start()
0505 13:59:32.598724 (+ 315us) write_op.cc:276] Timestamp: P: 1777989572598495 usec, L: 0
0505 13:59:32.598754 (+ 30us) op_driver.cc:348] REPLICATION: starting
0505 13:59:32.614056 (+ 15302us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:32.639820 (+ 25764us) op_driver.cc:464] REPLICATION: finished
0505 13:59:32.645697 (+ 5877us) write_op.cc:301] APPLY: starting
0505 13:59:32.646201 (+ 504us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:32.650356 (+ 4155us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:32.650466 (+ 110us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:32.749000 (+ 98534us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:32.751855 (+ 2855us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:32.751986 (+ 131us) write_op.cc:312] APPLY: finished
0505 13:59:32.755434 (+ 3448us) log.cc:844] Serialized 320 byte log entry
0505 13:59:32.764703 (+ 9269us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:32.776321 (+ 11618us) write_op.cc:454] Released schema lock
0505 13:59:32.776479 (+ 158us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":893,"num_ops":50,"prepare.queue_time_us":682172,"prepare.run_cpu_time_us":11117,"prepare.run_wall_time_us":218129,"raft.queue_time_us":1791,"raft.run_cpu_time_us":2827,"raft.run_wall_time_us":3153,"replication_time_us":40396,"row_lock_wait_count":1,"row_lock_wait_us":129506,"spinlock_wait_cycles":1168128}]]}
I20260505 13:59:32.852725 7621 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1599, attempt_no=0}) took 1154 ms. Trace:
I20260505 13:59:32.859624 7621 rpcz_store.cc:276] 0505 13:59:31.699189 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:31.704702 (+ 5513us) service_pool.cc:224] Handling call
0505 13:59:32.852602 (+1147900us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:32.617506 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:32.617947 (+ 441us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:32.617971 (+ 24us) write_op.cc:435] Acquired schema lock
0505 13:59:32.617983 (+ 12us) tablet.cc:662] Decoding operations
0505 13:59:32.620333 (+ 2350us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:32.620373 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:32.620394 (+ 21us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:32.765381 (+144987us) lock_manager.cc:598] Waited 119725us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:32.765543 (+ 162us) tablet.cc:701] Row locks acquired
0505 13:59:32.765653 (+ 110us) write_op.cc:260] PREPARE: finished
0505 13:59:32.766101 (+ 448us) write_op.cc:270] Start()
0505 13:59:32.766396 (+ 295us) write_op.cc:276] Timestamp: P: 1777989572766019 usec, L: 0
0505 13:59:32.766467 (+ 71us) op_driver.cc:348] REPLICATION: starting
0505 13:59:32.768280 (+ 1813us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:32.806361 (+ 38081us) op_driver.cc:464] REPLICATION: finished
0505 13:59:32.806769 (+ 408us) write_op.cc:301] APPLY: starting
0505 13:59:32.807070 (+ 301us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:32.807920 (+ 850us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:32.807956 (+ 36us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:32.845457 (+ 37501us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:32.846265 (+ 808us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:32.846310 (+ 45us) write_op.cc:312] APPLY: finished
0505 13:59:32.847361 (+ 1051us) log.cc:844] Serialized 320 byte log entry
0505 13:59:32.847651 (+ 290us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:32.847833 (+ 182us) write_op.cc:454] Released schema lock
0505 13:59:32.847925 (+ 92us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":198,"num_ops":50,"prepare.queue_time_us":899317,"prepare.run_cpu_time_us":8516,"prepare.run_wall_time_us":151366,"raft.queue_time_us":11852,"raft.run_cpu_time_us":4077,"raft.run_wall_time_us":12517,"replication_time_us":39757,"row_lock_wait_count":1,"row_lock_wait_us":119725,"spinlock_wait_cycles":3328}]]}
W20260505 13:59:33.652956 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 91757.6us
W20260505 13:59:33.811872 7334 meta_cache.cc:1261] Time spent looking up entry by key: real 0.056s user 0.001s sys 0.000s
I20260505 13:59:33.900429 7621 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1606, attempt_no=0}) took 1041 ms. Trace:
I20260505 13:59:33.901415 7621 rpcz_store.cc:276] 0505 13:59:32.858357 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:32.864463 (+ 6106us) service_pool.cc:224] Handling call
0505 13:59:33.899846 (+1035383us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:33.540054 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:33.540857 (+ 803us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:33.540884 (+ 27us) write_op.cc:435] Acquired schema lock
0505 13:59:33.540899 (+ 15us) tablet.cc:662] Decoding operations
0505 13:59:33.653988 (+113089us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:33.654233 (+ 245us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:33.654334 (+ 101us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:33.743575 (+ 89241us) tablet.cc:701] Row locks acquired
0505 13:59:33.743594 (+ 19us) write_op.cc:260] PREPARE: finished
0505 13:59:33.743962 (+ 368us) write_op.cc:270] Start()
0505 13:59:33.744103 (+ 141us) write_op.cc:276] Timestamp: P: 1777989573744012 usec, L: 0
0505 13:59:33.744147 (+ 44us) op_driver.cc:348] REPLICATION: starting
0505 13:59:33.745658 (+ 1511us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:33.773832 (+ 28174us) spinlock_profiling.cc:243] Waited 3.81 ms on lock 0x7b1800051700. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cd0b9251 00007f45cd0c5ab1 00007f45c42c2a33 00007f45c42c4e01 00007f45c42c4db9 00007f45c42c4d49 00007f45c42c4d11 00007f45c42c400d 00000000005b3bc4 00000000005b39f9 00007f45c42a5fba
0505 13:59:33.824601 (+ 50769us) op_driver.cc:464] REPLICATION: finished
0505 13:59:33.826051 (+ 1450us) write_op.cc:301] APPLY: starting
0505 13:59:33.826207 (+ 156us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:33.829576 (+ 3369us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:33.829661 (+ 85us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:33.886344 (+ 56683us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:33.887278 (+ 934us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:33.887363 (+ 85us) write_op.cc:312] APPLY: finished
0505 13:59:33.888572 (+ 1209us) log.cc:844] Serialized 320 byte log entry
0505 13:59:33.899143 (+ 10571us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:33.899321 (+ 178us) write_op.cc:454] Released schema lock
0505 13:59:33.899414 (+ 93us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":370,"num_ops":50,"prepare.queue_time_us":673722,"prepare.run_cpu_time_us":12909,"prepare.run_wall_time_us":206538,"raft.queue_time_us":872,"raft.run_cpu_time_us":4525,"raft.run_wall_time_us":32142,"replication_time_us":79763,"row_lock_wait_count":1,"row_lock_wait_us":84237,"spinlock_wait_cycles":8756992}]]}
W20260505 13:59:34.863971 6899 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.249.3:43967, user_credentials={real_user=slave}} blocked reactor thread for 153439us
I20260505 13:59:35.553912 7621 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1619, attempt_no=0}) took 1121 ms. Trace:
I20260505 13:59:35.555624 7621 rpcz_store.cc:276] 0505 13:59:34.416654 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:34.417149 (+ 495us) service_pool.cc:224] Handling call
0505 13:59:35.538185 (+1121036us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:35.158550 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:35.159147 (+ 597us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:35.159184 (+ 37us) write_op.cc:435] Acquired schema lock
0505 13:59:35.159197 (+ 13us) tablet.cc:662] Decoding operations
0505 13:59:35.183334 (+ 24137us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:35.183383 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:35.183409 (+ 26us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:35.322067 (+138658us) tablet.cc:701] Row locks acquired
0505 13:59:35.322085 (+ 18us) write_op.cc:260] PREPARE: finished
0505 13:59:35.322666 (+ 581us) write_op.cc:270] Start()
0505 13:59:35.323164 (+ 498us) write_op.cc:276] Timestamp: P: 1777989575322667 usec, L: 0
0505 13:59:35.323201 (+ 37us) op_driver.cc:348] REPLICATION: starting
0505 13:59:35.325903 (+ 2702us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:35.438601 (+112698us) op_driver.cc:464] REPLICATION: finished
0505 13:59:35.441454 (+ 2853us) write_op.cc:301] APPLY: starting
0505 13:59:35.441595 (+ 141us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:35.443194 (+ 1599us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:35.443667 (+ 473us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:35.513599 (+ 69932us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:35.514671 (+ 1072us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:35.514786 (+ 115us) write_op.cc:312] APPLY: finished
0505 13:59:35.516429 (+ 1643us) log.cc:844] Serialized 320 byte log entry
0505 13:59:35.516788 (+ 359us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:35.536597 (+ 19809us) write_op.cc:454] Released schema lock
0505 13:59:35.536815 (+ 218us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2429,"num_ops":50,"prepare.queue_time_us":739164,"prepare.run_cpu_time_us":18249,"prepare.run_wall_time_us":193324,"raft.queue_time_us":1403,"raft.run_cpu_time_us":4565,"raft.run_wall_time_us":18208,"replication_time_us":115003,"row_lock_wait_count":1,"row_lock_wait_us":54709}]]}
I20260505 13:59:35.666770 7621 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1620, attempt_no=0}) took 1088 ms. Trace:
I20260505 13:59:35.676805 7621 rpcz_store.cc:276] 0505 13:59:34.579598 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:34.581047 (+ 1449us) service_pool.cc:224] Handling call
0505 13:59:35.666672 (+1085625us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:35.355562 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:35.359218 (+ 3656us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:35.359272 (+ 54us) write_op.cc:435] Acquired schema lock
0505 13:59:35.371192 (+ 11920us) tablet.cc:662] Decoding operations
0505 13:59:35.398061 (+ 26869us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:35.398104 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:35.398128 (+ 24us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:35.516928 (+118800us) lock_manager.cc:598] Waited 117014us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:35.517017 (+ 89us) tablet.cc:701] Row locks acquired
0505 13:59:35.517040 (+ 23us) write_op.cc:260] PREPARE: finished
0505 13:59:35.517415 (+ 375us) write_op.cc:270] Start()
0505 13:59:35.517543 (+ 128us) write_op.cc:276] Timestamp: P: 1777989575517486 usec, L: 0
0505 13:59:35.517568 (+ 25us) op_driver.cc:348] REPLICATION: starting
0505 13:59:35.518701 (+ 1133us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:35.608144 (+ 89443us) op_driver.cc:464] REPLICATION: finished
0505 13:59:35.617512 (+ 9368us) write_op.cc:301] APPLY: starting
0505 13:59:35.617718 (+ 206us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:35.621870 (+ 4152us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:35.621906 (+ 36us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:35.644265 (+ 22359us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:35.644668 (+ 403us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:35.644718 (+ 50us) write_op.cc:312] APPLY: finished
0505 13:59:35.645464 (+ 746us) log.cc:844] Serialized 320 byte log entry
0505 13:59:35.647152 (+ 1688us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:35.664149 (+ 16997us) write_op.cc:454] Released schema lock
0505 13:59:35.666315 (+ 2166us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":9066,"num_ops":50,"prepare.queue_time_us":766084,"prepare.run_cpu_time_us":10457,"prepare.run_wall_time_us":164715,"raft.queue_time_us":1468,"raft.run_cpu_time_us":2858,"raft.run_wall_time_us":14604,"replication_time_us":90316,"row_lock_wait_count":1,"row_lock_wait_us":117014,"thread_start_us":19652,"threads_started":1}]]}
I20260505 13:59:35.760428 7653 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1621, attempt_no=0}) took 1121 ms. Trace:
I20260505 13:59:35.762928 7653 rpcz_store.cc:276] 0505 13:59:34.638380 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:34.638821 (+ 441us) service_pool.cc:224] Handling call
0505 13:59:35.758833 (+1120012us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:35.519282 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:35.525566 (+ 6284us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:35.525643 (+ 77us) write_op.cc:435] Acquired schema lock
0505 13:59:35.525677 (+ 34us) tablet.cc:662] Decoding operations
0505 13:59:35.547573 (+ 21896us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:35.547694 (+ 121us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:35.547767 (+ 73us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:35.647440 (+ 99673us) tablet.cc:701] Row locks acquired
0505 13:59:35.647473 (+ 33us) write_op.cc:260] PREPARE: finished
0505 13:59:35.647766 (+ 293us) write_op.cc:270] Start()
0505 13:59:35.647893 (+ 127us) write_op.cc:276] Timestamp: P: 1777989575647881 usec, L: 0
0505 13:59:35.647920 (+ 27us) op_driver.cc:348] REPLICATION: starting
0505 13:59:35.648969 (+ 1049us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:35.673262 (+ 24293us) op_driver.cc:464] REPLICATION: finished
0505 13:59:35.679689 (+ 6427us) write_op.cc:301] APPLY: starting
0505 13:59:35.679821 (+ 132us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:35.680596 (+ 775us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:35.680791 (+ 195us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:35.736559 (+ 55768us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:35.737730 (+ 1171us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:35.737969 (+ 239us) write_op.cc:312] APPLY: finished
0505 13:59:35.743629 (+ 5660us) log.cc:844] Serialized 320 byte log entry
0505 13:59:35.745365 (+ 1736us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:35.757715 (+ 12350us) write_op.cc:454] Released schema lock
0505 13:59:35.757899 (+ 184us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6094,"num_ops":50,"prepare.queue_time_us":874397,"prepare.run_cpu_time_us":18553,"prepare.run_wall_time_us":130201,"raft.queue_time_us":7582,"raft.run_cpu_time_us":2627,"raft.run_wall_time_us":2625,"replication_time_us":25241,"row_lock_wait_count":1,"row_lock_wait_us":80987}]]}
I20260505 13:59:35.930506 7653 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1622, attempt_no=0}) took 1191 ms. Trace:
I20260505 13:59:35.931280 7653 rpcz_store.cc:276] 0505 13:59:34.740064 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:34.745603 (+ 5539us) service_pool.cc:224] Handling call
0505 13:59:35.930323 (+1184720us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:35.649690 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:35.650453 (+ 763us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:35.650490 (+ 37us) write_op.cc:435] Acquired schema lock
0505 13:59:35.650515 (+ 25us) tablet.cc:662] Decoding operations
0505 13:59:35.661793 (+ 11278us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:35.661837 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:35.661872 (+ 35us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:35.746169 (+ 84297us) tablet.cc:701] Row locks acquired
0505 13:59:35.746226 (+ 57us) write_op.cc:260] PREPARE: finished
0505 13:59:35.746932 (+ 706us) write_op.cc:270] Start()
0505 13:59:35.747192 (+ 260us) write_op.cc:276] Timestamp: P: 1777989575746957 usec, L: 0
0505 13:59:35.747256 (+ 64us) op_driver.cc:348] REPLICATION: starting
0505 13:59:35.751206 (+ 3950us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:35.824073 (+ 72867us) op_driver.cc:464] REPLICATION: finished
0505 13:59:35.825475 (+ 1402us) write_op.cc:301] APPLY: starting
0505 13:59:35.825683 (+ 208us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:35.828805 (+ 3122us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:35.829176 (+ 371us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:35.895586 (+ 66410us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:35.899107 (+ 3521us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:35.899304 (+ 197us) write_op.cc:312] APPLY: finished
0505 13:59:35.904648 (+ 5344us) log.cc:844] Serialized 320 byte log entry
0505 13:59:35.905536 (+ 888us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:35.929528 (+ 23992us) write_op.cc:454] Released schema lock
0505 13:59:35.929704 (+ 176us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":568,"mutex_wait_us":16248,"num_ops":50,"prepare.queue_time_us":819768,"prepare.run_cpu_time_us":12120,"prepare.run_wall_time_us":124833,"raft.queue_time_us":19751,"raft.run_cpu_time_us":7896,"raft.run_wall_time_us":15980,"replication_time_us":76212,"row_lock_wait_count":1,"row_lock_wait_us":81856}]]}
I20260505 13:59:36.024955 7621 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1623, attempt_no=0}) took 1152 ms. Trace:
I20260505 13:59:36.107689 7621 rpcz_store.cc:276] 0505 13:59:34.872812 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:34.873258 (+ 446us) service_pool.cc:224] Handling call
0505 13:59:36.024560 (+1151302us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:35.775574 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:35.784641 (+ 9067us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:35.784703 (+ 62us) write_op.cc:435] Acquired schema lock
0505 13:59:35.784715 (+ 12us) tablet.cc:662] Decoding operations
0505 13:59:35.818412 (+ 33697us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:35.818616 (+ 204us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:35.818672 (+ 56us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:35.906034 (+ 87362us) tablet.cc:701] Row locks acquired
0505 13:59:35.906089 (+ 55us) write_op.cc:260] PREPARE: finished
0505 13:59:35.906477 (+ 388us) write_op.cc:270] Start()
0505 13:59:35.906675 (+ 198us) write_op.cc:276] Timestamp: P: 1777989575906256 usec, L: 0
0505 13:59:35.906806 (+ 131us) op_driver.cc:348] REPLICATION: starting
0505 13:59:35.909958 (+ 3152us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:35.934387 (+ 24429us) op_driver.cc:464] REPLICATION: finished
0505 13:59:35.934668 (+ 281us) write_op.cc:301] APPLY: starting
0505 13:59:35.934721 (+ 53us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:35.935429 (+ 708us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:35.935558 (+ 129us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:36.008191 (+ 72633us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:36.010226 (+ 2035us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:36.010312 (+ 86us) write_op.cc:312] APPLY: finished
0505 13:59:36.011891 (+ 1579us) log.cc:844] Serialized 320 byte log entry
0505 13:59:36.020099 (+ 8208us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:36.023306 (+ 3207us) write_op.cc:454] Released schema lock
0505 13:59:36.023521 (+ 215us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":104,"num_ops":50,"prepare.queue_time_us":895551,"prepare.run_cpu_time_us":20607,"prepare.run_wall_time_us":141490,"raft.queue_time_us":4739,"raft.run_cpu_time_us":3579,"raft.run_wall_time_us":3577,"replication_time_us":27452,"row_lock_wait_count":1,"row_lock_wait_us":57769}]]}
I20260505 13:59:36.353210 7653 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1624, attempt_no=0}) took 1063 ms. Trace:
I20260505 13:59:36.353835 7653 rpcz_store.cc:276] 0505 13:59:35.275553 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:35.275786 (+ 233us) service_pool.cc:224] Handling call
0505 13:59:36.339244 (+1063458us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:36.028871 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:36.030592 (+ 1721us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:36.030661 (+ 69us) write_op.cc:435] Acquired schema lock
0505 13:59:36.030811 (+ 150us) tablet.cc:662] Decoding operations
0505 13:59:36.052801 (+ 21990us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:36.052871 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:36.053009 (+ 138us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:36.256792 (+203783us) lock_manager.cc:598] Waited 186900us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:36.257311 (+ 519us) tablet.cc:701] Row locks acquired
0505 13:59:36.257349 (+ 38us) write_op.cc:260] PREPARE: finished
0505 13:59:36.258001 (+ 652us) write_op.cc:270] Start()
0505 13:59:36.258493 (+ 492us) write_op.cc:276] Timestamp: P: 1777989576258010 usec, L: 0
0505 13:59:36.258518 (+ 25us) op_driver.cc:348] REPLICATION: starting
0505 13:59:36.260751 (+ 2233us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:36.313025 (+ 52274us) op_driver.cc:464] REPLICATION: finished
0505 13:59:36.313418 (+ 393us) write_op.cc:301] APPLY: starting
0505 13:59:36.313625 (+ 207us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:36.314469 (+ 844us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:36.314510 (+ 41us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:36.336849 (+ 22339us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:36.337662 (+ 813us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:36.337708 (+ 46us) write_op.cc:312] APPLY: finished
0505 13:59:36.338389 (+ 681us) log.cc:844] Serialized 320 byte log entry
0505 13:59:36.338608 (+ 219us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:36.338740 (+ 132us) write_op.cc:454] Released schema lock
0505 13:59:36.338826 (+ 86us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":135,"num_ops":50,"prepare.queue_time_us":747070,"prepare.run_cpu_time_us":21322,"prepare.run_wall_time_us":233071,"raft.queue_time_us":8939,"raft.run_cpu_time_us":8038,"raft.run_wall_time_us":93619,"replication_time_us":54277,"row_lock_wait_count":1,"row_lock_wait_us":186900,"spinlock_wait_cycles":76672}]]}
W20260505 13:59:36.806286 7334 meta_cache.cc:1261] Time spent looking up entry by key: real 0.054s user 0.003s sys 0.000s
I20260505 13:59:37.498137 7653 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1633, attempt_no=0}) took 1079 ms. Trace:
I20260505 13:59:37.508072 7653 rpcz_store.cc:276] 0505 13:59:36.418175 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:36.418374 (+ 199us) service_pool.cc:224] Handling call
0505 13:59:37.496790 (+1078416us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:37.085895 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:37.094257 (+ 8362us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:37.094333 (+ 76us) write_op.cc:435] Acquired schema lock
0505 13:59:37.094374 (+ 41us) tablet.cc:662] Decoding operations
0505 13:59:37.125631 (+ 31257us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:37.125700 (+ 69us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:37.125743 (+ 43us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:37.201986 (+ 76243us) tablet.cc:701] Row locks acquired
0505 13:59:37.202009 (+ 23us) write_op.cc:260] PREPARE: finished
0505 13:59:37.202347 (+ 338us) write_op.cc:270] Start()
0505 13:59:37.202507 (+ 160us) write_op.cc:276] Timestamp: P: 1777989577202376 usec, L: 0
0505 13:59:37.202547 (+ 40us) op_driver.cc:348] REPLICATION: starting
0505 13:59:37.204589 (+ 2042us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:37.378933 (+174344us) op_driver.cc:464] REPLICATION: finished
0505 13:59:37.387908 (+ 8975us) write_op.cc:301] APPLY: starting
0505 13:59:37.388298 (+ 390us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:37.390868 (+ 2570us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:37.391388 (+ 520us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:37.476320 (+ 84932us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:37.477853 (+ 1533us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:37.478147 (+ 294us) write_op.cc:312] APPLY: finished
0505 13:59:37.493287 (+ 15140us) log.cc:844] Serialized 320 byte log entry
0505 13:59:37.494378 (+ 1091us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:37.494829 (+ 451us) write_op.cc:454] Released schema lock
0505 13:59:37.495026 (+ 197us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":7992,"num_ops":50,"prepare.queue_time_us":664201,"prepare.run_cpu_time_us":16647,"prepare.run_wall_time_us":242501,"raft.queue_time_us":18084,"raft.run_cpu_time_us":11939,"raft.run_wall_time_us":67092,"replication_time_us":175132,"row_lock_wait_count":1,"row_lock_wait_us":73100}]]}
I20260505 13:59:37.716542 7653 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1634, attempt_no=0}) took 1270 ms. Trace:
I20260505 13:59:37.717206 7653 rpcz_store.cc:276] 0505 13:59:36.446336 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:36.446526 (+ 190us) service_pool.cc:224] Handling call
0505 13:59:37.716365 (+1269839us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:37.330711 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:37.333142 (+ 2431us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:37.333351 (+ 209us) write_op.cc:435] Acquired schema lock
0505 13:59:37.333418 (+ 67us) tablet.cc:662] Decoding operations
0505 13:59:37.403233 (+ 69815us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:37.403590 (+ 357us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:37.403768 (+ 178us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:37.495089 (+ 91321us) tablet.cc:701] Row locks acquired
0505 13:59:37.495254 (+ 165us) write_op.cc:260] PREPARE: finished
0505 13:59:37.501216 (+ 5962us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b34000016d8. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45c8feef61 00007f45c902d2f4 00007f45cb7ed625 00007f45cb7ed1c8 00007f45cb7f1e61 00007f45cb7f1e19 00007f45cb7f1da9 00007f45cb7f1d71 00007f45cb7f0f9d 00000000005b3bc4 00000000005b39f9
0505 13:59:37.501715 (+ 499us) write_op.cc:270] Start()
0505 13:59:37.502423 (+ 708us) write_op.cc:276] Timestamp: P: 1777989577501791 usec, L: 0
0505 13:59:37.502546 (+ 123us) op_driver.cc:348] REPLICATION: starting
0505 13:59:37.506469 (+ 3923us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:37.587084 (+ 80615us) op_driver.cc:464] REPLICATION: finished
0505 13:59:37.592003 (+ 4919us) write_op.cc:301] APPLY: starting
0505 13:59:37.592108 (+ 105us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:37.592963 (+ 855us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:37.592995 (+ 32us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:37.706829 (+113834us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:37.708199 (+ 1370us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:37.708336 (+ 137us) write_op.cc:312] APPLY: finished
0505 13:59:37.709688 (+ 1352us) log.cc:844] Serialized 320 byte log entry
0505 13:59:37.714862 (+ 5174us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:37.715032 (+ 170us) write_op.cc:454] Released schema lock
0505 13:59:37.715160 (+ 128us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4327,"num_ops":50,"prepare.queue_time_us":876923,"prepare.run_cpu_time_us":30009,"prepare.run_wall_time_us":182741,"raft.queue_time_us":7600,"raft.run_cpu_time_us":2918,"raft.run_wall_time_us":2917,"replication_time_us":83766,"row_lock_wait_count":1,"row_lock_wait_us":68878,"spinlock_wait_cycles":2946432}]]}
I20260505 13:59:37.842173 7653 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1635, attempt_no=0}) took 1248 ms. Trace:
I20260505 13:59:37.842732 7653 rpcz_store.cc:276] 0505 13:59:36.593497 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:36.593691 (+ 194us) service_pool.cc:224] Handling call
0505 13:59:37.842051 (+1248360us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:37.512765 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:37.513823 (+ 1058us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:37.513887 (+ 64us) write_op.cc:435] Acquired schema lock
0505 13:59:37.513923 (+ 36us) tablet.cc:662] Decoding operations
0505 13:59:37.527093 (+ 13170us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:37.527177 (+ 84us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:37.527219 (+ 42us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:37.715085 (+187866us) lock_manager.cc:598] Waited 184849us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:37.715163 (+ 78us) tablet.cc:701] Row locks acquired
0505 13:59:37.715176 (+ 13us) write_op.cc:260] PREPARE: finished
0505 13:59:37.715500 (+ 324us) write_op.cc:270] Start()
0505 13:59:37.715585 (+ 85us) write_op.cc:276] Timestamp: P: 1777989577715777 usec, L: 0
0505 13:59:37.715600 (+ 15us) op_driver.cc:348] REPLICATION: starting
0505 13:59:37.716443 (+ 843us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:37.731145 (+ 14702us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2f8522 00007f45cb2d928a 00007f45cb2e0133 00007f45cb2e00b9 00007f45cb2e0049 00007f45cb2e0011 00007f45cb2df1dd 00000000005b3bc4 00000000005b39f9 00007f45c42c2ab5
0505 13:59:37.768527 (+ 37382us) op_driver.cc:464] REPLICATION: finished
0505 13:59:37.774221 (+ 5694us) write_op.cc:301] APPLY: starting
0505 13:59:37.774407 (+ 186us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:37.786332 (+ 11925us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:37.786489 (+ 157us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:37.829574 (+ 43085us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:37.830300 (+ 726us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:37.830370 (+ 70us) write_op.cc:312] APPLY: finished
0505 13:59:37.831350 (+ 980us) log.cc:844] Serialized 320 byte log entry
0505 13:59:37.831805 (+ 455us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:37.839928 (+ 8123us) write_op.cc:454] Released schema lock
0505 13:59:37.840097 (+ 169us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5099,"num_ops":50,"prepare.queue_time_us":883452,"prepare.run_cpu_time_us":14393,"prepare.run_wall_time_us":204152,"raft.queue_time_us":4052,"raft.run_cpu_time_us":5976,"raft.run_wall_time_us":25206,"replication_time_us":52511,"row_lock_wait_count":1,"row_lock_wait_us":184849,"spinlock_wait_cycles":23299968}]]}
I20260505 13:59:37.872288 7653 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1636, attempt_no=0}) took 1110 ms. Trace:
I20260505 13:59:37.872932 7653 rpcz_store.cc:276] 0505 13:59:36.761815 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:36.765206 (+ 3391us) service_pool.cc:224] Handling call
0505 13:59:37.872125 (+1106919us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:37.716935 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:37.717558 (+ 623us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:37.717580 (+ 22us) write_op.cc:435] Acquired schema lock
0505 13:59:37.717592 (+ 12us) tablet.cc:662] Decoding operations
0505 13:59:37.720497 (+ 2905us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:37.720534 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:37.720555 (+ 21us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:37.831979 (+111424us) lock_manager.cc:598] Waited 109621us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:37.832074 (+ 95us) tablet.cc:701] Row locks acquired
0505 13:59:37.832086 (+ 12us) write_op.cc:260] PREPARE: finished
0505 13:59:37.832514 (+ 428us) write_op.cc:270] Start()
0505 13:59:37.832612 (+ 98us) write_op.cc:276] Timestamp: P: 1777989577832829 usec, L: 0
0505 13:59:37.832628 (+ 16us) op_driver.cc:348] REPLICATION: starting
0505 13:59:37.833614 (+ 986us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:37.854383 (+ 20769us) op_driver.cc:464] REPLICATION: finished
0505 13:59:37.859066 (+ 4683us) write_op.cc:301] APPLY: starting
0505 13:59:37.859199 (+ 133us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:37.859774 (+ 575us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:37.859810 (+ 36us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:37.868267 (+ 8457us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:37.868814 (+ 547us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:37.868891 (+ 77us) write_op.cc:312] APPLY: finished
0505 13:59:37.869902 (+ 1011us) log.cc:844] Serialized 320 byte log entry
0505 13:59:37.870253 (+ 351us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:37.871473 (+ 1220us) write_op.cc:454] Released schema lock
0505 13:59:37.871594 (+ 121us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4465,"num_ops":50,"prepare.queue_time_us":939342,"prepare.run_cpu_time_us":6982,"prepare.run_wall_time_us":117110,"raft.queue_time_us":9348,"raft.run_cpu_time_us":1902,"raft.run_wall_time_us":1882,"replication_time_us":21614,"row_lock_wait_count":1,"row_lock_wait_us":109621,"spinlock_wait_cycles":4480,"threads_started":1}]]}
I20260505 13:59:37.977830 7662 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1637, attempt_no=0}) took 1006 ms. Trace:
I20260505 13:59:37.981768 7662 rpcz_store.cc:276] 0505 13:59:36.969100 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:36.969916 (+ 816us) service_pool.cc:224] Handling call
0505 13:59:37.974669 (+1004753us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:37.834235 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:37.834913 (+ 678us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:37.834984 (+ 71us) write_op.cc:435] Acquired schema lock
0505 13:59:37.835009 (+ 25us) tablet.cc:662] Decoding operations
0505 13:59:37.837856 (+ 2847us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:37.837897 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:37.837918 (+ 21us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:37.870501 (+ 32583us) tablet.cc:701] Row locks acquired
0505 13:59:37.870519 (+ 18us) write_op.cc:260] PREPARE: finished
0505 13:59:37.870800 (+ 281us) write_op.cc:270] Start()
0505 13:59:37.870902 (+ 102us) write_op.cc:276] Timestamp: P: 1777989577870584 usec, L: 0
0505 13:59:37.870920 (+ 18us) op_driver.cc:348] REPLICATION: starting
0505 13:59:37.902367 (+ 31447us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:37.903306 (+ 939us) spinlock_profiling.cc:243] Waited 30.5 ms on lock 0x7b5400060254. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb37b208 00007f45cb7ed9f1 00007f45cb7ed1c8 00007f45cb7f1e61 00007f45cb7f1e19 00007f45cb7f1da9 00007f45cb7f1d71 00007f45cb7f0f9d 00000000005b3bc4 00000000005b39f9
0505 13:59:37.918199 (+ 14893us) op_driver.cc:464] REPLICATION: finished
0505 13:59:37.919585 (+ 1386us) write_op.cc:301] APPLY: starting
0505 13:59:37.919710 (+ 125us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:37.920941 (+ 1231us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:37.921015 (+ 74us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:37.945855 (+ 24840us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:37.947191 (+ 1336us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:37.947297 (+ 106us) write_op.cc:312] APPLY: finished
0505 13:59:37.948216 (+ 919us) log.cc:844] Serialized 320 byte log entry
0505 13:59:37.948832 (+ 616us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:37.971810 (+ 22978us) write_op.cc:454] Released schema lock
0505 13:59:37.972379 (+ 569us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1085,"num_ops":50,"prepare.queue_time_us":857240,"prepare.run_cpu_time_us":8807,"prepare.run_wall_time_us":69434,"raft.queue_time_us":3100,"raft.run_cpu_time_us":2808,"raft.run_wall_time_us":2806,"replication_time_us":47093,"row_lock_wait_count":1,"row_lock_wait_us":28664,"spinlock_wait_cycles":70307072}]]}
I20260505 13:59:38.066345 7653 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1638, attempt_no=0}) took 1067 ms. Trace:
I20260505 13:59:38.082171 7653 rpcz_store.cc:276] 0505 13:59:36.998757 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:36.999210 (+ 453us) service_pool.cc:224] Handling call
0505 13:59:38.066101 (+1066891us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:37.903883 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:37.904575 (+ 692us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:37.904612 (+ 37us) write_op.cc:435] Acquired schema lock
0505 13:59:37.904623 (+ 11us) tablet.cc:662] Decoding operations
0505 13:59:37.922616 (+ 17993us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:37.922658 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:37.922683 (+ 25us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:37.949072 (+ 26389us) tablet.cc:701] Row locks acquired
0505 13:59:37.949100 (+ 28us) write_op.cc:260] PREPARE: finished
0505 13:59:37.949495 (+ 395us) write_op.cc:270] Start()
0505 13:59:37.949698 (+ 203us) write_op.cc:276] Timestamp: P: 1777989577949209 usec, L: 0
0505 13:59:37.949719 (+ 21us) op_driver.cc:348] REPLICATION: starting
0505 13:59:37.951314 (+ 1595us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:37.961804 (+ 10490us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2f8522 00007f45cb2d928a 00007f45cb2e0133 00007f45cb2e00b9 00007f45cb2e0049 00007f45cb2e0011 00007f45cb2df1dd 00000000005b3bc4 00000000005b39f9 00007f45c42c2ab5
0505 13:59:37.988709 (+ 26905us) op_driver.cc:464] REPLICATION: finished
0505 13:59:37.989659 (+ 950us) write_op.cc:301] APPLY: starting
0505 13:59:37.989917 (+ 258us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:37.991830 (+ 1913us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:37.991965 (+ 135us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:38.059114 (+ 67149us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:38.061143 (+ 2029us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:38.061234 (+ 91us) write_op.cc:312] APPLY: finished
0505 13:59:38.063465 (+ 2231us) log.cc:844] Serialized 320 byte log entry
0505 13:59:38.064362 (+ 897us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:38.064708 (+ 346us) write_op.cc:454] Released schema lock
0505 13:59:38.064993 (+ 285us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":384,"num_ops":50,"prepare.queue_time_us":893218,"prepare.run_cpu_time_us":8423,"prepare.run_wall_time_us":48130,"raft.queue_time_us":5631,"raft.run_cpu_time_us":7402,"raft.run_wall_time_us":16220,"replication_time_us":38848,"row_lock_wait_count":1,"row_lock_wait_us":22427,"spinlock_wait_cycles":5809408}]]}
I20260505 13:59:39.607213 7653 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1649, attempt_no=0}) took 1009 ms. Trace:
I20260505 13:59:39.610901 7653 rpcz_store.cc:276] 0505 13:59:38.597854 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:38.598086 (+ 232us) service_pool.cc:224] Handling call
0505 13:59:39.606585 (+1008499us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:39.244761 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:39.245385 (+ 624us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:39.245410 (+ 25us) write_op.cc:435] Acquired schema lock
0505 13:59:39.245422 (+ 12us) tablet.cc:662] Decoding operations
0505 13:59:39.268749 (+ 23327us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:39.268847 (+ 98us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:39.268890 (+ 43us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:39.430512 (+161622us) lock_manager.cc:598] Waited 142629us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:39.430762 (+ 250us) tablet.cc:701] Row locks acquired
0505 13:59:39.430775 (+ 13us) write_op.cc:260] PREPARE: finished
0505 13:59:39.431160 (+ 385us) write_op.cc:270] Start()
0505 13:59:39.431461 (+ 301us) write_op.cc:276] Timestamp: P: 1777989579431019 usec, L: 0
0505 13:59:39.431530 (+ 69us) op_driver.cc:348] REPLICATION: starting
0505 13:59:39.434042 (+ 2512us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:39.482184 (+ 48142us) op_driver.cc:464] REPLICATION: finished
0505 13:59:39.483106 (+ 922us) write_op.cc:301] APPLY: starting
0505 13:59:39.483522 (+ 416us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:39.485241 (+ 1719us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:39.485315 (+ 74us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:39.562664 (+ 77349us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:39.563264 (+ 600us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:39.563332 (+ 68us) write_op.cc:312] APPLY: finished
0505 13:59:39.564824 (+ 1492us) log.cc:844] Serialized 320 byte log entry
0505 13:59:39.565926 (+ 1102us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:39.605838 (+ 39912us) write_op.cc:454] Released schema lock
0505 13:59:39.605989 (+ 151us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":183,"num_ops":50,"prepare.queue_time_us":644167,"prepare.run_cpu_time_us":17854,"prepare.run_wall_time_us":212978,"raft.queue_time_us":6453,"raft.run_cpu_time_us":4535,"raft.run_wall_time_us":18533,"replication_time_us":50312,"row_lock_wait_count":1,"row_lock_wait_us":142629,"spinlock_wait_cycles":1270528}]]}
W20260505 13:59:40.733327 7338 meta_cache.cc:1261] Time spent looking up entry by key: real 0.068s user 0.004s sys 0.000s
I20260505 13:59:40.885484 7668 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1658, attempt_no=0}) took 1131 ms. Trace:
I20260505 13:59:40.919078 7668 rpcz_store.cc:276] 0505 13:59:39.752346 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:39.752569 (+ 223us) service_pool.cc:224] Handling call
0505 13:59:40.883184 (+1130615us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:40.446032 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:40.446521 (+ 489us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:40.446550 (+ 29us) write_op.cc:435] Acquired schema lock
0505 13:59:40.446576 (+ 26us) tablet.cc:662] Decoding operations
0505 13:59:40.484754 (+ 38178us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:40.484862 (+ 108us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:40.484928 (+ 66us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:40.548659 (+ 63731us) tablet.cc:701] Row locks acquired
0505 13:59:40.548694 (+ 35us) write_op.cc:260] PREPARE: finished
0505 13:59:40.549251 (+ 557us) write_op.cc:270] Start()
0505 13:59:40.549382 (+ 131us) write_op.cc:276] Timestamp: P: 1777989580549194 usec, L: 0
0505 13:59:40.549446 (+ 64us) op_driver.cc:348] REPLICATION: starting
0505 13:59:40.551591 (+ 2145us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:40.623556 (+ 71965us) op_driver.cc:464] REPLICATION: finished
0505 13:59:40.625470 (+ 1914us) write_op.cc:301] APPLY: starting
0505 13:59:40.625808 (+ 338us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:40.629273 (+ 3465us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:40.629645 (+ 372us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:40.796013 (+166368us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:40.800192 (+ 4179us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:40.800476 (+ 284us) write_op.cc:312] APPLY: finished
0505 13:59:40.807494 (+ 7018us) log.cc:844] Serialized 320 byte log entry
0505 13:59:40.810579 (+ 3085us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:40.879610 (+ 69031us) write_op.cc:454] Released schema lock
0505 13:59:40.880070 (+ 460us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1074,"num_ops":50,"prepare.queue_time_us":689629,"prepare.run_cpu_time_us":18939,"prepare.run_wall_time_us":111676,"raft.queue_time_us":685,"raft.run_cpu_time_us":5870,"raft.run_wall_time_us":31578,"replication_time_us":73739,"row_lock_wait_count":1,"row_lock_wait_us":24590,"spinlock_wait_cycles":216448}]]}
I20260505 13:59:40.938396 7653 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1659, attempt_no=0}) took 1045 ms. Trace:
I20260505 13:59:40.939035 7653 rpcz_store.cc:276] 0505 13:59:39.893669 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:39.894007 (+ 338us) service_pool.cc:224] Handling call
0505 13:59:40.937876 (+1043869us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:40.558248 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:40.559642 (+ 1394us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:40.559738 (+ 96us) write_op.cc:435] Acquired schema lock
0505 13:59:40.559761 (+ 23us) tablet.cc:662] Decoding operations
0505 13:59:40.569903 (+ 10142us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:40.569956 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:40.569988 (+ 32us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:40.811201 (+241213us) lock_manager.cc:598] Waited 229534us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:40.811529 (+ 328us) tablet.cc:701] Row locks acquired
0505 13:59:40.811582 (+ 53us) write_op.cc:260] PREPARE: finished
0505 13:59:40.812499 (+ 917us) write_op.cc:270] Start()
0505 13:59:40.812766 (+ 267us) write_op.cc:276] Timestamp: P: 1777989580812562 usec, L: 0
0505 13:59:40.812901 (+ 135us) op_driver.cc:348] REPLICATION: starting
0505 13:59:40.817442 (+ 4541us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:40.895969 (+ 78527us) op_driver.cc:464] REPLICATION: finished
0505 13:59:40.896343 (+ 374us) write_op.cc:301] APPLY: starting
0505 13:59:40.896624 (+ 281us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:40.897565 (+ 941us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:40.897596 (+ 31us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:40.934171 (+ 36575us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:40.935075 (+ 904us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:40.935135 (+ 60us) write_op.cc:312] APPLY: finished
0505 13:59:40.936593 (+ 1458us) log.cc:844] Serialized 320 byte log entry
0505 13:59:40.937037 (+ 444us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:40.937308 (+ 271us) write_op.cc:454] Released schema lock
0505 13:59:40.937439 (+ 131us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":134,"num_ops":50,"prepare.queue_time_us":658421,"prepare.run_cpu_time_us":18261,"prepare.run_wall_time_us":264286,"raft.queue_time_us":373,"raft.run_cpu_time_us":3227,"raft.run_wall_time_us":3227,"replication_time_us":82830,"row_lock_wait_count":1,"row_lock_wait_us":229534}]]}
I20260505 13:59:41.045779 7653 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1660, attempt_no=0}) took 1101 ms. Trace:
I20260505 13:59:41.056638 7653 rpcz_store.cc:276] 0505 13:59:39.944541 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:39.945235 (+ 694us) service_pool.cc:224] Handling call
0505 13:59:41.045630 (+1100395us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:40.822910 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:40.823808 (+ 898us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:40.823834 (+ 26us) write_op.cc:435] Acquired schema lock
0505 13:59:40.823847 (+ 13us) tablet.cc:662] Decoding operations
0505 13:59:40.897208 (+ 73361us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:40.897305 (+ 97us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:40.897347 (+ 42us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:40.937319 (+ 39972us) tablet.cc:701] Row locks acquired
0505 13:59:40.937336 (+ 17us) write_op.cc:260] PREPARE: finished
0505 13:59:40.937603 (+ 267us) write_op.cc:270] Start()
0505 13:59:40.937716 (+ 113us) write_op.cc:276] Timestamp: P: 1777989580937558 usec, L: 0
0505 13:59:40.937733 (+ 17us) op_driver.cc:348] REPLICATION: starting
0505 13:59:40.938821 (+ 1088us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:40.972012 (+ 33191us) op_driver.cc:464] REPLICATION: finished
0505 13:59:40.972359 (+ 347us) write_op.cc:301] APPLY: starting
0505 13:59:40.972434 (+ 75us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:40.973208 (+ 774us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:40.973337 (+ 129us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:40.989439 (+ 16102us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:40.989866 (+ 427us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:40.989922 (+ 56us) write_op.cc:312] APPLY: finished
0505 13:59:40.991061 (+ 1139us) log.cc:844] Serialized 320 byte log entry
0505 13:59:40.991510 (+ 449us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:41.044380 (+ 52870us) write_op.cc:454] Released schema lock
0505 13:59:41.044717 (+ 337us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":144,"num_ops":50,"prepare.queue_time_us":871106,"prepare.run_cpu_time_us":13614,"prepare.run_wall_time_us":116602,"raft.queue_time_us":1320,"raft.run_cpu_time_us":1587,"raft.run_wall_time_us":1587,"replication_time_us":33783,"row_lock_wait_count":1,"row_lock_wait_us":37368}]]}
W20260505 13:59:41.216969 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 58869.6us
W20260505 13:59:41.335632 7335 meta_cache.cc:1261] Time spent looking up entry by key: real 0.058s user 0.003s sys 0.000s
I20260505 13:59:41.572729 7668 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1663, attempt_no=0}) took 1079 ms. Trace:
I20260505 13:59:41.630224 7668 rpcz_store.cc:276] 0505 13:59:40.492583 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:40.493395 (+ 812us) service_pool.cc:224] Handling call
0505 13:59:41.571006 (+1077611us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:41.175028 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:41.176116 (+ 1088us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:41.176205 (+ 89us) write_op.cc:435] Acquired schema lock
0505 13:59:41.176233 (+ 28us) tablet.cc:662] Decoding operations
0505 13:59:41.213653 (+ 37420us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:41.213865 (+ 212us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:41.213952 (+ 87us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:41.308011 (+ 94059us) tablet.cc:701] Row locks acquired
0505 13:59:41.308073 (+ 62us) write_op.cc:260] PREPARE: finished
0505 13:59:41.308668 (+ 595us) write_op.cc:270] Start()
0505 13:59:41.308973 (+ 305us) write_op.cc:276] Timestamp: P: 1777989581308612 usec, L: 0
0505 13:59:41.309010 (+ 37us) op_driver.cc:348] REPLICATION: starting
0505 13:59:41.310362 (+ 1352us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:41.394039 (+ 83677us) op_driver.cc:464] REPLICATION: finished
0505 13:59:41.396565 (+ 2526us) write_op.cc:301] APPLY: starting
0505 13:59:41.396835 (+ 270us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:41.399512 (+ 2677us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:41.416657 (+ 17145us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:41.489530 (+ 72873us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:41.492487 (+ 2957us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:41.492826 (+ 339us) write_op.cc:312] APPLY: finished
0505 13:59:41.495966 (+ 3140us) log.cc:844] Serialized 320 byte log entry
0505 13:59:41.497025 (+ 1059us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:41.569575 (+ 72550us) write_op.cc:454] Released schema lock
0505 13:59:41.569698 (+ 123us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1792,"num_ops":50,"prepare.queue_time_us":677433,"prepare.run_cpu_time_us":17898,"prepare.run_wall_time_us":135841,"raft.queue_time_us":4683,"raft.run_cpu_time_us":2427,"raft.run_wall_time_us":2425,"replication_time_us":84258,"row_lock_wait_count":1,"row_lock_wait_us":56516}]]}
W20260505 13:59:41.982259 7137 debug-util.cc:398] Leaking SignalData structure 0x7b08000401c0 after lost signal to thread 7013
W20260505 13:59:41.986541 7137 debug-util.cc:398] Leaking SignalData structure 0x7b08000837e0 after lost signal to thread 7140
W20260505 13:59:42.288106 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 66265.1us
I20260505 13:59:43.139899 7653 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1673, attempt_no=0}) took 1023 ms. Trace:
I20260505 13:59:43.177587 7653 rpcz_store.cc:276] 0505 13:59:42.115735 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:42.116235 (+ 500us) service_pool.cc:224] Handling call
0505 13:59:43.138793 (+1022558us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:42.647049 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:42.649194 (+ 2145us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:42.649372 (+ 178us) write_op.cc:435] Acquired schema lock
0505 13:59:42.649397 (+ 25us) tablet.cc:662] Decoding operations
0505 13:59:42.661282 (+ 11885us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:42.661328 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:42.661355 (+ 27us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:42.775311 (+113956us) tablet.cc:701] Row locks acquired
0505 13:59:42.775354 (+ 43us) write_op.cc:260] PREPARE: finished
0505 13:59:42.775804 (+ 450us) write_op.cc:270] Start()
0505 13:59:42.776093 (+ 289us) write_op.cc:276] Timestamp: P: 1777989582775761 usec, L: 0
0505 13:59:42.776116 (+ 23us) op_driver.cc:348] REPLICATION: starting
0505 13:59:42.778609 (+ 2493us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:43.023982 (+245373us) spinlock_profiling.cc:243] Waited 5.17 ms on lock 0x7b18000551e0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cd0b9251 00007f45cd0c5ab1 00007f45cb7ee88d 00007f45cb7f0b8a 00007f45cb7f0b18 00007f45cb7f0a98 00007f45cb7f0a53 00007f45cb7efc6c 00007f45cd148d16 00007f45cd1434c8 00007f45cb38b6f0
0505 13:59:43.024203 (+ 221us) op_driver.cc:464] REPLICATION: finished
0505 13:59:43.028849 (+ 4646us) write_op.cc:301] APPLY: starting
0505 13:59:43.029085 (+ 236us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:43.031166 (+ 2081us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:43.031205 (+ 39us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:43.104635 (+ 73430us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:43.106774 (+ 2139us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:43.106926 (+ 152us) write_op.cc:312] APPLY: finished
0505 13:59:43.134253 (+ 27327us) log.cc:844] Serialized 320 byte log entry
0505 13:59:43.136432 (+ 2179us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:43.137105 (+ 673us) write_op.cc:454] Released schema lock
0505 13:59:43.137342 (+ 237us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4015,"mutex_wait_us":46762,"num_ops":50,"prepare.queue_time_us":524888,"prepare.run_cpu_time_us":19868,"prepare.run_wall_time_us":259353,"raft.queue_time_us":76166,"raft.run_cpu_time_us":5263,"raft.run_wall_time_us":43422,"replication_time_us":159269,"row_lock_wait_count":1,"row_lock_wait_us":54640,"spinlock_wait_cycles":11890432,"thread_start_us":41020,"threads_started":1}]]}
I20260505 13:59:43.432802 7681 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1674, attempt_no=0}) took 1233 ms. Trace:
I20260505 13:59:43.438005 7681 rpcz_store.cc:276] 0505 13:59:42.199592 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:42.200127 (+ 535us) service_pool.cc:224] Handling call
0505 13:59:43.432204 (+1232077us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:42.906467 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:42.907082 (+ 615us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:42.907110 (+ 28us) write_op.cc:435] Acquired schema lock
0505 13:59:42.907133 (+ 23us) tablet.cc:662] Decoding operations
0505 13:59:42.910217 (+ 3084us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:42.910288 (+ 71us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:42.910310 (+ 22us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:43.137125 (+226815us) lock_manager.cc:598] Waited 223812us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:43.137311 (+ 186us) tablet.cc:701] Row locks acquired
0505 13:59:43.137465 (+ 154us) write_op.cc:260] PREPARE: finished
0505 13:59:43.137913 (+ 448us) write_op.cc:270] Start()
0505 13:59:43.138448 (+ 535us) write_op.cc:276] Timestamp: P: 1777989583137993 usec, L: 0
0505 13:59:43.138497 (+ 49us) op_driver.cc:348] REPLICATION: starting
0505 13:59:43.141425 (+ 2928us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:43.161237 (+ 19812us) op_driver.cc:464] REPLICATION: finished
0505 13:59:43.211614 (+ 50377us) write_op.cc:301] APPLY: starting
0505 13:59:43.212032 (+ 418us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:43.213466 (+ 1434us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:43.213529 (+ 63us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:43.390977 (+177448us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:43.404540 (+ 13563us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:43.404594 (+ 54us) write_op.cc:312] APPLY: finished
0505 13:59:43.405363 (+ 769us) log.cc:844] Serialized 320 byte log entry
0505 13:59:43.429645 (+ 24282us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:43.429817 (+ 172us) write_op.cc:454] Released schema lock
0505 13:59:43.429943 (+ 126us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":50138,"num_ops":50,"prepare.queue_time_us":700465,"prepare.run_cpu_time_us":12532,"prepare.run_wall_time_us":240474,"raft.queue_time_us":505,"raft.run_cpu_time_us":3566,"raft.run_wall_time_us":3566,"replication_time_us":22595,"row_lock_wait_count":1,"row_lock_wait_us":223812,"thread_start_us":50514,"threads_started":1}]]}
I20260505 13:59:43.503113 7681 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1675, attempt_no=0}) took 1029 ms. Trace:
I20260505 13:59:43.503953 7681 rpcz_store.cc:276] 0505 13:59:42.474227 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:42.474408 (+ 181us) service_pool.cc:224] Handling call
0505 13:59:43.502971 (+1028563us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:43.147295 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:43.148607 (+ 1312us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:43.148659 (+ 52us) write_op.cc:435] Acquired schema lock
0505 13:59:43.148789 (+ 130us) tablet.cc:662] Decoding operations
0505 13:59:43.154272 (+ 5483us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:43.154315 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:43.154336 (+ 21us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:43.429819 (+275483us) lock_manager.cc:598] Waited 265746us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:43.429924 (+ 105us) tablet.cc:701] Row locks acquired
0505 13:59:43.429940 (+ 16us) write_op.cc:260] PREPARE: finished
0505 13:59:43.430264 (+ 324us) write_op.cc:270] Start()
0505 13:59:43.430443 (+ 179us) write_op.cc:276] Timestamp: P: 1777989583430489 usec, L: 0
0505 13:59:43.430467 (+ 24us) op_driver.cc:348] REPLICATION: starting
0505 13:59:43.432545 (+ 2078us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:43.458963 (+ 26418us) op_driver.cc:464] REPLICATION: finished
0505 13:59:43.460441 (+ 1478us) write_op.cc:301] APPLY: starting
0505 13:59:43.460683 (+ 242us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:43.461614 (+ 931us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:43.461743 (+ 129us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:43.484541 (+ 22798us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:43.485297 (+ 756us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:43.485362 (+ 65us) write_op.cc:312] APPLY: finished
0505 13:59:43.486295 (+ 933us) log.cc:844] Serialized 320 byte log entry
0505 13:59:43.487090 (+ 795us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:43.502531 (+ 15441us) write_op.cc:454] Released schema lock
0505 13:59:43.502620 (+ 89us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1266,"num_ops":50,"prepare.queue_time_us":670522,"prepare.run_cpu_time_us":11434,"prepare.run_wall_time_us":285889,"raft.queue_time_us":12509,"raft.run_cpu_time_us":2481,"raft.run_wall_time_us":5320,"replication_time_us":28312,"row_lock_wait_count":1,"row_lock_wait_us":265746}]]}
I20260505 13:59:43.650401 7653 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1676, attempt_no=0}) took 1124 ms. Trace:
I20260505 13:59:43.651659 7653 rpcz_store.cc:276] 0505 13:59:42.527044 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:42.528924 (+ 1880us) service_pool.cc:224] Handling call
0505 13:59:43.650073 (+1121149us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:43.433346 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:43.434157 (+ 811us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:43.434200 (+ 43us) write_op.cc:435] Acquired schema lock
0505 13:59:43.434218 (+ 18us) tablet.cc:662] Decoding operations
0505 13:59:43.447179 (+ 12961us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:43.447223 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:43.447256 (+ 33us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:43.487299 (+ 40043us) tablet.cc:701] Row locks acquired
0505 13:59:43.487317 (+ 18us) write_op.cc:260] PREPARE: finished
0505 13:59:43.487497 (+ 180us) write_op.cc:270] Start()
0505 13:59:43.487579 (+ 82us) write_op.cc:276] Timestamp: P: 1777989583487765 usec, L: 0
0505 13:59:43.487595 (+ 16us) op_driver.cc:348] REPLICATION: starting
0505 13:59:43.489264 (+ 1669us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:43.529405 (+ 40141us) op_driver.cc:464] REPLICATION: finished
0505 13:59:43.530478 (+ 1073us) write_op.cc:301] APPLY: starting
0505 13:59:43.530644 (+ 166us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:43.531512 (+ 868us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:43.531634 (+ 122us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:43.599849 (+ 68215us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:43.602056 (+ 2207us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:43.602378 (+ 322us) write_op.cc:312] APPLY: finished
0505 13:59:43.605963 (+ 3585us) log.cc:844] Serialized 320 byte log entry
0505 13:59:43.607871 (+ 1908us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:43.648383 (+ 40512us) write_op.cc:454] Released schema lock
0505 13:59:43.648643 (+ 260us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":1283840,"child_traces":[["op",{"apply.queue_time_us":569,"num_ops":50,"prepare.queue_time_us":896411,"prepare.run_cpu_time_us":8648,"prepare.run_wall_time_us":56337,"raft.queue_time_us":6422,"raft.run_cpu_time_us":1277,"raft.run_wall_time_us":1275,"replication_time_us":41546,"row_lock_wait_count":1,"row_lock_wait_us":37833}]]}
I20260505 13:59:43.854266 7681 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1677, attempt_no=0}) took 1271 ms. Trace:
I20260505 13:59:43.880360 7681 rpcz_store.cc:276] 0505 13:59:42.583269 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:42.583625 (+ 356us) service_pool.cc:224] Handling call
0505 13:59:43.853906 (+1270281us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:43.489763 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:43.490289 (+ 526us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:43.490312 (+ 23us) write_op.cc:435] Acquired schema lock
0505 13:59:43.490325 (+ 13us) tablet.cc:662] Decoding operations
0505 13:59:43.497721 (+ 7396us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:43.497764 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:43.497786 (+ 22us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:43.608542 (+110756us) lock_manager.cc:598] Waited 108785us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:43.608826 (+ 284us) tablet.cc:701] Row locks acquired
0505 13:59:43.608855 (+ 29us) write_op.cc:260] PREPARE: finished
0505 13:59:43.609455 (+ 600us) write_op.cc:270] Start()
0505 13:59:43.609553 (+ 98us) write_op.cc:276] Timestamp: P: 1777989583609422 usec, L: 0
0505 13:59:43.609569 (+ 16us) op_driver.cc:348] REPLICATION: starting
0505 13:59:43.610385 (+ 816us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:43.649243 (+ 38858us) op_driver.cc:464] REPLICATION: finished
0505 13:59:43.650719 (+ 1476us) write_op.cc:301] APPLY: starting
0505 13:59:43.650991 (+ 272us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:43.654923 (+ 3932us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:43.654961 (+ 38us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:43.765587 (+110626us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:43.768125 (+ 2538us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:43.768261 (+ 136us) write_op.cc:312] APPLY: finished
0505 13:59:43.769449 (+ 1188us) log.cc:844] Serialized 320 byte log entry
0505 13:59:43.770723 (+ 1274us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:43.852294 (+ 81571us) write_op.cc:454] Released schema lock
0505 13:59:43.852684 (+ 390us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":749,"num_ops":50,"prepare.queue_time_us":903119,"prepare.run_cpu_time_us":7366,"prepare.run_wall_time_us":121049,"raft.queue_time_us":2121,"raft.run_cpu_time_us":2746,"raft.run_wall_time_us":2778,"replication_time_us":39124,"row_lock_wait_count":1,"row_lock_wait_us":108785}]]}
I20260505 13:59:44.026077 7653 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1678, attempt_no=0}) took 1298 ms. Trace:
I20260505 13:59:44.029692 7653 rpcz_store.cc:276] 0505 13:59:42.727959 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:42.728581 (+ 622us) service_pool.cc:224] Handling call
0505 13:59:44.025916 (+1297335us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:43.610978 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:43.611599 (+ 621us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:43.611639 (+ 40us) write_op.cc:435] Acquired schema lock
0505 13:59:43.611651 (+ 12us) tablet.cc:662] Decoding operations
0505 13:59:43.677810 (+ 66159us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:43.677890 (+ 80us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:43.677934 (+ 44us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:43.771186 (+ 93252us) tablet.cc:701] Row locks acquired
0505 13:59:43.771216 (+ 30us) write_op.cc:260] PREPARE: finished
0505 13:59:43.771605 (+ 389us) write_op.cc:270] Start()
0505 13:59:43.771730 (+ 125us) write_op.cc:276] Timestamp: P: 1777989583771638 usec, L: 0
0505 13:59:43.771750 (+ 20us) op_driver.cc:348] REPLICATION: starting
0505 13:59:43.772834 (+ 1084us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:43.836574 (+ 63740us) op_driver.cc:464] REPLICATION: finished
0505 13:59:43.839972 (+ 3398us) write_op.cc:301] APPLY: starting
0505 13:59:43.840356 (+ 384us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:43.842611 (+ 2255us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:43.843123 (+ 512us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:43.942808 (+ 99685us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:43.945430 (+ 2622us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:43.945611 (+ 181us) write_op.cc:312] APPLY: finished
0505 13:59:43.948560 (+ 2949us) log.cc:844] Serialized 320 byte log entry
0505 13:59:43.951804 (+ 3244us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:44.025102 (+ 73298us) write_op.cc:454] Released schema lock
0505 13:59:44.025300 (+ 198us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2679,"num_ops":50,"prepare.queue_time_us":878444,"prepare.run_cpu_time_us":14984,"prepare.run_wall_time_us":162342,"raft.queue_time_us":11144,"raft.run_cpu_time_us":8738,"raft.run_wall_time_us":71227,"replication_time_us":64476,"row_lock_wait_count":1,"row_lock_wait_us":85591}]]}
I20260505 13:59:44.052726 7681 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1679, attempt_no=0}) took 1195 ms. Trace:
I20260505 13:59:44.053416 7681 rpcz_store.cc:276] 0505 13:59:42.858875 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:42.859396 (+ 521us) service_pool.cc:224] Handling call
0505 13:59:44.052604 (+1193208us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:43.773524 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:43.774262 (+ 738us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:43.774292 (+ 30us) write_op.cc:435] Acquired schema lock
0505 13:59:43.774318 (+ 26us) tablet.cc:662] Decoding operations
0505 13:59:43.819249 (+ 44931us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:43.819293 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:43.819317 (+ 24us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:43.952048 (+132731us) tablet.cc:701] Row locks acquired
0505 13:59:43.952085 (+ 37us) write_op.cc:260] PREPARE: finished
0505 13:59:43.952419 (+ 334us) write_op.cc:270] Start()
0505 13:59:43.952655 (+ 236us) write_op.cc:276] Timestamp: P: 1777989583952503 usec, L: 0
0505 13:59:43.952672 (+ 17us) op_driver.cc:348] REPLICATION: starting
0505 13:59:43.954319 (+ 1647us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:43.999813 (+ 45494us) op_driver.cc:464] REPLICATION: finished
0505 13:59:44.001641 (+ 1828us) write_op.cc:301] APPLY: starting
0505 13:59:44.001751 (+ 110us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:44.002835 (+ 1084us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:44.003035 (+ 200us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:44.037868 (+ 34833us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:44.039026 (+ 1158us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:44.039111 (+ 85us) write_op.cc:312] APPLY: finished
0505 13:59:44.040649 (+ 1538us) log.cc:844] Serialized 320 byte log entry
0505 13:59:44.042650 (+ 2001us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:44.051839 (+ 9189us) write_op.cc:454] Released schema lock
0505 13:59:44.052065 (+ 226us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1554,"num_ops":50,"prepare.queue_time_us":906605,"prepare.run_cpu_time_us":17085,"prepare.run_wall_time_us":181411,"raft.queue_time_us":10259,"raft.run_cpu_time_us":7557,"raft.run_wall_time_us":28954,"replication_time_us":46941,"row_lock_wait_count":1,"row_lock_wait_us":39079}]]}
I20260505 13:59:44.094220 7681 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1680, attempt_no=0}) took 1132 ms. Trace:
I20260505 13:59:44.095474 7681 rpcz_store.cc:276] 0505 13:59:42.962252 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:42.963557 (+ 1305us) service_pool.cc:224] Handling call
0505 13:59:44.094072 (+1130515us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:43.955573 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:43.982578 (+ 27005us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:43.982629 (+ 51us) write_op.cc:435] Acquired schema lock
0505 13:59:43.982661 (+ 32us) tablet.cc:662] Decoding operations
0505 13:59:44.022456 (+ 39795us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:44.022513 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:44.022548 (+ 35us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:44.042876 (+ 20328us) tablet.cc:701] Row locks acquired
0505 13:59:44.042896 (+ 20us) write_op.cc:260] PREPARE: finished
0505 13:59:44.043111 (+ 215us) write_op.cc:270] Start()
0505 13:59:44.043247 (+ 136us) write_op.cc:276] Timestamp: P: 1777989584043076 usec, L: 0
0505 13:59:44.043265 (+ 18us) op_driver.cc:348] REPLICATION: starting
0505 13:59:44.051596 (+ 8331us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:44.073915 (+ 22319us) op_driver.cc:464] REPLICATION: finished
0505 13:59:44.074538 (+ 623us) write_op.cc:301] APPLY: starting
0505 13:59:44.074665 (+ 127us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:44.075660 (+ 995us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:44.075831 (+ 171us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:44.087991 (+ 12160us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:44.089307 (+ 1316us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:44.089394 (+ 87us) write_op.cc:312] APPLY: finished
0505 13:59:44.092119 (+ 2725us) log.cc:844] Serialized 320 byte log entry
0505 13:59:44.092540 (+ 421us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:44.092840 (+ 300us) write_op.cc:454] Released schema lock
0505 13:59:44.093056 (+ 216us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":306,"num_ops":50,"prepare.queue_time_us":985121,"prepare.run_cpu_time_us":11247,"prepare.run_wall_time_us":110924,"raft.queue_time_us":5028,"raft.run_cpu_time_us":2621,"raft.run_wall_time_us":2631,"replication_time_us":30423,"row_lock_wait_count":1,"row_lock_wait_us":17822}]]}
W20260505 13:59:44.296097 7331 meta_cache.cc:1261] Time spent looking up entry by key: real 0.094s user 0.001s sys 0.000s
I20260505 13:59:44.946512 6808 maintenance_manager.cc:419] P 972d2d5e1f784c78bc1f3799a13bc74f: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033353
I20260505 13:59:45.497735 6807 maintenance_manager.cc:643] P 972d2d5e1f784c78bc1f3799a13bc74f: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.545s user 0.114s sys 0.090s Metrics: {"bytes_written":6625,"cfile_init":1,"dirs.queue_time_us":5281,"dirs.run_cpu_time_us":486,"dirs.run_wall_time_us":12650,"drs_written":1,"lbm_read_time_us":607,"lbm_reads_lt_1ms":4,"lbm_write_time_us":13499,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":25,"peak_mem_usage":0,"rows_written":5,"thread_start_us":29218,"threads_started":2,"wal-append.queue_time_us":22400}
I20260505 13:59:45.534833 6808 maintenance_manager.cc:419] P 972d2d5e1f784c78bc1f3799a13bc74f: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 552 bytes on disk
I20260505 13:59:45.557512 6807 maintenance_manager.cc:643] P 972d2d5e1f784c78bc1f3799a13bc74f: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.002s sys 0.002s Metrics: {"cfile_init":1,"lbm_read_time_us":3599,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20260505 13:59:45.841658 7690 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1692, attempt_no=0}) took 1016 ms. Trace:
I20260505 13:59:45.854707 7690 rpcz_store.cc:276] 0505 13:59:44.824189 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:44.824412 (+ 223us) service_pool.cc:224] Handling call
0505 13:59:45.840358 (+1015946us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:45.479338 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:45.479819 (+ 481us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:45.479844 (+ 25us) write_op.cc:435] Acquired schema lock
0505 13:59:45.479857 (+ 13us) tablet.cc:662] Decoding operations
0505 13:59:45.482707 (+ 2850us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:45.482749 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:45.482774 (+ 25us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:45.635528 (+152754us) tablet.cc:701] Row locks acquired
0505 13:59:45.635569 (+ 41us) write_op.cc:260] PREPARE: finished
0505 13:59:45.636114 (+ 545us) write_op.cc:270] Start()
0505 13:59:45.636473 (+ 359us) write_op.cc:276] Timestamp: P: 1777989585636044 usec, L: 0
0505 13:59:45.636502 (+ 29us) op_driver.cc:348] REPLICATION: starting
0505 13:59:45.638303 (+ 1801us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:45.705938 (+ 67635us) op_driver.cc:464] REPLICATION: finished
0505 13:59:45.724828 (+ 18890us) write_op.cc:301] APPLY: starting
0505 13:59:45.724936 (+ 108us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:45.725837 (+ 901us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:45.725967 (+ 130us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:45.801355 (+ 75388us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:45.820631 (+ 19276us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:45.820758 (+ 127us) write_op.cc:312] APPLY: finished
0505 13:59:45.822965 (+ 2207us) log.cc:844] Serialized 320 byte log entry
0505 13:59:45.838224 (+ 15259us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:45.838712 (+ 488us) write_op.cc:454] Released schema lock
0505 13:59:45.838914 (+ 202us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":18686,"mutex_wait_us":19140,"num_ops":50,"prepare.queue_time_us":652819,"prepare.run_cpu_time_us":10568,"prepare.run_wall_time_us":167658,"raft.queue_time_us":11009,"raft.run_cpu_time_us":5676,"raft.run_wall_time_us":57184,"replication_time_us":69001,"row_lock_wait_count":1,"row_lock_wait_us":69355}]]}
I20260505 13:59:45.953967 7686 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1691, attempt_no=0}) took 1124 ms. Trace:
I20260505 13:59:45.954831 7686 rpcz_store.cc:276] 0505 13:59:44.829445 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:44.829675 (+ 230us) service_pool.cc:224] Handling call
0505 13:59:45.953415 (+1123740us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:45.671408 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:45.672693 (+ 1285us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:45.672862 (+ 169us) write_op.cc:435] Acquired schema lock
0505 13:59:45.672902 (+ 40us) tablet.cc:662] Decoding operations
0505 13:59:45.728389 (+ 55487us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:45.728434 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:45.728456 (+ 22us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:45.838781 (+110325us) tablet.cc:701] Row locks acquired
0505 13:59:45.838842 (+ 61us) write_op.cc:260] PREPARE: finished
0505 13:59:45.839558 (+ 716us) write_op.cc:270] Start()
0505 13:59:45.839900 (+ 342us) write_op.cc:276] Timestamp: P: 1777989585839570 usec, L: 0
0505 13:59:45.839944 (+ 44us) op_driver.cc:348] REPLICATION: starting
0505 13:59:45.842282 (+ 2338us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:45.883057 (+ 40775us) op_driver.cc:464] REPLICATION: finished
0505 13:59:45.883650 (+ 593us) write_op.cc:301] APPLY: starting
0505 13:59:45.883750 (+ 100us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:45.884742 (+ 992us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:45.884887 (+ 145us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:45.945908 (+ 61021us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:45.946567 (+ 659us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:45.946640 (+ 73us) write_op.cc:312] APPLY: finished
0505 13:59:45.947709 (+ 1069us) log.cc:844] Serialized 320 byte log entry
0505 13:59:45.952461 (+ 4752us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:45.952681 (+ 220us) write_op.cc:454] Released schema lock
0505 13:59:45.952804 (+ 123us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":387,"num_ops":50,"prepare.queue_time_us":815781,"prepare.run_cpu_time_us":17211,"prepare.run_wall_time_us":201503,"raft.queue_time_us":2954,"raft.run_cpu_time_us":5417,"raft.run_wall_time_us":5415,"replication_time_us":43015,"row_lock_wait_count":1,"row_lock_wait_us":79992,"spinlock_wait_cycles":1127808}]]}
I20260505 13:59:46.367749 7686 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1695, attempt_no=0}) took 1115 ms. Trace:
I20260505 13:59:46.375535 7686 rpcz_store.cc:276] 0505 13:59:45.252702 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:45.252868 (+ 166us) service_pool.cc:224] Handling call
0505 13:59:46.367391 (+1114523us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:46.080883 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:46.082351 (+ 1468us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:46.082423 (+ 72us) write_op.cc:435] Acquired schema lock
0505 13:59:46.082457 (+ 34us) tablet.cc:662] Decoding operations
0505 13:59:46.086397 (+ 3940us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:46.086453 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:46.086489 (+ 36us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:46.200770 (+114281us) lock_manager.cc:598] Waited 112368us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:46.200935 (+ 165us) tablet.cc:701] Row locks acquired
0505 13:59:46.200986 (+ 51us) write_op.cc:260] PREPARE: finished
0505 13:59:46.201996 (+ 1010us) write_op.cc:270] Start()
0505 13:59:46.202262 (+ 266us) write_op.cc:276] Timestamp: P: 1777989586201866 usec, L: 0
0505 13:59:46.202407 (+ 145us) op_driver.cc:348] REPLICATION: starting
0505 13:59:46.207729 (+ 5322us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:46.281942 (+ 74213us) op_driver.cc:464] REPLICATION: finished
0505 13:59:46.283684 (+ 1742us) write_op.cc:301] APPLY: starting
0505 13:59:46.284380 (+ 696us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:46.286962 (+ 2582us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:46.286994 (+ 32us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:46.355644 (+ 68650us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:46.359456 (+ 3812us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:46.359524 (+ 68us) write_op.cc:312] APPLY: finished
0505 13:59:46.362744 (+ 3220us) log.cc:844] Serialized 320 byte log entry
0505 13:59:46.365185 (+ 2441us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:46.365744 (+ 559us) write_op.cc:454] Released schema lock
0505 13:59:46.366092 (+ 348us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":849,"mutex_wait_us":2909,"num_ops":50,"prepare.queue_time_us":824578,"prepare.run_cpu_time_us":13371,"prepare.run_wall_time_us":129640,"raft.queue_time_us":9959,"raft.run_cpu_time_us":4058,"raft.run_wall_time_us":4076,"replication_time_us":78900,"row_lock_wait_count":1,"row_lock_wait_us":112368}]]}
I20260505 13:59:46.591080 7686 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1696, attempt_no=0}) took 1085 ms. Trace:
I20260505 13:59:46.592020 7686 rpcz_store.cc:276] 0505 13:59:45.505704 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:45.506433 (+ 729us) service_pool.cc:224] Handling call
0505 13:59:46.590948 (+1084515us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:46.210815 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:46.213117 (+ 2302us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:46.213263 (+ 146us) write_op.cc:435] Acquired schema lock
0505 13:59:46.213290 (+ 27us) tablet.cc:662] Decoding operations
0505 13:59:46.241156 (+ 27866us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:46.241247 (+ 91us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:46.241278 (+ 31us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:46.365549 (+124271us) lock_manager.cc:598] Waited 120447us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:46.365695 (+ 146us) tablet.cc:701] Row locks acquired
0505 13:59:46.365714 (+ 19us) write_op.cc:260] PREPARE: finished
0505 13:59:46.366055 (+ 341us) write_op.cc:270] Start()
0505 13:59:46.366196 (+ 141us) write_op.cc:276] Timestamp: P: 1777989586366064 usec, L: 0
0505 13:59:46.366214 (+ 18us) op_driver.cc:348] REPLICATION: starting
0505 13:59:46.367759 (+ 1545us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:46.388458 (+ 20699us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b1800015fa0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cd0b9251 00007f45cd0c5ab1 00007f45c42c2a33 00007f45c42c4e01 00007f45c42c4db9 00007f45c42c4d49 00007f45c42c4d11 00007f45c42c400d 00000000005b3bc4 00000000005b39f9 00007f45c42a5fba
0505 13:59:46.501982 (+113524us) op_driver.cc:464] REPLICATION: finished
0505 13:59:46.503129 (+ 1147us) write_op.cc:301] APPLY: starting
0505 13:59:46.503323 (+ 194us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:46.506424 (+ 3101us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:46.506595 (+ 171us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:46.565807 (+ 59212us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:46.567717 (+ 1910us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:46.568095 (+ 378us) write_op.cc:312] APPLY: finished
0505 13:59:46.576871 (+ 8776us) log.cc:844] Serialized 320 byte log entry
0505 13:59:46.590200 (+ 13329us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:46.590427 (+ 227us) write_op.cc:454] Released schema lock
0505 13:59:46.590549 (+ 122us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":412,"num_ops":50,"prepare.queue_time_us":659872,"prepare.run_cpu_time_us":15411,"prepare.run_wall_time_us":157605,"raft.queue_time_us":4286,"raft.run_cpu_time_us":7777,"raft.run_wall_time_us":63163,"replication_time_us":135142,"row_lock_wait_count":1,"row_lock_wait_us":120447,"spinlock_wait_cycles":28908928}]]}
W20260505 13:59:47.682549 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 154579us
W20260505 13:59:47.771613 6899 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.249.3:43967, user_credentials={real_user=slave}} blocked reactor thread for 50861.8us
I20260505 13:59:47.805469 7686 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1705, attempt_no=0}) took 1064 ms. Trace:
W20260505 13:59:47.833770 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 109055us
W20260505 13:59:47.880178 6899 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.249.2:41615, user_credentials={real_user=slave}} blocked reactor thread for 53718.3us
I20260505 13:59:47.844388 7686 rpcz_store.cc:276] 0505 13:59:46.739960 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:46.740229 (+ 269us) service_pool.cc:224] Handling call
0505 13:59:47.803672 (+1063443us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:47.373871 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:47.374913 (+ 1042us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:47.375101 (+ 188us) write_op.cc:435] Acquired schema lock
0505 13:59:47.375242 (+ 141us) tablet.cc:662] Decoding operations
0505 13:59:47.425188 (+ 49946us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:47.425407 (+ 219us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:47.425640 (+ 233us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:47.608959 (+183319us) lock_manager.cc:598] Waited 143567us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:47.610519 (+ 1560us) tablet.cc:701] Row locks acquired
0505 13:59:47.610703 (+ 184us) write_op.cc:260] PREPARE: finished
0505 13:59:47.613061 (+ 2358us) write_op.cc:270] Start()
0505 13:59:47.613709 (+ 648us) write_op.cc:276] Timestamp: P: 1777989587612571 usec, L: 0
0505 13:59:47.613948 (+ 239us) op_driver.cc:348] REPLICATION: starting
0505 13:59:47.630455 (+ 16507us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:47.669819 (+ 39364us) op_driver.cc:464] REPLICATION: finished
0505 13:59:47.673669 (+ 3850us) write_op.cc:301] APPLY: starting
0505 13:59:47.674669 (+ 1000us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:47.679760 (+ 5091us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:47.679933 (+ 173us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:47.751281 (+ 71348us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:47.752331 (+ 1050us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:47.752395 (+ 64us) write_op.cc:312] APPLY: finished
0505 13:59:47.764953 (+ 12558us) log.cc:844] Serialized 320 byte log entry
0505 13:59:47.766020 (+ 1067us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:47.766472 (+ 452us) write_op.cc:454] Released schema lock
0505 13:59:47.767068 (+ 596us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2374,"num_ops":50,"prepare.queue_time_us":631508,"prepare.run_cpu_time_us":27295,"prepare.run_wall_time_us":282187,"raft.queue_time_us":5020,"raft.run_cpu_time_us":8035,"raft.run_wall_time_us":58470,"replication_time_us":55076,"row_lock_wait_count":1,"row_lock_wait_us":143567}]]}
I20260505 13:59:48.234200 7699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1706, attempt_no=0}) took 1257 ms. Trace:
I20260505 13:59:48.283504 7686 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1707, attempt_no=0}) took 1101 ms. Trace:
I20260505 13:59:48.288437 7686 rpcz_store.cc:276] 0505 13:59:47.182358 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:47.182864 (+ 506us) service_pool.cc:224] Handling call
0505 13:59:48.283090 (+1100226us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:47.906981 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:47.907639 (+ 658us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:47.907714 (+ 75us) write_op.cc:435] Acquired schema lock
0505 13:59:47.907727 (+ 13us) tablet.cc:662] Decoding operations
0505 13:59:47.910989 (+ 3262us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:47.911074 (+ 85us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:47.911132 (+ 58us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:48.070965 (+159833us) tablet.cc:701] Row locks acquired
0505 13:59:48.071045 (+ 80us) write_op.cc:260] PREPARE: finished
0505 13:59:48.071643 (+ 598us) write_op.cc:270] Start()
0505 13:59:48.071771 (+ 128us) write_op.cc:276] Timestamp: P: 1777989588071995 usec, L: 0
0505 13:59:48.071788 (+ 17us) op_driver.cc:348] REPLICATION: starting
0505 13:59:48.074829 (+ 3041us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:48.100345 (+ 25516us) spinlock_profiling.cc:243] Waited 8.47 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2f8522 00007f45cb2d928a 00007f45cb2e0133 00007f45cb2e00b9 00007f45cb2e0049 00007f45cb2e0011 00007f45cb2df1dd 00000000005b3bc4 00000000005b39f9 00007f45c42c2ab5
0505 13:59:48.140521 (+ 40176us) op_driver.cc:464] REPLICATION: finished
0505 13:59:48.141107 (+ 586us) write_op.cc:301] APPLY: starting
0505 13:59:48.141224 (+ 117us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:48.142888 (+ 1664us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:48.142954 (+ 66us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:48.198121 (+ 55167us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:48.199978 (+ 1857us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:48.200069 (+ 91us) write_op.cc:312] APPLY: finished
0505 13:59:48.247542 (+ 47473us) log.cc:844] Serialized 320 byte log entry
0505 13:59:48.248447 (+ 905us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:48.280636 (+ 32189us) write_op.cc:454] Released schema lock
0505 13:59:48.280911 (+ 275us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":149,"num_ops":50,"prepare.queue_time_us":721643,"prepare.run_cpu_time_us":16749,"prepare.run_wall_time_us":168656,"raft.queue_time_us":4793,"raft.run_cpu_time_us":12174,"raft.run_wall_time_us":74182,"replication_time_us":68522,"row_lock_wait_count":1,"row_lock_wait_us":61004,"spinlock_wait_cycles":19488640}]]}
I20260505 13:59:48.250304 7699 rpcz_store.cc:276] 0505 13:59:46.975186 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:46.975434 (+ 248us) service_pool.cc:224] Handling call
0505 13:59:48.231787 (+1256353us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:47.656356 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:47.658716 (+ 2360us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:47.658854 (+ 138us) write_op.cc:435] Acquired schema lock
0505 13:59:47.659045 (+ 191us) tablet.cc:662] Decoding operations
0505 13:59:47.753559 (+ 94514us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:47.759134 (+ 5575us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:47.759424 (+ 290us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:47.802208 (+ 42784us) tablet.cc:701] Row locks acquired
0505 13:59:47.802238 (+ 30us) write_op.cc:260] PREPARE: finished
0505 13:59:47.822425 (+ 20187us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b34000016d8. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45c8feef61 00007f45c902d2f4 00007f45cb7ed625 00007f45cb7ed1c8 00007f45cb7f1e61 00007f45cb7f1e19 00007f45cb7f1da9 00007f45cb7f1d71 00007f45cb7f0f9d 00000000005b3bc4 00000000005b39f9
0505 13:59:47.822628 (+ 203us) write_op.cc:270] Start()
0505 13:59:47.822892 (+ 264us) write_op.cc:276] Timestamp: P: 1777989587822887 usec, L: 0
0505 13:59:47.822918 (+ 26us) op_driver.cc:348] REPLICATION: starting
0505 13:59:47.823951 (+ 1033us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:47.919003 (+ 95052us) op_driver.cc:464] REPLICATION: finished
0505 13:59:47.920086 (+ 1083us) write_op.cc:301] APPLY: starting
0505 13:59:47.920415 (+ 329us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:47.922787 (+ 2372us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:47.922816 (+ 29us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:48.064392 (+141576us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:48.065751 (+ 1359us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:48.065826 (+ 75us) write_op.cc:312] APPLY: finished
0505 13:59:48.069631 (+ 3805us) log.cc:844] Serialized 320 byte log entry
0505 13:59:48.070293 (+ 662us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:48.228386 (+158093us) spinlock_profiling.cc:243] Waited 58.4 ms on lock 0x7b0c000c9f90. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb721671 00007f45cb862bb0 00007f45cb863480 00007f45cb8632fe 00007f45cb804bc0 00007f45cb803dc8 00007f45cb8037cc 00007f45cb7ef7f9 00007f45cb7eefc3 00007f45cb7f31d1 00007f45cb7f3189
0505 13:59:48.228632 (+ 246us) write_op.cc:454] Released schema lock
0505 13:59:48.229120 (+ 488us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":389,"num_ops":50,"prepare.queue_time_us":675458,"prepare.run_cpu_time_us":24216,"prepare.run_wall_time_us":250402,"raft.queue_time_us":56256,"raft.run_cpu_time_us":4100,"raft.run_wall_time_us":23983,"replication_time_us":95588,"spinlock_wait_cycles":170170496}]]}
I20260505 13:59:48.377895 7703 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1708, attempt_no=0}) took 1139 ms. Trace:
I20260505 13:59:48.378630 7703 rpcz_store.cc:276] 0505 13:59:47.238930 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:47.239690 (+ 760us) service_pool.cc:224] Handling call
0505 13:59:48.376963 (+1137273us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:48.076310 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:48.077653 (+ 1343us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:48.077925 (+ 272us) write_op.cc:435] Acquired schema lock
0505 13:59:48.077998 (+ 73us) tablet.cc:662] Decoding operations
0505 13:59:48.142878 (+ 64880us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:48.142981 (+ 103us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:48.143141 (+ 160us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:48.248836 (+105695us) tablet.cc:701] Row locks acquired
0505 13:59:48.248924 (+ 88us) write_op.cc:260] PREPARE: finished
0505 13:59:48.250021 (+ 1097us) write_op.cc:270] Start()
0505 13:59:48.250257 (+ 236us) write_op.cc:276] Timestamp: P: 1777989588249909 usec, L: 0
0505 13:59:48.250297 (+ 40us) op_driver.cc:348] REPLICATION: starting
0505 13:59:48.253047 (+ 2750us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:48.309537 (+ 56490us) op_driver.cc:464] REPLICATION: finished
0505 13:59:48.314352 (+ 4815us) write_op.cc:301] APPLY: starting
0505 13:59:48.314457 (+ 105us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:48.315593 (+ 1136us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:48.315868 (+ 275us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:48.372687 (+ 56819us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:48.373899 (+ 1212us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:48.374019 (+ 120us) write_op.cc:312] APPLY: finished
0505 13:59:48.375611 (+ 1592us) log.cc:844] Serialized 320 byte log entry
0505 13:59:48.375948 (+ 337us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:48.376252 (+ 304us) write_op.cc:454] Released schema lock
0505 13:59:48.376402 (+ 150us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4573,"num_ops":50,"prepare.queue_time_us":829278,"prepare.run_cpu_time_us":24582,"prepare.run_wall_time_us":183028,"raft.queue_time_us":5532,"raft.run_cpu_time_us":7287,"raft.run_wall_time_us":46926,"replication_time_us":59066,"row_lock_wait_count":1,"row_lock_wait_us":32725,"spinlock_wait_cycles":2816,"thread_start_us":4080,"threads_started":1}]]}
W20260505 13:59:48.540673 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 102138us
W20260505 13:59:49.349113 7332 meta_cache.cc:1261] Time spent looking up entry by key: real 0.084s user 0.004s sys 0.000s
I20260505 13:59:49.809857 7699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1717, attempt_no=0}) took 1158 ms. Trace:
I20260505 13:59:49.815248 7699 rpcz_store.cc:276] 0505 13:59:48.650326 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:48.652338 (+ 2012us) service_pool.cc:224] Handling call
0505 13:59:49.808419 (+1156081us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:49.209906 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:49.210513 (+ 607us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:49.210569 (+ 56us) write_op.cc:435] Acquired schema lock
0505 13:59:49.210582 (+ 13us) tablet.cc:662] Decoding operations
0505 13:59:49.222171 (+ 11589us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:49.222498 (+ 327us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:49.222533 (+ 35us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:49.448120 (+225587us) lock_manager.cc:598] Waited 211939us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:49.448606 (+ 486us) tablet.cc:701] Row locks acquired
0505 13:59:49.448771 (+ 165us) write_op.cc:260] PREPARE: finished
0505 13:59:49.449559 (+ 788us) write_op.cc:270] Start()
0505 13:59:49.450007 (+ 448us) write_op.cc:276] Timestamp: P: 1777989589449506 usec, L: 0
0505 13:59:49.450108 (+ 101us) op_driver.cc:348] REPLICATION: starting
0505 13:59:49.452641 (+ 2533us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:49.603188 (+150547us) op_driver.cc:464] REPLICATION: finished
0505 13:59:49.604765 (+ 1577us) write_op.cc:301] APPLY: starting
0505 13:59:49.605340 (+ 575us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:49.608591 (+ 3251us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:49.608698 (+ 107us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:49.697486 (+ 88788us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:49.699474 (+ 1988us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:49.699580 (+ 106us) write_op.cc:312] APPLY: finished
0505 13:59:49.703750 (+ 4170us) log.cc:844] Serialized 320 byte log entry
0505 13:59:49.704694 (+ 944us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:49.788145 (+ 83451us) write_op.cc:454] Released schema lock
0505 13:59:49.788775 (+ 630us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1056,"num_ops":50,"prepare.queue_time_us":553199,"prepare.run_cpu_time_us":20594,"prepare.run_wall_time_us":287345,"raft.queue_time_us":4592,"raft.run_cpu_time_us":10569,"raft.run_wall_time_us":109372,"replication_time_us":152733,"row_lock_wait_count":1,"row_lock_wait_us":211939}]]}
I20260505 13:59:50.046133 7708 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1719, attempt_no=0}) took 1035 ms. Trace:
I20260505 13:59:50.046926 7708 rpcz_store.cc:276] 0505 13:59:49.004732 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:49.004967 (+ 235us) service_pool.cc:224] Handling call
0505 13:59:50.039698 (+1034731us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:49.707980 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:49.708715 (+ 735us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:49.708763 (+ 48us) write_op.cc:435] Acquired schema lock
0505 13:59:49.708777 (+ 14us) tablet.cc:662] Decoding operations
0505 13:59:49.755328 (+ 46551us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:49.755397 (+ 69us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:49.755440 (+ 43us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:49.861015 (+105575us) tablet.cc:701] Row locks acquired
0505 13:59:49.861033 (+ 18us) write_op.cc:260] PREPARE: finished
0505 13:59:49.861285 (+ 252us) write_op.cc:270] Start()
0505 13:59:49.861429 (+ 144us) write_op.cc:276] Timestamp: P: 1777989589861357 usec, L: 0
0505 13:59:49.861444 (+ 15us) op_driver.cc:348] REPLICATION: starting
0505 13:59:49.862199 (+ 755us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:49.895934 (+ 33735us) op_driver.cc:464] REPLICATION: finished
0505 13:59:49.896428 (+ 494us) write_op.cc:301] APPLY: starting
0505 13:59:49.896504 (+ 76us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:49.897258 (+ 754us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:49.897365 (+ 107us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:49.928683 (+ 31318us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:49.932966 (+ 4283us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:49.933052 (+ 86us) write_op.cc:312] APPLY: finished
0505 13:59:49.934368 (+ 1316us) log.cc:844] Serialized 320 byte log entry
0505 13:59:49.937451 (+ 3083us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:50.038082 (+100631us) write_op.cc:454] Released schema lock
0505 13:59:50.038745 (+ 663us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":286,"num_ops":50,"prepare.queue_time_us":700496,"prepare.run_cpu_time_us":12521,"prepare.run_wall_time_us":154711,"raft.queue_time_us":4963,"raft.run_cpu_time_us":2103,"raft.run_wall_time_us":2102,"replication_time_us":34313,"row_lock_wait_count":1,"row_lock_wait_us":47887}]]}
I20260505 13:59:50.139830 7699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1720, attempt_no=0}) took 1030 ms. Trace:
I20260505 13:59:50.141673 7699 rpcz_store.cc:276] 0505 13:59:49.109651 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:49.109887 (+ 236us) service_pool.cc:224] Handling call
0505 13:59:50.139411 (+1029524us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:49.862932 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:49.863515 (+ 583us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:49.863541 (+ 26us) write_op.cc:435] Acquired schema lock
0505 13:59:49.863554 (+ 13us) tablet.cc:662] Decoding operations
0505 13:59:49.902932 (+ 39378us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:49.902982 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:49.903009 (+ 27us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:49.937644 (+ 34635us) tablet.cc:701] Row locks acquired
0505 13:59:49.937661 (+ 17us) write_op.cc:260] PREPARE: finished
0505 13:59:49.937830 (+ 169us) write_op.cc:270] Start()
0505 13:59:49.937917 (+ 87us) write_op.cc:276] Timestamp: P: 1777989589937937 usec, L: 0
0505 13:59:49.937932 (+ 15us) op_driver.cc:348] REPLICATION: starting
0505 13:59:49.938784 (+ 852us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:49.984127 (+ 45343us) op_driver.cc:464] REPLICATION: finished
0505 13:59:49.991533 (+ 7406us) write_op.cc:301] APPLY: starting
0505 13:59:49.991844 (+ 311us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:49.993849 (+ 2005us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:49.994125 (+ 276us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:50.070781 (+ 76656us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:50.074340 (+ 3559us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:50.074662 (+ 322us) write_op.cc:312] APPLY: finished
0505 13:59:50.082231 (+ 7569us) log.cc:844] Serialized 320 byte log entry
0505 13:59:50.092021 (+ 9790us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:50.136506 (+ 44485us) write_op.cc:454] Released schema lock
0505 13:59:50.138226 (+ 1720us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":107,"num_ops":50,"prepare.queue_time_us":747511,"prepare.run_cpu_time_us":7283,"prepare.run_wall_time_us":129292,"raft.queue_time_us":338,"raft.run_cpu_time_us":5467,"raft.run_wall_time_us":36672,"replication_time_us":45635,"row_lock_wait_count":1,"row_lock_wait_us":15617}]]}
W20260505 13:59:50.580260 7337 meta_cache.cc:1261] Time spent looking up entry by key: real 0.075s user 0.004s sys 0.000s
I20260505 13:59:52.130600 7699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1734, attempt_no=0}) took 1031 ms. Trace:
I20260505 13:59:52.165921 7699 rpcz_store.cc:276] 0505 13:59:51.098902 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:51.099600 (+ 698us) service_pool.cc:224] Handling call
0505 13:59:52.128726 (+1029126us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:51.856695 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:51.858325 (+ 1630us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:51.858393 (+ 68us) write_op.cc:435] Acquired schema lock
0505 13:59:51.858426 (+ 33us) tablet.cc:662] Decoding operations
0505 13:59:51.865954 (+ 7528us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:51.866038 (+ 84us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:51.866083 (+ 45us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:51.979072 (+112989us) tablet.cc:701] Row locks acquired
0505 13:59:51.979121 (+ 49us) write_op.cc:260] PREPARE: finished
0505 13:59:51.979856 (+ 735us) write_op.cc:270] Start()
0505 13:59:51.980508 (+ 652us) write_op.cc:276] Timestamp: P: 1777989591979898 usec, L: 0
0505 13:59:51.980561 (+ 53us) op_driver.cc:348] REPLICATION: starting
0505 13:59:51.982674 (+ 2113us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:52.044491 (+ 61817us) op_driver.cc:464] REPLICATION: finished
0505 13:59:52.045987 (+ 1496us) write_op.cc:301] APPLY: starting
0505 13:59:52.046398 (+ 411us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:52.048640 (+ 2242us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:52.049056 (+ 416us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:52.124946 (+ 75890us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:52.125771 (+ 825us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:52.125876 (+ 105us) write_op.cc:312] APPLY: finished
0505 13:59:52.127487 (+ 1611us) log.cc:844] Serialized 320 byte log entry
0505 13:59:52.127831 (+ 344us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:52.128036 (+ 205us) write_op.cc:454] Released schema lock
0505 13:59:52.128230 (+ 194us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":980,"mutex_wait_us":10597,"num_ops":50,"prepare.queue_time_us":749413,"prepare.run_cpu_time_us":20011,"prepare.run_wall_time_us":167930,"raft.queue_time_us":29830,"raft.run_cpu_time_us":5753,"raft.run_wall_time_us":33365,"replication_time_us":63425,"row_lock_wait_count":1,"row_lock_wait_us":87093}]]}
W20260505 13:59:52.180945 7332 meta_cache.cc:1261] Time spent looking up entry by key: real 0.109s user 0.002s sys 0.000s
W20260505 13:59:52.397661 7331 meta_cache.cc:1261] Time spent looking up entry by key: real 0.060s user 0.004s sys 0.000s
I20260505 13:59:52.540234 7716 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1735, attempt_no=0}) took 1080 ms. Trace:
I20260505 13:59:52.561717 7716 rpcz_store.cc:276] 0505 13:59:51.458964 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:51.459217 (+ 253us) service_pool.cc:224] Handling call
0505 13:59:52.538326 (+1079109us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:52.131048 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:52.131578 (+ 530us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:52.131604 (+ 26us) write_op.cc:435] Acquired schema lock
0505 13:59:52.131617 (+ 13us) tablet.cc:662] Decoding operations
0505 13:59:52.134694 (+ 3077us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:52.134734 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:52.134766 (+ 32us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:52.232864 (+ 98098us) tablet.cc:701] Row locks acquired
0505 13:59:52.233124 (+ 260us) write_op.cc:260] PREPARE: finished
0505 13:59:52.233749 (+ 625us) write_op.cc:270] Start()
0505 13:59:52.234183 (+ 434us) write_op.cc:276] Timestamp: P: 1777989592233576 usec, L: 0
0505 13:59:52.234228 (+ 45us) op_driver.cc:348] REPLICATION: starting
0505 13:59:52.241520 (+ 7292us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:52.363152 (+121632us) op_driver.cc:464] REPLICATION: finished
0505 13:59:52.372840 (+ 9688us) write_op.cc:301] APPLY: starting
0505 13:59:52.373108 (+ 268us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:52.395737 (+ 22629us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:52.396274 (+ 537us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:52.507686 (+111412us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:52.526615 (+ 18929us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:52.526940 (+ 325us) write_op.cc:312] APPLY: finished
0505 13:59:52.532373 (+ 5433us) log.cc:844] Serialized 320 byte log entry
0505 13:59:52.535839 (+ 3466us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:52.536407 (+ 568us) write_op.cc:454] Released schema lock
0505 13:59:52.536767 (+ 360us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":9039,"num_ops":50,"prepare.queue_time_us":666440,"prepare.run_cpu_time_us":12575,"prepare.run_wall_time_us":129620,"raft.queue_time_us":5590,"raft.run_cpu_time_us":3368,"raft.run_wall_time_us":12858,"replication_time_us":128608,"row_lock_wait_count":1,"row_lock_wait_us":86255,"spinlock_wait_cycles":1235328}]]}
I20260505 13:59:52.725508 7699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1737, attempt_no=0}) took 1099 ms. Trace:
I20260505 13:59:52.740471 7699 rpcz_store.cc:276] 0505 13:59:51.624966 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:51.626496 (+ 1530us) service_pool.cc:224] Handling call
0505 13:59:52.723861 (+1097365us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:52.286437 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:52.287856 (+ 1419us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:52.287944 (+ 88us) write_op.cc:435] Acquired schema lock
0505 13:59:52.287960 (+ 16us) tablet.cc:662] Decoding operations
0505 13:59:52.295546 (+ 7586us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:52.295603 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:52.295630 (+ 27us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:52.538001 (+242371us) lock_manager.cc:598] Waited 194092us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:52.538456 (+ 455us) tablet.cc:701] Row locks acquired
0505 13:59:52.538477 (+ 21us) write_op.cc:260] PREPARE: finished
0505 13:59:52.538938 (+ 461us) write_op.cc:270] Start()
0505 13:59:52.539214 (+ 276us) write_op.cc:276] Timestamp: P: 1777989592538895 usec, L: 0
0505 13:59:52.539234 (+ 20us) op_driver.cc:348] REPLICATION: starting
0505 13:59:52.547521 (+ 8287us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:52.590588 (+ 43067us) op_driver.cc:464] REPLICATION: finished
0505 13:59:52.590900 (+ 312us) write_op.cc:301] APPLY: starting
0505 13:59:52.591117 (+ 217us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:52.591758 (+ 641us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:52.591777 (+ 19us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:52.673256 (+ 81479us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:52.674202 (+ 946us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:52.674247 (+ 45us) write_op.cc:312] APPLY: finished
0505 13:59:52.675467 (+ 1220us) log.cc:844] Serialized 320 byte log entry
0505 13:59:52.711344 (+ 35877us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:52.712150 (+ 806us) write_op.cc:454] Released schema lock
0505 13:59:52.722251 (+ 10101us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":94,"num_ops":50,"prepare.queue_time_us":635752,"prepare.run_cpu_time_us":22941,"prepare.run_wall_time_us":274358,"raft.queue_time_us":6632,"raft.run_cpu_time_us":8093,"raft.run_wall_time_us":8197,"replication_time_us":51175,"row_lock_wait_count":1,"row_lock_wait_us":194092,"spinlock_wait_cycles":1009536}]]}
I20260505 13:59:52.979014 7699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1738, attempt_no=0}) took 1244 ms. Trace:
I20260505 13:59:52.993098 7699 rpcz_store.cc:276] 0505 13:59:51.734841 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:51.734976 (+ 135us) service_pool.cc:224] Handling call
0505 13:59:52.978632 (+1243656us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:52.561146 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:52.561987 (+ 841us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:52.562040 (+ 53us) write_op.cc:435] Acquired schema lock
0505 13:59:52.562077 (+ 37us) tablet.cc:662] Decoding operations
0505 13:59:52.596612 (+ 34535us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:52.596684 (+ 72us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:52.596728 (+ 44us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:52.712247 (+115519us) tablet.cc:701] Row locks acquired
0505 13:59:52.712349 (+ 102us) write_op.cc:260] PREPARE: finished
0505 13:59:52.712771 (+ 422us) write_op.cc:270] Start()
0505 13:59:52.713031 (+ 260us) write_op.cc:276] Timestamp: P: 1777989592712756 usec, L: 0
0505 13:59:52.713051 (+ 20us) op_driver.cc:348] REPLICATION: starting
0505 13:59:52.715507 (+ 2456us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:52.874952 (+159445us) op_driver.cc:464] REPLICATION: finished
0505 13:59:52.876604 (+ 1652us) write_op.cc:301] APPLY: starting
0505 13:59:52.876901 (+ 297us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:52.878733 (+ 1832us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:52.879178 (+ 445us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:52.947929 (+ 68751us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:52.968994 (+ 21065us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:52.969196 (+ 202us) write_op.cc:312] APPLY: finished
0505 13:59:52.973252 (+ 4056us) log.cc:844] Serialized 320 byte log entry
0505 13:59:52.976388 (+ 3136us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:52.976879 (+ 491us) write_op.cc:454] Released schema lock
0505 13:59:52.977299 (+ 420us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":485,"num_ops":50,"prepare.queue_time_us":820987,"prepare.run_cpu_time_us":22201,"prepare.run_wall_time_us":185817,"raft.queue_time_us":903,"raft.run_cpu_time_us":10702,"raft.run_wall_time_us":43475,"replication_time_us":161026,"row_lock_wait_count":1,"row_lock_wait_us":89008,"spinlock_wait_cycles":693504}]]}
W20260505 13:59:53.080224 7322 log.cc:927] Time spent T 88ffad060bde428e92d12021c59d85f7 P b11284b44f424719ba754e92c1af3e33: Append to log took a long time: real 0.076s user 0.003s sys 0.004s
I20260505 13:59:53.192044 7699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1739, attempt_no=0}) took 1114 ms. Trace:
I20260505 13:59:53.192772 7699 rpcz_store.cc:276] 0505 13:59:52.078597 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:52.080630 (+ 2033us) service_pool.cc:224] Handling call
0505 13:59:53.191920 (+1111290us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:52.748915 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:52.750515 (+ 1600us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:52.750584 (+ 69us) write_op.cc:435] Acquired schema lock
0505 13:59:52.750623 (+ 39us) tablet.cc:662] Decoding operations
0505 13:59:52.841838 (+ 91215us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:52.841906 (+ 68us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:52.841950 (+ 44us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:52.977133 (+135183us) lock_manager.cc:598] Waited 127818us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:52.977511 (+ 378us) tablet.cc:701] Row locks acquired
0505 13:59:52.977590 (+ 79us) write_op.cc:260] PREPARE: finished
0505 13:59:52.988469 (+ 10879us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b34000016d8. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45c8feef61 00007f45c902d2f4 00007f45cb7ed625 00007f45cb7ed1c8 00007f45cb7f1e61 00007f45cb7f1e19 00007f45cb7f1da9 00007f45cb7f1d71 00007f45cb7f0f9d 00000000005b3bc4 00000000005b39f9
0505 13:59:52.989061 (+ 592us) write_op.cc:270] Start()
0505 13:59:52.989203 (+ 142us) write_op.cc:276] Timestamp: P: 1777989592988797 usec, L: 0
0505 13:59:52.989223 (+ 20us) op_driver.cc:348] REPLICATION: starting
0505 13:59:52.990301 (+ 1078us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:53.118291 (+127990us) op_driver.cc:464] REPLICATION: finished
0505 13:59:53.118920 (+ 629us) write_op.cc:301] APPLY: starting
0505 13:59:53.119005 (+ 85us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:53.120543 (+ 1538us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:53.120720 (+ 177us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:53.187581 (+ 66861us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:53.188994 (+ 1413us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:53.189077 (+ 83us) write_op.cc:312] APPLY: finished
0505 13:59:53.190612 (+ 1535us) log.cc:844] Serialized 320 byte log entry
0505 13:59:53.191013 (+ 401us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:53.191227 (+ 214us) write_op.cc:454] Released schema lock
0505 13:59:53.191388 (+ 161us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":315,"num_ops":50,"prepare.queue_time_us":654480,"prepare.run_cpu_time_us":24648,"prepare.run_wall_time_us":242354,"raft.queue_time_us":1608,"raft.run_cpu_time_us":1724,"raft.run_wall_time_us":1722,"replication_time_us":128892,"row_lock_wait_count":1,"row_lock_wait_us":127818,"spinlock_wait_cycles":3174400}]]}
I20260505 13:59:53.282752 7699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1740, attempt_no=0}) took 1122 ms. Trace:
I20260505 13:59:53.283419 7699 rpcz_store.cc:276] 0505 13:59:52.161312 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:52.161843 (+ 531us) service_pool.cc:224] Handling call
0505 13:59:53.282661 (+1120818us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:52.991127 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:52.991658 (+ 531us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:52.991693 (+ 35us) write_op.cc:435] Acquired schema lock
0505 13:59:52.991706 (+ 13us) tablet.cc:662] Decoding operations
0505 13:59:52.994705 (+ 2999us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:52.994757 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:52.994791 (+ 34us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:53.191225 (+196434us) lock_manager.cc:598] Waited 164138us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:53.191359 (+ 134us) tablet.cc:701] Row locks acquired
0505 13:59:53.191372 (+ 13us) write_op.cc:260] PREPARE: finished
0505 13:59:53.191617 (+ 245us) write_op.cc:270] Start()
0505 13:59:53.191761 (+ 144us) write_op.cc:276] Timestamp: P: 1777989593191785 usec, L: 0
0505 13:59:53.191776 (+ 15us) op_driver.cc:348] REPLICATION: starting
0505 13:59:53.192701 (+ 925us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:53.224284 (+ 31583us) op_driver.cc:464] REPLICATION: finished
0505 13:59:53.224662 (+ 378us) write_op.cc:301] APPLY: starting
0505 13:59:53.224834 (+ 172us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:53.225647 (+ 813us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:53.225686 (+ 39us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:53.240588 (+ 14902us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:53.240964 (+ 376us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:53.241021 (+ 57us) write_op.cc:312] APPLY: finished
0505 13:59:53.241794 (+ 773us) log.cc:844] Serialized 320 byte log entry
0505 13:59:53.252266 (+ 10472us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:53.281598 (+ 29332us) write_op.cc:454] Released schema lock
0505 13:59:53.281786 (+ 188us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":3328,"child_traces":[["op",{"apply.queue_time_us":190,"num_ops":50,"prepare.queue_time_us":754331,"prepare.run_cpu_time_us":9858,"prepare.run_wall_time_us":201999,"raft.queue_time_us":6173,"raft.run_cpu_time_us":1954,"raft.run_wall_time_us":1954,"replication_time_us":32370,"row_lock_wait_count":1,"row_lock_wait_us":164138}]]}
W20260505 13:59:53.365342 7270 debug-util.cc:398] Leaking SignalData structure 0x7b08000cdd40 after lost signal to thread 7146
W20260505 13:59:53.436352 7270 debug-util.cc:398] Leaking SignalData structure 0x7b0800082ae0 after lost signal to thread 7273
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:2164: Failure
Failed
Bad status: Timed out: Timed out waiting for number of WAL segments on tablet 88ffad060bde428e92d12021c59d85f7 on TS 0 to be 6. Found 4
I20260505 13:59:53.755827 7699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1743, attempt_no=0}) took 1141 ms. Trace:
I20260505 13:59:53.756827 7699 rpcz_store.cc:276] 0505 13:59:52.615002 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:52.615892 (+ 890us) service_pool.cc:224] Handling call
0505 13:59:53.755609 (+1139717us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:53.350262 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:53.351999 (+ 1737us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:53.369592 (+ 17593us) write_op.cc:435] Acquired schema lock
0505 13:59:53.369614 (+ 22us) tablet.cc:662] Decoding operations
0505 13:59:53.375644 (+ 6030us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:53.375928 (+ 284us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:53.375972 (+ 44us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:53.484433 (+108461us) lock_manager.cc:598] Waited 103752us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:53.484634 (+ 201us) tablet.cc:701] Row locks acquired
0505 13:59:53.484666 (+ 32us) write_op.cc:260] PREPARE: finished
0505 13:59:53.485026 (+ 360us) write_op.cc:270] Start()
0505 13:59:53.485157 (+ 131us) write_op.cc:276] Timestamp: P: 1777989593485089 usec, L: 0
0505 13:59:53.485177 (+ 20us) op_driver.cc:348] REPLICATION: starting
0505 13:59:53.486287 (+ 1110us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:53.609014 (+122727us) op_driver.cc:464] REPLICATION: finished
0505 13:59:53.609616 (+ 602us) write_op.cc:301] APPLY: starting
0505 13:59:53.609830 (+ 214us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:53.610760 (+ 930us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:53.610805 (+ 45us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:53.686788 (+ 75983us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:53.710347 (+ 23559us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:53.710805 (+ 458us) write_op.cc:312] APPLY: finished
0505 13:59:53.714884 (+ 4079us) log.cc:844] Serialized 320 byte log entry
0505 13:59:53.718512 (+ 3628us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:53.735604 (+ 17092us) write_op.cc:454] Released schema lock
0505 13:59:53.735939 (+ 335us) write_op.cc:341] FINISH: Updating metrics
0505 13:59:53.755014 (+ 19075us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b54000600a8. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cd683af1 00007f45cd68275c 00007f45cb80389d 00007f45cb7ef7f9 00007f45cb7eefc3 00007f45cb7f31d1 00007f45cb7f3189 00007f45cb7f3119 00007f45cb7f30e1 00007f45cb7f230d 00000000005b3bc4
Metrics: {"child_traces":[["op",{"apply.queue_time_us":341,"num_ops":50,"prepare.queue_time_us":702621,"prepare.run_cpu_time_us":15701,"prepare.run_wall_time_us":136651,"raft.queue_time_us":1415,"raft.run_cpu_time_us":4865,"raft.run_wall_time_us":23548,"replication_time_us":123653,"row_lock_wait_count":1,"row_lock_wait_us":103752,"spinlock_wait_cycles":25232000}]]}
I20260505 13:59:53.923501 7720 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1744, attempt_no=0}) took 1109 ms. Trace:
I20260505 13:59:53.925941 7720 rpcz_store.cc:276] 0505 13:59:52.813888 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:52.814655 (+ 767us) service_pool.cc:224] Handling call
0505 13:59:53.922005 (+1107350us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:53.486890 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:53.487336 (+ 446us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:53.487359 (+ 23us) write_op.cc:435] Acquired schema lock
0505 13:59:53.487372 (+ 13us) tablet.cc:662] Decoding operations
0505 13:59:53.499527 (+ 12155us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:53.499704 (+ 177us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:53.499793 (+ 89us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:53.719033 (+219240us) lock_manager.cc:598] Waited 178610us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:53.719239 (+ 206us) tablet.cc:701] Row locks acquired
0505 13:59:53.719316 (+ 77us) write_op.cc:260] PREPARE: finished
0505 13:59:53.719970 (+ 654us) write_op.cc:270] Start()
0505 13:59:53.720425 (+ 455us) write_op.cc:276] Timestamp: P: 1777989593719749 usec, L: 0
0505 13:59:53.720579 (+ 154us) op_driver.cc:348] REPLICATION: starting
0505 13:59:53.723862 (+ 3283us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:53.742652 (+ 18790us) spinlock_profiling.cc:243] Waited 6.42 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2f8522 00007f45cb2d928a 00007f45cb2e0133 00007f45cb2e00b9 00007f45cb2e0049 00007f45cb2e0011 00007f45cb2df1dd 00000000005b3bc4 00000000005b39f9 00007f45c42c2ab5
0505 13:59:53.744749 (+ 2097us) spinlock_profiling.cc:243] Waited 6.12 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2f8522 00007f45cb2d928a 00007f45cb2e0133 00007f45cb2e00b9 00007f45cb2e0049 00007f45cb2e0011 00007f45cb2df1dd 00000000005b3bc4 00000000005b39f9 00007f45c42c2ab5
0505 13:59:53.802497 (+ 57748us) op_driver.cc:464] REPLICATION: finished
0505 13:59:53.816405 (+ 13908us) write_op.cc:301] APPLY: starting
0505 13:59:53.816656 (+ 251us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:53.819333 (+ 2677us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:53.819451 (+ 118us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:53.878103 (+ 58652us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:53.881317 (+ 3214us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:53.881575 (+ 258us) write_op.cc:312] APPLY: finished
0505 13:59:53.886605 (+ 5030us) log.cc:844] Serialized 320 byte log entry
0505 13:59:53.905237 (+ 18632us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:53.919848 (+ 14611us) write_op.cc:454] Released schema lock
0505 13:59:53.920409 (+ 561us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":13295,"num_ops":50,"prepare.queue_time_us":658545,"prepare.run_cpu_time_us":20376,"prepare.run_wall_time_us":246147,"raft.queue_time_us":3785,"raft.run_cpu_time_us":11834,"raft.run_wall_time_us":37513,"replication_time_us":81607,"row_lock_wait_count":1,"row_lock_wait_us":178610,"spinlock_wait_cycles":28838656}]]}
I20260505 13:59:54.276075 7699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1746, attempt_no=0}) took 1062 ms. Trace:
I20260505 13:59:54.301784 7699 rpcz_store.cc:276] 0505 13:59:53.213988 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:53.214161 (+ 173us) service_pool.cc:224] Handling call
0505 13:59:54.275537 (+1061376us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:53.943538 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:53.944075 (+ 537us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:53.944101 (+ 26us) write_op.cc:435] Acquired schema lock
0505 13:59:53.944114 (+ 13us) tablet.cc:662] Decoding operations
0505 13:59:53.947221 (+ 3107us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:53.947294 (+ 73us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:53.947334 (+ 40us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:54.096848 (+149514us) lock_manager.cc:598] Waited 146669us for lock on <redacted> (key hash 11785448468237383701)
0505 13:59:54.097008 (+ 160us) tablet.cc:701] Row locks acquired
0505 13:59:54.097047 (+ 39us) write_op.cc:260] PREPARE: finished
0505 13:59:54.097881 (+ 834us) write_op.cc:270] Start()
0505 13:59:54.098089 (+ 208us) write_op.cc:276] Timestamp: P: 1777989594097758 usec, L: 0
0505 13:59:54.098232 (+ 143us) op_driver.cc:348] REPLICATION: starting
0505 13:59:54.101719 (+ 3487us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:54.121731 (+ 20012us) spinlock_profiling.cc:243] Waited 9 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2f8522 00007f45cb2d928a 00007f45cb2e0133 00007f45cb2e00b9 00007f45cb2e0049 00007f45cb2e0011 00007f45cb2df1dd 00000000005b3bc4 00000000005b39f9 00007f45c42c2ab5
0505 13:59:54.141468 (+ 19737us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b54000701a0. stack: 00007f45c4290c83 00007f45c4290b80 00007f45c3e01574 000000000050e094 000000000050e029 00007f45cb2e8951 00007f45cb2f8522 00007f45cb2d928a 00007f45cb2e0133 00007f45cb2e00b9 00007f45cb2e0049 00007f45cb2e0011 00007f45cb2df1dd 00000000005b3bc4 00000000005b39f9 00007f45c42c2ab5
0505 13:59:54.165124 (+ 23656us) op_driver.cc:464] REPLICATION: finished
0505 13:59:54.166505 (+ 1381us) write_op.cc:301] APPLY: starting
0505 13:59:54.166985 (+ 480us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:54.169674 (+ 2689us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:54.169697 (+ 23us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:54.245915 (+ 76218us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:54.248906 (+ 2991us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:54.249234 (+ 328us) write_op.cc:312] APPLY: finished
0505 13:59:54.265090 (+ 15856us) log.cc:844] Serialized 320 byte log entry
0505 13:59:54.273049 (+ 7959us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:54.273853 (+ 804us) write_op.cc:454] Released schema lock
0505 13:59:54.274219 (+ 366us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":680,"mutex_wait_us":54,"num_ops":50,"prepare.queue_time_us":727870,"prepare.run_cpu_time_us":12863,"prepare.run_wall_time_us":165691,"raft.queue_time_us":6115,"raft.run_cpu_time_us":17568,"raft.run_wall_time_us":133937,"replication_time_us":66665,"row_lock_wait_count":1,"row_lock_wait_us":146669,"spinlock_wait_cycles":69218944}]]}
I20260505 13:59:54.395138 7720 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1747, attempt_no=0}) took 1088 ms. Trace:
I20260505 13:59:54.400281 7720 rpcz_store.cc:276] 0505 13:59:53.306552 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:53.306750 (+ 198us) service_pool.cc:224] Handling call
0505 13:59:54.394987 (+1088237us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:54.110176 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:54.111266 (+ 1090us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:54.111344 (+ 78us) write_op.cc:435] Acquired schema lock
0505 13:59:54.111370 (+ 26us) tablet.cc:662] Decoding operations
0505 13:59:54.199520 (+ 88150us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:54.199564 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:54.199589 (+ 25us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:54.273709 (+ 74120us) tablet.cc:701] Row locks acquired
0505 13:59:54.273769 (+ 60us) write_op.cc:260] PREPARE: finished
0505 13:59:54.274619 (+ 850us) write_op.cc:270] Start()
0505 13:59:54.274898 (+ 279us) write_op.cc:276] Timestamp: P: 1777989594274264 usec, L: 0
0505 13:59:54.274934 (+ 36us) op_driver.cc:348] REPLICATION: starting
0505 13:59:54.278300 (+ 3366us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:54.331288 (+ 52988us) op_driver.cc:464] REPLICATION: finished
0505 13:59:54.331870 (+ 582us) write_op.cc:301] APPLY: starting
0505 13:59:54.331943 (+ 73us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:54.332618 (+ 675us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:54.332704 (+ 86us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:54.349987 (+ 17283us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:54.351133 (+ 1146us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:54.351230 (+ 97us) write_op.cc:312] APPLY: finished
0505 13:59:54.352732 (+ 1502us) log.cc:844] Serialized 320 byte log entry
0505 13:59:54.353836 (+ 1104us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:54.394259 (+ 40423us) write_op.cc:454] Released schema lock
0505 13:59:54.394429 (+ 170us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":311,"mutex_wait_us":15024,"num_ops":50,"prepare.queue_time_us":801165,"prepare.run_cpu_time_us":16842,"prepare.run_wall_time_us":184644,"raft.queue_time_us":17657,"raft.run_cpu_time_us":6677,"raft.run_wall_time_us":33388,"replication_time_us":56196,"row_lock_wait_count":1,"row_lock_wait_us":71744}]]}
I20260505 13:59:54.418702 7699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1748, attempt_no=0}) took 1074 ms. Trace:
I20260505 13:59:54.476152 7699 rpcz_store.cc:276] 0505 13:59:53.345551 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:53.346735 (+ 1184us) service_pool.cc:224] Handling call
0505 13:59:54.418591 (+1071856us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:54.313755 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:54.315641 (+ 1886us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:54.315695 (+ 54us) write_op.cc:435] Acquired schema lock
0505 13:59:54.315836 (+ 141us) tablet.cc:662] Decoding operations
0505 13:59:54.335487 (+ 19651us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:54.335566 (+ 79us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:54.335591 (+ 25us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:54.354005 (+ 18414us) tablet.cc:701] Row locks acquired
0505 13:59:54.354027 (+ 22us) write_op.cc:260] PREPARE: finished
0505 13:59:54.354229 (+ 202us) write_op.cc:270] Start()
0505 13:59:54.354312 (+ 83us) write_op.cc:276] Timestamp: P: 1777989594354091 usec, L: 0
0505 13:59:54.354327 (+ 15us) op_driver.cc:348] REPLICATION: starting
0505 13:59:54.355162 (+ 835us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:54.380400 (+ 25238us) op_driver.cc:464] REPLICATION: finished
0505 13:59:54.380872 (+ 472us) write_op.cc:301] APPLY: starting
0505 13:59:54.380931 (+ 59us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:54.381793 (+ 862us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:54.381914 (+ 121us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:54.393268 (+ 11354us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:54.394239 (+ 971us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:54.394289 (+ 50us) write_op.cc:312] APPLY: finished
0505 13:59:54.395576 (+ 1287us) log.cc:844] Serialized 320 byte log entry
0505 13:59:54.395893 (+ 317us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:54.396147 (+ 254us) write_op.cc:454] Released schema lock
0505 13:59:54.418187 (+ 22040us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":224,"num_ops":50,"prepare.queue_time_us":942610,"prepare.run_cpu_time_us":9172,"prepare.run_wall_time_us":82015,"raft.queue_time_us":112,"raft.run_cpu_time_us":911,"raft.run_wall_time_us":912,"replication_time_us":25822,"row_lock_wait_count":1,"row_lock_wait_us":16567}]]}
W20260505 13:59:54.539775 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 53086.7us
W20260505 13:59:54.623626 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 68896.8us
I20260505 13:59:54.733887 7699 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47532 (ReqId={client: b24ae52190d045f7822517590116ebd0, seq_no=1750, attempt_no=0}) took 1024 ms. Trace:
I20260505 13:59:54.745572 7699 rpcz_store.cc:276] 0505 13:59:53.710596 (+ 0us) service_pool.cc:167] Inserting onto call queue
0505 13:59:53.711379 (+ 783us) service_pool.cc:224] Handling call
0505 13:59:54.733668 (+1022289us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0505 13:59:54.445682 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 88ffad060bde428e92d12021c59d85f7
0505 13:59:54.446297 (+ 615us) write_op.cc:432] Acquiring schema lock in shared mode
0505 13:59:54.446322 (+ 25us) write_op.cc:435] Acquired schema lock
0505 13:59:54.446346 (+ 24us) tablet.cc:662] Decoding operations
0505 13:59:54.467968 (+ 21622us) write_op.cc:620] Acquiring the partition lock for write op
0505 13:59:54.468491 (+ 523us) write_op.cc:641] Partition lock acquired for write op
0505 13:59:54.468570 (+ 79us) tablet.cc:685] Acquiring locks for 50 operations
0505 13:59:54.555531 (+ 86961us) tablet.cc:701] Row locks acquired
0505 13:59:54.555569 (+ 38us) write_op.cc:260] PREPARE: finished
0505 13:59:54.556141 (+ 572us) write_op.cc:270] Start()
0505 13:59:54.556575 (+ 434us) write_op.cc:276] Timestamp: P: 1777989594555967 usec, L: 0
0505 13:59:54.556592 (+ 17us) op_driver.cc:348] REPLICATION: starting
0505 13:59:54.558593 (+ 2001us) log.cc:844] Serialized 2467 byte log entry
0505 13:59:54.623028 (+ 64435us) op_driver.cc:464] REPLICATION: finished
0505 13:59:54.628749 (+ 5721us) write_op.cc:301] APPLY: starting
0505 13:59:54.628876 (+ 127us) tablet.cc:1367] starting BulkCheckPresence
0505 13:59:54.630980 (+ 2104us) tablet.cc:1370] finished BulkCheckPresence
0505 13:59:54.631262 (+ 282us) tablet.cc:1372] starting ApplyRowOperation cycle
0505 13:59:54.728762 (+ 97500us) tablet.cc:1383] finished ApplyRowOperation cycle
0505 13:59:54.730244 (+ 1482us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=50
0505 13:59:54.730429 (+ 185us) write_op.cc:312] APPLY: finished
0505 13:59:54.732318 (+ 1889us) log.cc:844] Serialized 320 byte log entry
0505 13:59:54.732747 (+ 429us) write_op.cc:489] Releasing partition, row and schema locks
0505 13:59:54.733072 (+ 325us) write_op.cc:454] Released schema lock
0505 13:59:54.733204 (+ 132us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5082,"num_ops":50,"prepare.queue_time_us":721148,"prepare.run_cpu_time_us":20173,"prepare.run_wall_time_us":139657,"raft.queue_time_us":5054,"raft.run_cpu_time_us":9264,"raft.run_wall_time_us":64936,"replication_time_us":66053,"row_lock_wait_count":1,"row_lock_wait_us":36678}]]}
I20260505 13:59:54.762014 6116 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20260505 13:59:54.763522 6116 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 05d24c1cf32a4a18bfb58c541a620c2f and pid 6878
W20260505 13:59:55.001066 7286 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.249.1:39501, user_credentials={real_user=slave}} blocked reactor thread for 244155us
************************ BEGIN STACKS **************************
[New LWP 6879]
[New LWP 6880]
[New LWP 6881]
[New LWP 6882]
[New LWP 6883]
[New LWP 6891]
[New LWP 6892]
[New LWP 6893]
[New LWP 6896]
[New LWP 6897]
[New LWP 6898]
[New LWP 6899]
[New LWP 6900]
[New LWP 6901]
[New LWP 6902]
[New LWP 6903]
[New LWP 6904]
[New LWP 6905]
[New LWP 6906]
[New LWP 6907]
[New LWP 6908]
[New LWP 6909]
[New LWP 6910]
[New LWP 6911]
[New LWP 6912]
[New LWP 6913]
[New LWP 6914]
[New LWP 6915]
[New LWP 6916]
[New LWP 6917]
[New LWP 6918]
[New LWP 6919]
[New LWP 6920]
[New LWP 6921]
[New LWP 6922]
[New LWP 6923]
[New LWP 6924]
[New LWP 6925]
[New LWP 6926]
[New LWP 6927]
[New LWP 6928]
[New LWP 6929]
[New LWP 6930]
[New LWP 6931]
[New LWP 6932]
[New LWP 6933]
[New LWP 6934]
[New LWP 6935]
[New LWP 6936]
[New LWP 6937]
[New LWP 6938]
[New LWP 6939]
[New LWP 6940]
[New LWP 6941]
[New LWP 6942]
[New LWP 6943]
[New LWP 6944]
[New LWP 6945]
[New LWP 6946]
[New LWP 6947]
[New LWP 6948]
[New LWP 6949]
[New LWP 6950]
[New LWP 6951]
[New LWP 6952]
[New LWP 6953]
[New LWP 6954]
[New LWP 6955]
[New LWP 6956]
[New LWP 6957]
[New LWP 6958]
[New LWP 6959]
[New LWP 6960]
[New LWP 6961]
[New LWP 6962]
[New LWP 6963]
[New LWP 6964]
[New LWP 6965]
[New LWP 6966]
[New LWP 6967]
[New LWP 6968]
[New LWP 6969]
[New LWP 6970]
[New LWP 6971]
[New LWP 6972]
[New LWP 6973]
[New LWP 6974]
[New LWP 6975]
[New LWP 6976]
[New LWP 6977]
[New LWP 6978]
[New LWP 6979]
[New LWP 6980]
[New LWP 6981]
[New LWP 6982]
[New LWP 6983]
[New LWP 6984]
[New LWP 6985]
[New LWP 6986]
[New LWP 6987]
[New LWP 6988]
[New LWP 6989]
[New LWP 6990]
[New LWP 6991]
[New LWP 6992]
[New LWP 6993]
[New LWP 6994]
[New LWP 6995]
[New LWP 6996]
[New LWP 6997]
[New LWP 6998]
[New LWP 6999]
[New LWP 7000]
[New LWP 7001]
[New LWP 7002]
[New LWP 7003]
[New LWP 7004]
[New LWP 7005]
[New LWP 7006]
[New LWP 7007]
[New LWP 7008]
[New LWP 7009]
[New LWP 7726]
Cannot access memory at address 0xc4108070c6b0396
Cannot access memory at address 0xc4108070c6b038e
Cannot access memory at address 0xc4108070c6b0396
Cannot access memory at address 0xc4108070c6b0396
Cannot access memory at address 0xc4108070c6b038e
0x00007f45c59abd50 in ?? ()
Id Target Id Frame
* 1 LWP 6878 "kudu" 0x00007f45c59abd50 in ?? ()
2 LWP 6879 "kudu" 0x00007f45c0ce7680 in ?? ()
3 LWP 6880 "kudu" 0x00007f45c59a7fb9 in ?? ()
4 LWP 6881 "kudu" 0x00007f45c59a7fb9 in ?? ()
5 LWP 6882 "kudu" 0x00007f45c59a7fb9 in ?? ()
6 LWP 6883 "kernel-watcher-" 0x00007f45c59a7fb9 in ?? ()
7 LWP 6891 "ntp client-6891" 0x00007f45c59ab9e2 in ?? ()
8 LWP 6892 "file cache-evic" 0x00007f45c59a7fb9 in ?? ()
9 LWP 6893 "sq_acceptor" 0x00007f45c0d17bb9 in ?? ()
10 LWP 6896 "rpc reactor-689" 0x00007f45c0d24947 in ?? ()
11 LWP 6897 "rpc reactor-689" 0x00007f45c0d24947 in ?? ()
12 LWP 6898 "rpc reactor-689" 0x00007f45c0d24947 in ?? ()
13 LWP 6899 "rpc reactor-689" 0x00007f45c0d24947 in ?? ()
14 LWP 6900 "MaintenanceMgr " 0x00007f45c59a7ad3 in ?? ()
15 LWP 6901 "txn-status-mana" 0x00007f45c59a7fb9 in ?? ()
16 LWP 6902 "collect_and_rem" 0x00007f45c59a7fb9 in ?? ()
17 LWP 6903 "tc-session-exp-" 0x00007f45c59a7fb9 in ?? ()
18 LWP 6904 "rpc worker-6904" 0x00007f45c59a7ad3 in ?? ()
19 LWP 6905 "rpc worker-6905" 0x00007f45c59a7ad3 in ?? ()
20 LWP 6906 "rpc worker-6906" 0x00007f45c59a7ad3 in ?? ()
21 LWP 6907 "rpc worker-6907" 0x00007f45c59a7ad3 in ?? ()
22 LWP 6908 "rpc worker-6908" 0x00007f45c59a7ad3 in ?? ()
23 LWP 6909 "rpc worker-6909" 0x00007f45c59a7ad3 in ?? ()
24 LWP 6910 "rpc worker-6910" 0x00007f45c59a7ad3 in ?? ()
25 LWP 6911 "rpc worker-6911" 0x00007f45c59a7ad3 in ?? ()
26 LWP 6912 "rpc worker-6912" 0x00007f45c59a7ad3 in ?? ()
27 LWP 6913 "rpc worker-6913" 0x00007f45c59a7ad3 in ?? ()
28 LWP 6914 "rpc worker-6914" 0x00007f45c59a7ad3 in ?? ()
29 LWP 6915 "rpc worker-6915" 0x00007f45c59a7ad3 in ?? ()
30 LWP 6916 "rpc worker-6916" 0x00007f45c59a7ad3 in ?? ()
31 LWP 6917 "rpc worker-6917" 0x00007f45c59a7ad3 in ?? ()
32 LWP 6918 "rpc worker-6918" 0x00007f45c59a7ad3 in ?? ()
33 LWP 6919 "rpc worker-6919" 0x00007f45c59a7ad3 in ?? ()
34 LWP 6920 "rpc worker-6920" 0x00007f45c59a7ad3 in ?? ()
35 LWP 6921 "rpc worker-6921" 0x00007f45c59a7ad3 in ?? ()
36 LWP 6922 "rpc worker-6922" 0x00007f45c59a7ad3 in ?? ()
37 LWP 6923 "rpc worker-6923" 0x00007f45c59a7ad3 in ?? ()
38 LWP 6924 "rpc worker-6924" 0x00007f45c59a7ad3 in ?? ()
39 LWP 6925 "rpc worker-6925" 0x00007f45c59a7ad3 in ?? ()
40 LWP 6926 "rpc worker-6926" 0x00007f45c59a7ad3 in ?? ()
41 LWP 6927 "rpc worker-6927" 0x00007f45c59a7ad3 in ?? ()
42 LWP 6928 "rpc worker-6928" 0x00007f45c59a7ad3 in ?? ()
43 LWP 6929 "rpc worker-6929" 0x00007f45c59a7ad3 in ?? ()
44 LWP 6930 "rpc worker-6930" 0x00007f45c59a7ad3 in ?? ()
45 LWP 6931 "rpc worker-6931" 0x00007f45c59a7ad3 in ?? ()
46 LWP 6932 "rpc worker-6932" 0x00007f45c59a7ad3 in ?? ()
47 LWP 6933 "rpc worker-6933" 0x00007f45c59a7ad3 in ?? ()
48 LWP 6934 "rpc worker-6934" 0x00007f45c59a7ad3 in ?? ()
49 LWP 6935 "rpc worker-6935" 0x00007f45c59a7ad3 in ?? ()
50 LWP 6936 "rpc worker-6936" 0x00007f45c59a7ad3 in ?? ()
51 LWP 6937 "rpc worker-6937" 0x00007f45c59a7ad3 in ?? ()
52 LWP 6938 "rpc worker-6938" 0x00007f45c59a7ad3 in ?? ()
53 LWP 6939 "rpc worker-6939" 0x00007f45c59a7ad3 in ?? ()
54 LWP 6940 "rpc worker-6940" 0x00007f45c59a7ad3 in ?? ()
55 LWP 6941 "rpc worker-6941" 0x00007f45c59a7ad3 in ?? ()
56 LWP 6942 "rpc worker-6942" 0x00007f45c59a7ad3 in ?? ()
57 LWP 6943 "rpc worker-6943" 0x00007f45c59a7ad3 in ?? ()
58 LWP 6944 "rpc worker-6944" 0x00007f45c59a7ad3 in ?? ()
59 LWP 6945 "rpc worker-6945" 0x00007f45c59a7ad3 in ?? ()
60 LWP 6946 "rpc worker-6946" 0x00007f45c59a7ad3 in ?? ()
61 LWP 6947 "rpc worker-6947" 0x00007f45c59a7ad3 in ?? ()
62 LWP 6948 "rpc worker-6948" 0x00007f45c59a7ad3 in ?? ()
63 LWP 6949 "rpc worker-6949" 0x00007f45c59a7ad3 in ?? ()
64 LWP 6950 "rpc worker-6950" 0x00007f45c59a7ad3 in ?? ()
65 LWP 6951 "rpc worker-6951" 0x00007f45c59a7ad3 in ?? ()
66 LWP 6952 "rpc worker-6952" 0x00007f45c59a7ad3 in ?? ()
67 LWP 6953 "rpc worker-6953" 0x00007f45c59a7ad3 in ?? ()
68 LWP 6954 "rpc worker-6954" 0x00007f45c59a7ad3 in ?? ()
69 LWP 6955 "rpc worker-6955" 0x00007f45c59a7ad3 in ?? ()
70 LWP 6956 "rpc worker-6956" 0x00007f45c59a7ad3 in ?? ()
71 LWP 6957 "rpc worker-6957" 0x00007f45c59a7ad3 in ?? ()
72 LWP 6958 "rpc worker-6958" 0x00007f45c59a7ad3 in ?? ()
73 LWP 6959 "rpc worker-6959" 0x00007f45c59a7ad3 in ?? ()
74 LWP 6960 "rpc worker-6960" 0x00007f45c59a7ad3 in ?? ()
75 LWP 6961 "rpc worker-6961" 0x00007f45c59a7ad3 in ?? ()
76 LWP 6962 "rpc worker-6962" 0x00007f45c59a7ad3 in ?? ()
77 LWP 6963 "rpc worker-6963" 0x00007f45c59a7ad3 in ?? ()
78 LWP 6964 "rpc worker-6964" 0x00007f45c59a7ad3 in ?? ()
79 LWP 6965 "rpc worker-6965" 0x00007f45c59a7ad3 in ?? ()
80 LWP 6966 "rpc worker-6966" 0x00007f45c59a7ad3 in ?? ()
81 LWP 6967 "rpc worker-6967" 0x00007f45c59a7ad3 in ?? ()
82 LWP 6968 "rpc worker-6968" 0x00007f45c59a7ad3 in ?? ()
83 LWP 6969 "rpc worker-6969" 0x00007f45c59a7ad3 in ?? ()
84 LWP 6970 "rpc worker-6970" 0x00007f45c59a7ad3 in ?? ()
85 LWP 6971 "rpc worker-6971" 0x00007f45c59a7ad3 in ?? ()
86 LWP 6972 "rpc worker-6972" 0x00007f45c59a7ad3 in ?? ()
87 LWP 6973 "rpc worker-6973" 0x00007f45c59a7ad3 in ?? ()
88 LWP 6974 "rpc worker-6974" 0x00007f45c59a7ad3 in ?? ()
89 LWP 6975 "rpc worker-6975" 0x00007f45c59a7ad3 in ?? ()
90 LWP 6976 "rpc worker-6976" 0x00007f45c59a7ad3 in ?? ()
91 LWP 6977 "rpc worker-6977" 0x00007f45c59a7ad3 in ?? ()
92 LWP 6978 "rpc worker-6978" 0x00007f45c59a7ad3 in ?? ()
93 LWP 6979 "rpc worker-6979" 0x00007f45c59a7ad3 in ?? ()
94 LWP 6980 "rpc worker-6980" 0x00007f45c59a7ad3 in ?? ()
95 LWP 6981 "rpc worker-6981" 0x00007f45c59a7ad3 in ?? ()
96 LWP 6982 "rpc worker-6982" 0x00007f45c59a7ad3 in ?? ()
97 LWP 6983 "rpc worker-6983" 0x00007f45c59a7ad3 in ?? ()
98 LWP 6984 "rpc worker-6984" 0x00007f45c59a7ad3 in ?? ()
99 LWP 6985 "rpc worker-6985" 0x00007f45c59a7ad3 in ?? ()
100 LWP 6986 "rpc worker-6986" 0x00007f45c59a7ad3 in ?? ()
101 LWP 6987 "rpc worker-6987" 0x00007f45c59a7ad3 in ?? ()
102 LWP 6988 "rpc worker-6988" 0x00007f45c59a7ad3 in ?? ()
103 LWP 6989 "rpc worker-6989" 0x00007f45c59a7ad3 in ?? ()
104 LWP 6990 "rpc worker-6990" 0x00007f45c59a7ad3 in ?? ()
105 LWP 6991 "rpc worker-6991" 0x00007f45c59a7ad3 in ?? ()
106 LWP 6992 "rpc worker-6992" 0x00007f45c59a7ad3 in ?? ()
107 LWP 6993 "rpc worker-6993" 0x00007f45c59a7ad3 in ?? ()
108 LWP 6994 "rpc worker-6994" 0x00007f45c59a7ad3 in ?? ()
109 LWP 6995 "rpc worker-6995" 0x00007f45c59a7ad3 in ?? ()
110 LWP 6996 "rpc worker-6996" 0x00007f45c59a7ad3 in ?? ()
111 LWP 6997 "rpc worker-6997" 0x00007f45c59a7ad3 in ?? ()
112 LWP 6998 "rpc worker-6998" 0x00007f45c59a7ad3 in ?? ()
113 LWP 6999 "rpc worker-6999" 0x00007f45c59a7ad3 in ?? ()
114 LWP 7000 "rpc worker-7000" 0x00007f45c59a7ad3 in ?? ()
115 LWP 7001 "rpc worker-7001" 0x00007f45c59a7ad3 in ?? ()
116 LWP 7002 "rpc worker-7002" 0x00007f45c59a7ad3 in ?? ()
117 LWP 7003 "rpc worker-7003" 0x00007f45c59a7ad3 in ?? ()
118 LWP 7004 "diag-logger-700" 0x00007f45c59a7fb9 in ?? ()
119 LWP 7005 "result-tracker-" 0x00007f45c59a7fb9 in ?? ()
120 LWP 7006 "excess-log-dele" 0x00007f45c59a7fb9 in ?? ()
121 LWP 7007 "acceptor-7007" 0x00007f45c0d25fc7 in ?? ()
122 LWP 7008 "heartbeat-7008" 0x00007f45c59a7fb9 in ?? ()
123 LWP 7009 "maintenance_sch" 0x00007f45c59a7fb9 in ?? ()
124 LWP 7726 "raft [worker]-7" 0x00007f45c59a7fb9 in ?? ()
Thread 124 (LWP 7726):
#0 0x00007f45c59a7fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 123 (LWP 7009):
#0 0x00007f45c59a7fb9 in ?? ()
#1 0x00007b0100000000 in ?? ()
#2 0x00000000000001fd in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007f4579bb96d0 in ?? ()
#6 0x00000000000003fa in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 7008):
#0 0x00007f45c59a7fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 7007):
#0 0x00007f45c0d25fc7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 7006):
#0 0x00007f45c59a7fb9 in ?? ()
#1 0x00007f457b3bc950 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fffe6ec8590 in ?? ()
#5 0x00007f457b3bc7c0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 7005):
#0 0x00007f45c59a7fb9 in ?? ()
#1 0x00007b34000016c0 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001758 in ?? ()
#5 0x00007f457bbbd810 in ?? ()
#6 0x0000000000000102 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 7004):
#0 0x00007f45c59a7fb9 in ?? ()
#1 0x00007f45bec88008 in ?? ()
#2 0x000000000000006d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000590 in ?? ()
#5 0x00007f457c3be760 in ?? ()
#6 0x00000000000000da in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 7003):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 7002):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 7001):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 7000):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 6999):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 6998):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 6997):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 6996):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 6995):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 6994):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 6993):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 6992):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 6991):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 6990):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 6989):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 6988):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 6987):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 6986):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 6985):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 6984):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 6983):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 6982):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 6981):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 6980):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 6979):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 6978):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 6977):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 6976):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 6975):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 6974):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 6973):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 6972):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 6971):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 6970):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 6969):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 6968):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 6967):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 6966):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 6965):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 6964):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 6963):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000006 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b24001147c8 in ?? ()
#4 0x00007f45915ba730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f45915ba750 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 76 (LWP 6962):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 75 (LWP 6961):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 6960):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 6959):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 6958):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 6957):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 6956):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 6955):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 6954):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 6953):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 6952):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 6951):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 6950):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 6949):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 6948):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 6947):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 6946):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 6945):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 6944):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 6943):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000b902c in ?? ()
#4 0x00007f459b9bc730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f459b9bc750 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x007f0400000026b3 in ?? ()
#9 0x00007f45c59a7770 in ?? ()
#10 0x00007f459b9bc750 in ?? ()
#11 0x0002008300000dfe in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 6942):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 6941):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 6940):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 6939):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 6938):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 6937):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 6936):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 6935):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 6934):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 6933):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 6932):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 6931):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 6930):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 6929):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 6928):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 6927):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 6926):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 6925):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 6924):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 6923):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x000000000000008f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240005ffec in ?? ()
#4 0x00007f45a5dbe730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f45a5dbe750 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x0000000000471b79 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f45c59a7770 in ?? ()
#10 0x00007f45a5dbe750 in ?? ()
#11 0x00007f45af370290 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 36 (LWP 6922):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x00000000000003e4 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240005d7f8 in ?? ()
#4 0x00007f45a67b6730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f45a67b6750 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 6921):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000263 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b2400058ffc in ?? ()
#4 0x00007f45a6fb7730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f45a6fb7750 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x0000000000471b79 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f45c59a7770 in ?? ()
#10 0x00007f45a6fb7750 in ?? ()
#11 0x00007f45859b8c98 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 34 (LWP 6920):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b24000547f8 in ?? ()
#4 0x00007f45a77b8730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f45a77b8750 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 33 (LWP 6919):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 6918):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 6917):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 6916):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 6915):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 6914):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 6913):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 6912):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 6911):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 6910):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 6909):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 6908):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 6907):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 6906):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 6905):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 6904):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 6903):
#0 0x00007f45c59a7fb9 in ?? ()
#1 0x00007f45b0394ab8 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800004000 in ?? ()
#5 0x00007f45b0394710 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 6902):
#0 0x00007f45c59a7fb9 in ?? ()
#1 0x00007f45b0b959b8 in ?? ()
#2 0x0000000000000019 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000345d8 in ?? ()
#5 0x00007f45b0b95850 in ?? ()
#6 0x0000000000000032 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 6901):
#0 0x00007f45c59a7fb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007f45b1396420 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 6900):
#0 0x00007f45c59a7ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 6899):
#0 0x00007f45c0d24947 in ?? ()
#1 0x00007f45b2398560 in ?? ()
#2 0x0048e00002ffe1e2 in ?? ()
#3 0x00007f45b2398510 in ?? ()
#4 0x00007f45b2399b80 in ?? ()
#5 0x00007f45b2398510 in ?? ()
#6 0x0000000000000012 in ?? ()
#7 0x00007b5800001b00 in ?? ()
#8 0x000000000049bd45 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f45be486000 in ?? ()
#10 0x000000000049bc49 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f45b2399b80 in ?? ()
#12 0x00007f45c377d069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f45c8fa34c0 in ?? ()
#15 0x00007b4c000034d0 in ?? ()
#16 0x00007b4c000034d8 in ?? ()
#17 0x00007f45b23987b0 in ?? ()
#18 0x00007b4400035890 in ?? ()
#19 0x00007f45b2398ce0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 12 (LWP 6898):
#0 0x00007f45c0d24947 in ?? ()
#1 0x00007f45b2b99560 in ?? ()
#2 0x0044e0000010bcae in ?? ()
#3 0x00007f45b2b99510 in ?? ()
#4 0x00007f45b2b9ab80 in ?? ()
#5 0x00007f45b2b99510 in ?? ()
#6 0x0000000000000011 in ?? ()
#7 0x00007b5800001800 in ?? ()
#8 0x000000000049bd45 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f45be498000 in ?? ()
#10 0x000000000049bc49 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f45b2b9ab80 in ?? ()
#12 0x00007f45c377d069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f45c8fa34c0 in ?? ()
#15 0x00007b4c00002f90 in ?? ()
#16 0x00007b4c00002f98 in ?? ()
#17 0x00007f45b2b997b0 in ?? ()
#18 0x00007b44000368d0 in ?? ()
#19 0x00007f45b2b99ce0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 11 (LWP 6897):
#0 0x00007f45c0d24947 in ?? ()
#1 0x00007f45b339a560 in ?? ()
#2 0x0040e00000d7889d in ?? ()
#3 0x00007f45b339a510 in ?? ()
#4 0x00007f45b339bb80 in ?? ()
#5 0x00007f45b339a510 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x00007b5800001200 in ?? ()
#8 0x000000000049bd45 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f45be4aa000 in ?? ()
#10 0x000000000049bc49 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f45b339bb80 in ?? ()
#12 0x00007f45c377d069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f45c8fa34c0 in ?? ()
#15 0x00007b4c000026d0 in ?? ()
#16 0x00007b4c000026d8 in ?? ()
#17 0x00007f45b339a7b0 in ?? ()
#18 0x00007b4400035c50 in ?? ()
#19 0x00007f45b339ace0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 10 (LWP 6896):
#0 0x00007f45c0d24947 in ?? ()
#1 0x00007f45b63be560 in ?? ()
#2 0x003ce000001719fe in ?? ()
#3 0x00007f45b63be510 in ?? ()
#4 0x00007f45b63bfb80 in ?? ()
#5 0x00007f45b63be510 in ?? ()
#6 0x000000000000000d in ?? ()
#7 0x00007b5800000f00 in ?? ()
#8 0x000000000049bd45 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f45be4bc000 in ?? ()
#10 0x000000000049bc49 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f45b63bfb80 in ?? ()
#12 0x00007f45c377d069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f45c8fa34c0 in ?? ()
#15 0x00007b4c00002c10 in ?? ()
#16 0x00007b4c00002c18 in ?? ()
#17 0x00007f45b63be7b0 in ?? ()
#18 0x00007b4400037050 in ?? ()
#19 0x00007f45b63bece0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 9 (LWP 6893):
#0 0x00007f45c0d17bb9 in ?? ()
#1 0x00007f45b9bbcbf0 in ?? ()
#2 0x00007b040000a830 in ?? ()
#3 0x00007f45b9bbdb80 in ?? ()
#4 0x00007f45b9bbcbf0 in ?? ()
#5 0x00007b040000a830 in ?? ()
#6 0x000000000049bf53 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007f45be612000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007f45b9bbdb80 in ?? ()
#10 0x00007f45ca797f2d in ?? ()
#11 0x00007f45ca790000 in ?? ()
#12 0x00000000004a975d in __sanitizer::internal_alloc_placeholder ()
#13 0x0000000000000000 in ?? ()
Thread 8 (LWP 6892):
#0 0x00007f45c59a7fb9 in ?? ()
#1 0x000000000bd62290 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400034518 in ?? ()
#5 0x00007f45b93bb800 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 6891):
#0 0x00007f45c59ab9e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 6883):
#0 0x00007f45c59a7fb9 in ?? ()
#1 0x00007f45babbea50 in ?? ()
#2 0x0000000000000281 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400037718 in ?? ()
#5 0x00007f45babbe5e0 in ?? ()
#6 0x0000000000000502 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 6882):
#0 0x00007f45c59a7fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 6881):
#0 0x00007f45c59a7fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 6880):
#0 0x00007f45c59a7fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 6879):
#0 0x00007f45c0ce7680 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 6878):
#0 0x00007f45c59abd50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x000000000047b1db in __sanitizer::internal_alloc_placeholder ()
#3 0x00007f45bff45cc0 in ?? ()
#4 0x00007f45bff45cc0 in ?? ()
#5 0x00007fffe6ec83a0 in ?? ()
#6 0x000000000049e5a4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000b151c0 in ?? ()
#9 0x00007f45bff45cc0 in ?? ()
#10 0x00007f45c3e4bf0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20260505 13:59:56.514349 6116 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID b11284b44f424719ba754e92c1af3e33 and pid 7012
************************ BEGIN STACKS **************************
[New LWP 7013]
[New LWP 7014]
[New LWP 7015]
[New LWP 7016]
[New LWP 7017]
[New LWP 7024]
[New LWP 7025]
[New LWP 7026]
[New LWP 7029]
[New LWP 7030]
[New LWP 7031]
[New LWP 7032]
[New LWP 7033]
[New LWP 7034]
[New LWP 7035]
[New LWP 7036]
[New LWP 7037]
[New LWP 7038]
[New LWP 7039]
[New LWP 7040]
[New LWP 7041]
[New LWP 7042]
[New LWP 7043]
[New LWP 7044]
[New LWP 7045]
[New LWP 7046]
[New LWP 7047]
[New LWP 7048]
[New LWP 7049]
[New LWP 7050]
[New LWP 7051]
[New LWP 7052]
[New LWP 7053]
[New LWP 7054]
[New LWP 7055]
[New LWP 7056]
[New LWP 7057]
[New LWP 7058]
[New LWP 7059]
[New LWP 7060]
[New LWP 7061]
[New LWP 7062]
[New LWP 7063]
[New LWP 7064]
[New LWP 7065]
[New LWP 7066]
[New LWP 7067]
[New LWP 7068]
[New LWP 7069]
[New LWP 7070]
[New LWP 7071]
[New LWP 7072]
[New LWP 7073]
[New LWP 7074]
[New LWP 7075]
[New LWP 7076]
[New LWP 7077]
[New LWP 7078]
[New LWP 7079]
[New LWP 7080]
[New LWP 7081]
[New LWP 7082]
[New LWP 7083]
[New LWP 7084]
[New LWP 7085]
[New LWP 7086]
[New LWP 7087]
[New LWP 7088]
[New LWP 7089]
[New LWP 7090]
[New LWP 7091]
[New LWP 7092]
[New LWP 7093]
[New LWP 7094]
[New LWP 7095]
[New LWP 7096]
[New LWP 7097]
[New LWP 7098]
[New LWP 7099]
[New LWP 7100]
[New LWP 7101]
[New LWP 7102]
[New LWP 7103]
[New LWP 7104]
[New LWP 7105]
[New LWP 7106]
[New LWP 7107]
[New LWP 7108]
[New LWP 7109]
[New LWP 7110]
[New LWP 7111]
[New LWP 7112]
[New LWP 7113]
[New LWP 7114]
[New LWP 7115]
[New LWP 7116]
[New LWP 7117]
[New LWP 7118]
[New LWP 7119]
[New LWP 7120]
[New LWP 7121]
[New LWP 7122]
[New LWP 7123]
[New LWP 7124]
[New LWP 7125]
[New LWP 7126]
[New LWP 7127]
[New LWP 7128]
[New LWP 7129]
[New LWP 7130]
[New LWP 7131]
[New LWP 7132]
[New LWP 7133]
[New LWP 7134]
[New LWP 7135]
[New LWP 7136]
[New LWP 7137]
[New LWP 7138]
[New LWP 7139]
[New LWP 7140]
[New LWP 7141]
[New LWP 7142]
Cannot access memory at address 0xc4108070c6b0396
Cannot access memory at address 0xc4108070c6b038e
Cannot access memory at address 0xc4108070c6b0396
Cannot access memory at address 0xc4108070c6b0396
Cannot access memory at address 0xc4108070c6b038e
0x00007f7cb5ecad50 in ?? ()
Id Target Id Frame
* 1 LWP 7012 "kudu" 0x00007f7cb5ecad50 in ?? ()
2 LWP 7013 "kudu" 0x00007f7cb1206680 in ?? ()
3 LWP 7014 "kudu" 0x00007f7cb5ec6fb9 in ?? ()
4 LWP 7015 "kudu" 0x00007f7cb5ec6fb9 in ?? ()
5 LWP 7016 "kudu" 0x00007f7cb5ec6fb9 in ?? ()
6 LWP 7017 "kernel-watcher-" 0x00007f7cb5ec6fb9 in ?? ()
7 LWP 7024 "ntp client-7024" 0x00007f7cb5eca9e2 in ?? ()
8 LWP 7025 "file cache-evic" 0x00007f7cb5ec6fb9 in ?? ()
9 LWP 7026 "sq_acceptor" 0x00007f7cb1236bb9 in ?? ()
10 LWP 7029 "rpc reactor-702" 0x00007f7cb1243947 in ?? ()
11 LWP 7030 "rpc reactor-703" 0x00007f7cb1243947 in ?? ()
12 LWP 7031 "rpc reactor-703" 0x00007f7cb1243947 in ?? ()
13 LWP 7032 "rpc reactor-703" 0x00007f7cb1243947 in ?? ()
14 LWP 7033 "MaintenanceMgr " 0x00007f7cb5ec6ad3 in ?? ()
15 LWP 7034 "txn-status-mana" 0x00007f7cb5ec6fb9 in ?? ()
16 LWP 7035 "collect_and_rem" 0x00007f7cb5ec6fb9 in ?? ()
17 LWP 7036 "tc-session-exp-" 0x00007f7cb5ec6fb9 in ?? ()
18 LWP 7037 "rpc worker-7037" 0x00007f7cb5ec6ad3 in ?? ()
19 LWP 7038 "rpc worker-7038" 0x00007f7cb5ec6ad3 in ?? ()
20 LWP 7039 "rpc worker-7039" 0x00007f7cb5ec6ad3 in ?? ()
21 LWP 7040 "rpc worker-7040" 0x00007f7cb5ec6ad3 in ?? ()
22 LWP 7041 "rpc worker-7041" 0x00007f7cb5ec6ad3 in ?? ()
23 LWP 7042 "rpc worker-7042" 0x00007f7cb5ec6ad3 in ?? ()
24 LWP 7043 "rpc worker-7043" 0x00007f7cb5ec6ad3 in ?? ()
25 LWP 7044 "rpc worker-7044" 0x00007f7cb5ec6ad3 in ?? ()
26 LWP 7045 "rpc worker-7045" 0x00007f7cb5ec6ad3 in ?? ()
27 LWP 7046 "rpc worker-7046" 0x00007f7cb5ec6ad3 in ?? ()
28 LWP 7047 "rpc worker-7047" 0x00007f7cb5ec6ad3 in ?? ()
29 LWP 7048 "rpc worker-7048" 0x00007f7cb5ec6ad3 in ?? ()
30 LWP 7049 "rpc worker-7049" 0x00007f7cb5ec6ad3 in ?? ()
31 LWP 7050 "rpc worker-7050" 0x00007f7cb5ec6ad3 in ?? ()
32 LWP 7051 "rpc worker-7051" 0x00007f7cb5ec6ad3 in ?? ()
33 LWP 7052 "rpc worker-7052" 0x00007f7cb5ec6ad3 in ?? ()
34 LWP 7053 "rpc worker-7053" 0x00007f7cb5ec6ad3 in ?? ()
35 LWP 7054 "rpc worker-7054" 0x00007f7cb5ec6ad3 in ?? ()
36 LWP 7055 "rpc worker-7055" 0x00007f7cb5ec6ad3 in ?? ()
37 LWP 7056 "rpc worker-7056" 0x00007f7cb5ec6ad3 in ?? ()
38 LWP 7057 "rpc worker-7057" 0x00007f7cb5ec6ad3 in ?? ()
39 LWP 7058 "rpc worker-7058" 0x00007f7cb5ec6ad3 in ?? ()
40 LWP 7059 "rpc worker-7059" 0x00007f7cb5ec6ad3 in ?? ()
41 LWP 7060 "rpc worker-7060" 0x00007f7cb5ec6ad3 in ?? ()
42 LWP 7061 "rpc worker-7061" 0x00007f7cb5ec6ad3 in ?? ()
43 LWP 7062 "rpc worker-7062" 0x00007f7cb5ec6ad3 in ?? ()
44 LWP 7063 "rpc worker-7063" 0x00007f7cb5ec6ad3 in ?? ()
45 LWP 7064 "rpc worker-7064" 0x00007f7cb5ec6ad3 in ?? ()
46 LWP 7065 "rpc worker-7065" 0x00007f7cb5ec6ad3 in ?? ()
47 LWP 7066 "rpc worker-7066" 0x00007f7cb5ec6ad3 in ?? ()
48 LWP 7067 "rpc worker-7067" 0x00007f7cb5ec6ad3 in ?? ()
49 LWP 7068 "rpc worker-7068" 0x00007f7cb5ec6ad3 in ?? ()
50 LWP 7069 "rpc worker-7069" 0x00007f7cb5ec6ad3 in ?? ()
51 LWP 7070 "rpc worker-7070" 0x00007f7cb5ec6ad3 in ?? ()
52 LWP 7071 "rpc worker-7071" 0x00007f7cb5ec6ad3 in ?? ()
53 LWP 7072 "rpc worker-7072" 0x00007f7cb5ec6ad3 in ?? ()
54 LWP 7073 "rpc worker-7073" 0x00007f7cb5ec6ad3 in ?? ()
55 LWP 7074 "rpc worker-7074" 0x00007f7cb5ec6ad3 in ?? ()
56 LWP 7075 "rpc worker-7075" 0x00007f7cb5ec6ad3 in ?? ()
57 LWP 7076 "rpc worker-7076" 0x00007f7cb5ec6ad3 in ?? ()
58 LWP 7077 "rpc worker-7077" 0x00007f7cb5ec6ad3 in ?? ()
59 LWP 7078 "rpc worker-7078" 0x00007f7cb5ec6ad3 in ?? ()
60 LWP 7079 "rpc worker-7079" 0x00007f7cb5ec6ad3 in ?? ()
61 LWP 7080 "rpc worker-7080" 0x00007f7cb5ec6ad3 in ?? ()
62 LWP 7081 "rpc worker-7081" 0x00007f7cb5ec6ad3 in ?? ()
63 LWP 7082 "rpc worker-7082" 0x00007f7cb5ec6ad3 in ?? ()
64 LWP 7083 "rpc worker-7083" 0x00007f7cb5ec6ad3 in ?? ()
65 LWP 7084 "rpc worker-7084" 0x00007f7cb5ec6ad3 in ?? ()
66 LWP 7085 "rpc worker-7085" 0x00007f7cb5ec6ad3 in ?? ()
67 LWP 7086 "rpc worker-7086" 0x00007f7cb5ec6ad3 in ?? ()
68 LWP 7087 "rpc worker-7087" 0x00007f7cb5ec6ad3 in ?? ()
69 LWP 7088 "rpc worker-7088" 0x00007f7cb5ec6ad3 in ?? ()
70 LWP 7089 "rpc worker-7089" 0x00007f7cb5ec6ad3 in ?? ()
71 LWP 7090 "rpc worker-7090" 0x00007f7cb5ec6ad3 in ?? ()
72 LWP 7091 "rpc worker-7091" 0x00007f7cb5ec6ad3 in ?? ()
73 LWP 7092 "rpc worker-7092" 0x00007f7cb5ec6ad3 in ?? ()
74 LWP 7093 "rpc worker-7093" 0x00007f7cb5ec6ad3 in ?? ()
75 LWP 7094 "rpc worker-7094" 0x00007f7cb5ec6ad3 in ?? ()
76 LWP 7095 "rpc worker-7095" 0x00007f7cb5ec6ad3 in ?? ()
77 LWP 7096 "rpc worker-7096" 0x00007f7cb5ec6ad3 in ?? ()
78 LWP 7097 "rpc worker-7097" 0x00007f7cb5ec6ad3 in ?? ()
79 LWP 7098 "rpc worker-7098" 0x00007f7cb5ec6ad3 in ?? ()
80 LWP 7099 "rpc worker-7099" 0x00007f7cb5ec6ad3 in ?? ()
81 LWP 7100 "rpc worker-7100" 0x00007f7cb5ec6ad3 in ?? ()
82 LWP 7101 "rpc worker-7101" 0x00007f7cb5ec6ad3 in ?? ()
83 LWP 7102 "rpc worker-7102" 0x00007f7cb5ec6ad3 in ?? ()
84 LWP 7103 "rpc worker-7103" 0x00007f7cb5ec6ad3 in ?? ()
85 LWP 7104 "rpc worker-7104" 0x00007f7cb5ec6ad3 in ?? ()
86 LWP 7105 "rpc worker-7105" 0x00007f7cb5ec6ad3 in ?? ()
87 LWP 7106 "rpc worker-7106" 0x00007f7cb5ec6ad3 in ?? ()
88 LWP 7107 "rpc worker-7107" 0x00007f7cb5ec6ad3 in ?? ()
89 LWP 7108 "rpc worker-7108" 0x00007f7cb5ec6ad3 in ?? ()
90 LWP 7109 "rpc worker-7109" 0x00007f7cb5ec6ad3 in ?? ()
91 LWP 7110 "rpc worker-7110" 0x00007f7cb5ec6ad3 in ?? ()
92 LWP 7111 "rpc worker-7111" 0x00007f7cb5ec6ad3 in ?? ()
93 LWP 7112 "rpc worker-7112" 0x00007f7cb5ec6ad3 in ?? ()
94 LWP 7113 "rpc worker-7113" 0x00007f7cb5ec6ad3 in ?? ()
95 LWP 7114 "rpc worker-7114" 0x00007f7cb5ec6ad3 in ?? ()
96 LWP 7115 "rpc worker-7115" 0x00007f7cb5ec6ad3 in ?? ()
97 LWP 7116 "rpc worker-7116" 0x00007f7cb5ec6ad3 in ?? ()
98 LWP 7117 "rpc worker-7117" 0x00007f7cb5ec6ad3 in ?? ()
99 LWP 7118 "rpc worker-7118" 0x00007f7cb5ec6ad3 in ?? ()
100 LWP 7119 "rpc worker-7119" 0x00007f7cb5ec6ad3 in ?? ()
101 LWP 7120 "rpc worker-7120" 0x00007f7cb5ec6ad3 in ?? ()
102 LWP 7121 "rpc worker-7121" 0x00007f7cb5ec6ad3 in ?? ()
103 LWP 7122 "rpc worker-7122" 0x00007f7cb5ec6ad3 in ?? ()
104 LWP 7123 "rpc worker-7123" 0x00007f7cb5ec6ad3 in ?? ()
105 LWP 7124 "rpc worker-7124" 0x00007f7cb5ec6ad3 in ?? ()
106 LWP 7125 "rpc worker-7125" 0x00007f7cb5ec6ad3 in ?? ()
107 LWP 7126 "rpc worker-7126" 0x00007f7cb5ec6ad3 in ?? ()
108 LWP 7127 "rpc worker-7127" 0x00007f7cb5ec6ad3 in ?? ()
109 LWP 7128 "rpc worker-7128" 0x00007f7cb5ec6ad3 in ?? ()
110 LWP 7129 "rpc worker-7129" 0x00007f7cb5ec6ad3 in ?? ()
111 LWP 7130 "rpc worker-7130" 0x00007f7cb5ec6ad3 in ?? ()
112 LWP 7131 "rpc worker-7131" 0x00007f7cb5ec6ad3 in ?? ()
113 LWP 7132 "rpc worker-7132" 0x00007f7cb5ec6ad3 in ?? ()
114 LWP 7133 "rpc worker-7133" 0x00007f7cb5ec6ad3 in ?? ()
115 LWP 7134 "rpc worker-7134" 0x00007f7cb5ec6ad3 in ?? ()
116 LWP 7135 "rpc worker-7135" 0x00007f7cb5ec6ad3 in ?? ()
117 LWP 7136 "rpc worker-7136" 0x00007f7cb5ec6ad3 in ?? ()
118 LWP 7137 "diag-logger-713" 0x00007f7cb5ec6fb9 in ?? ()
119 LWP 7138 "result-tracker-" 0x00007f7cb5ec6fb9 in ?? ()
120 LWP 7139 "excess-log-dele" 0x00007f7cb5ec6fb9 in ?? ()
121 LWP 7140 "acceptor-7140" 0x00007f7cb1244fc7 in ?? ()
122 LWP 7141 "heartbeat-7141" 0x00007f7cb5ec6fb9 in ?? ()
123 LWP 7142 "maintenance_sch" 0x00007f7cb5ec6fb9 in ?? ()
Thread 123 (LWP 7142):
#0 0x00007f7cb5ec6fb9 in ?? ()
#1 0x00007b0100000000 in ?? ()
#2 0x00000000000001f9 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007f7c6a0b96d0 in ?? ()
#6 0x00000000000003f2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 7141):
#0 0x00007f7cb5ec6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 7140):
#0 0x00007f7cb1244fc7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 7139):
#0 0x00007f7cb5ec6fb9 in ?? ()
#1 0x00007f7c6b8bc950 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff198b1460 in ?? ()
#5 0x00007f7c6b8bc7c0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 7138):
#0 0x00007f7cb5ec6fb9 in ?? ()
#1 0x00007b34000016c0 in ?? ()
#2 0x0000000000000080 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001758 in ?? ()
#5 0x00007f7c6c0bd810 in ?? ()
#6 0x0000000000000100 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 7137):
#0 0x00007f7cb5ec6fb9 in ?? ()
#1 0x00007f7caf188008 in ?? ()
#2 0x000000000000006d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000990 in ?? ()
#5 0x00007f7c6c8be760 in ?? ()
#6 0x00000000000000da in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 7136):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 7135):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 7134):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 7133):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 7132):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 7131):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 7130):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 7129):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 7128):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 7127):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 7126):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 7125):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 7124):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 7123):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 7122):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 7121):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 7120):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 7119):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 7118):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 7117):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 7116):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 7115):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 7114):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 7113):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 7112):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 7111):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 7110):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 7109):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 7108):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 7107):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 7106):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 7105):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 7104):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 7103):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 7102):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 7101):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 7100):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 7099):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 7098):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 7097):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 7096):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000596 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b2400118fc8 in ?? ()
#4 0x00007f7c81aba730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7c81aba750 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 76 (LWP 7095):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000507 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24001147cc in ?? ()
#4 0x00007f7c822bb730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7c822bb750 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x0000000000471b79 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f7cb5ec6770 in ?? ()
#10 0x00007f7c822bb750 in ?? ()
#11 0x00007f7c664a3680 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 7094):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000263 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240010ffcc in ?? ()
#4 0x00007f7c82abc730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7c82abc750 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x0000000000000000 in ?? ()
Thread 74 (LWP 7093):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 7092):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 7091):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 7090):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 7089):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 7088):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 7087):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 7086):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 7085):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 7084):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 7083):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 7082):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 7081):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 7080):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 7079):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 7078):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 7077):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 7076):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000c001c in ?? ()
#4 0x00007f7c8bebc730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7c8bebc750 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x007f0400000026b3 in ?? ()
#9 0x00007f7cb5ec6770 in ?? ()
#10 0x00007f7c8bebc750 in ?? ()
#11 0x0002008300000dfe in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 7075):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 7074):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 7073):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 7072):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 7071):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 7070):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 7069):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 7068):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 7067):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 7066):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 7065):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 7064):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 7063):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 7062):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 7061):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 7060):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 7059):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 7058):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 7057):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 7056):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000647ec in ?? ()
#4 0x00007f7c962be730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7c962be750 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x0000000000471b79 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f7cb5ec6770 in ?? ()
#10 0x00007f7c962be750 in ?? ()
#11 0x00007f7cae29fc48 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 36 (LWP 7055):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 7054):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 7053):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 7052):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 7051):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 7050):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 7049):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 7048):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 7047):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 7046):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 7045):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 7044):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 7043):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 7042):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 7041):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 7040):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 7039):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 7038):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 7037):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 7036):
#0 0x00007f7cb5ec6fb9 in ?? ()
#1 0x00007f7ca0894ab8 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800004000 in ?? ()
#5 0x00007f7ca0894710 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 7035):
#0 0x00007f7cb5ec6fb9 in ?? ()
#1 0x00007f7ca10959b8 in ?? ()
#2 0x0000000000000019 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000345d8 in ?? ()
#5 0x00007f7ca1095850 in ?? ()
#6 0x0000000000000032 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 7034):
#0 0x00007f7cb5ec6fb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007f7ca1896420 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 7033):
#0 0x00007f7cb5ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 7032):
#0 0x00007f7cb1243947 in ?? ()
#1 0x00007f7ca2898560 in ?? ()
#2 0x0048e000000b0b34 in ?? ()
#3 0x00007f7ca2898510 in ?? ()
#4 0x00007f7ca2899b80 in ?? ()
#5 0x00007f7ca2898510 in ?? ()
#6 0x0000000000000016 in ?? ()
#7 0x00007b5800001b00 in ?? ()
#8 0x000000000049bd45 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f7cae998000 in ?? ()
#10 0x000000000049bc49 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f7ca2899b80 in ?? ()
#12 0x00007f7cb3c9c069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f7cb94c24c0 in ?? ()
#15 0x00007b4c000034d0 in ?? ()
#16 0x00007b4c000034d8 in ?? ()
#17 0x00007f7ca28987b0 in ?? ()
#18 0x00007b4400035890 in ?? ()
#19 0x00007f7ca2898ce0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 12 (LWP 7031):
#0 0x00007f7cb1243947 in ?? ()
#1 0x00007f7ca3099560 in ?? ()
#2 0x0044e000015798d8 in ?? ()
#3 0x00007f7ca3099510 in ?? ()
#4 0x00007f7ca309ab80 in ?? ()
#5 0x00007f7ca3099510 in ?? ()
#6 0x0000000000000015 in ?? ()
#7 0x00007b5800001800 in ?? ()
#8 0x000000000049bd45 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f7cae9aa000 in ?? ()
#10 0x000000000049bc49 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f7ca309ab80 in ?? ()
#12 0x00007f7cb3c9c069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f7cb94c24c0 in ?? ()
#15 0x00007b4c00002f90 in ?? ()
#16 0x00007b4c00002f98 in ?? ()
#17 0x00007f7ca30997b0 in ?? ()
#18 0x00007b44000368d0 in ?? ()
#19 0x00007f7ca3099ce0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 11 (LWP 7030):
#0 0x00007f7cb1243947 in ?? ()
#1 0x00007f7ca389a560 in ?? ()
#2 0x0040e00000092a48 in ?? ()
#3 0x00007f7ca389a510 in ?? ()
#4 0x00007f7ca389bb80 in ?? ()
#5 0x00007f7ca389a510 in ?? ()
#6 0x0000000000000014 in ?? ()
#7 0x00007b5800001200 in ?? ()
#8 0x000000000049bd45 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f7cae9bc000 in ?? ()
#10 0x000000000049bc49 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f7ca389bb80 in ?? ()
#12 0x00007f7cb3c9c069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f7cb94c24c0 in ?? ()
#15 0x00007b4c000026d0 in ?? ()
#16 0x00007b4c000026d8 in ?? ()
#17 0x00007f7ca389a7b0 in ?? ()
#18 0x00007b4400035c50 in ?? ()
#19 0x00007f7ca389ace0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 10 (LWP 7029):
#0 0x00007f7cb1243947 in ?? ()
#1 0x00007f7ca68be560 in ?? ()
#2 0x003ce00000188123 in ?? ()
#3 0x00007f7ca68be510 in ?? ()
#4 0x00007f7ca68bfb80 in ?? ()
#5 0x00007f7ca68be510 in ?? ()
#6 0x0000000000000011 in ?? ()
#7 0x00007b5800000f00 in ?? ()
#8 0x000000000049bd45 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f7cae9ce000 in ?? ()
#10 0x000000000049bc49 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f7ca68bfb80 in ?? ()
#12 0x00007f7cb3c9c069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f7cb94c24c0 in ?? ()
#15 0x00007b4c00002c10 in ?? ()
#16 0x00007b4c00002c18 in ?? ()
#17 0x00007f7ca68be7b0 in ?? ()
#18 0x00007b4400037050 in ?? ()
#19 0x00007f7ca68bece0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 9 (LWP 7026):
#0 0x00007f7cb1236bb9 in ?? ()
#1 0x00007f7caa0bcbf0 in ?? ()
#2 0x00007b040000a830 in ?? ()
#3 0x00007f7caa0bdb80 in ?? ()
#4 0x00007f7caa0bcbf0 in ?? ()
#5 0x00007b040000a830 in ?? ()
#6 0x000000000049bf53 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007f7caeb5a000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007f7caa0bdb80 in ?? ()
#10 0x00007f7cbacb6f2d in ?? ()
#11 0x00007f7cbacb0000 in ?? ()
#12 0x00000000004a975d in __sanitizer::internal_alloc_placeholder ()
#13 0x0000000000000000 in ?? ()
Thread 8 (LWP 7025):
#0 0x00007f7cb5ec6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 7 (LWP 7024):
#0 0x00007f7cb5eca9e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 7017):
#0 0x00007f7cb5ec6fb9 in ?? ()
#1 0x00007f7cab0bea50 in ?? ()
#2 0x0000000000000275 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400037718 in ?? ()
#5 0x00007f7cab0be5e0 in ?? ()
#6 0x00000000000004ea in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 7016):
#0 0x00007f7cb5ec6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 7015):
#0 0x00007f7cb5ec6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 7014):
#0 0x00007f7cb5ec6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 7013):
#0 0x00007f7cb1206680 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 7012):
#0 0x00007f7cb5ecad50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x000000000047b1db in __sanitizer::internal_alloc_placeholder ()
#3 0x00007f7cb0464cc0 in ?? ()
#4 0x00007f7cb0464cc0 in ?? ()
#5 0x00007fff198b1270 in ?? ()
#6 0x000000000049e5a4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000b0f86b in ?? ()
#9 0x00007f7cb0464cc0 in ?? ()
#10 0x00007f7cb436af0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20260505 13:59:57.568051 6116 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID 7f7d3baeb7b54d7fa07d485e506e97d2 and pid 7145
************************ BEGIN STACKS **************************
[New LWP 7146]
[New LWP 7147]
[New LWP 7148]
[New LWP 7149]
[New LWP 7150]
[New LWP 7157]
[New LWP 7158]
[New LWP 7159]
[New LWP 7162]
[New LWP 7163]
[New LWP 7164]
[New LWP 7165]
[New LWP 7166]
[New LWP 7167]
[New LWP 7168]
[New LWP 7169]
[New LWP 7170]
[New LWP 7171]
[New LWP 7172]
[New LWP 7173]
[New LWP 7174]
[New LWP 7175]
[New LWP 7176]
[New LWP 7177]
[New LWP 7178]
[New LWP 7179]
[New LWP 7180]
[New LWP 7181]
[New LWP 7182]
[New LWP 7183]
[New LWP 7184]
[New LWP 7185]
[New LWP 7186]
[New LWP 7187]
[New LWP 7188]
[New LWP 7189]
[New LWP 7190]
[New LWP 7191]
[New LWP 7192]
[New LWP 7193]
[New LWP 7194]
[New LWP 7195]
[New LWP 7196]
[New LWP 7197]
[New LWP 7198]
[New LWP 7199]
[New LWP 7200]
[New LWP 7201]
[New LWP 7202]
[New LWP 7203]
[New LWP 7204]
[New LWP 7205]
[New LWP 7206]
[New LWP 7207]
[New LWP 7208]
[New LWP 7209]
[New LWP 7210]
[New LWP 7211]
[New LWP 7212]
[New LWP 7213]
[New LWP 7214]
[New LWP 7215]
[New LWP 7216]
[New LWP 7217]
[New LWP 7218]
[New LWP 7219]
[New LWP 7220]
[New LWP 7221]
[New LWP 7222]
[New LWP 7223]
[New LWP 7224]
[New LWP 7225]
[New LWP 7226]
[New LWP 7227]
[New LWP 7228]
[New LWP 7229]
[New LWP 7230]
[New LWP 7231]
[New LWP 7232]
[New LWP 7233]
[New LWP 7234]
[New LWP 7235]
[New LWP 7236]
[New LWP 7237]
[New LWP 7238]
[New LWP 7239]
[New LWP 7240]
[New LWP 7241]
[New LWP 7242]
[New LWP 7243]
[New LWP 7244]
[New LWP 7245]
[New LWP 7246]
[New LWP 7247]
[New LWP 7248]
[New LWP 7249]
[New LWP 7250]
[New LWP 7251]
[New LWP 7252]
[New LWP 7253]
[New LWP 7254]
[New LWP 7255]
[New LWP 7256]
[New LWP 7257]
[New LWP 7258]
[New LWP 7259]
[New LWP 7260]
[New LWP 7261]
[New LWP 7262]
[New LWP 7263]
[New LWP 7264]
[New LWP 7265]
[New LWP 7266]
[New LWP 7267]
[New LWP 7268]
[New LWP 7269]
[New LWP 7270]
[New LWP 7271]
[New LWP 7272]
[New LWP 7273]
[New LWP 7274]
[New LWP 7275]
Cannot access memory at address 0xc4108070c6b0396
Cannot access memory at address 0xc4108070c6b038e
Cannot access memory at address 0xc4108070c6b0396
Cannot access memory at address 0xc4108070c6b0396
Cannot access memory at address 0xc4108070c6b038e
0x00007f39b81a6d50 in ?? ()
Id Target Id Frame
* 1 LWP 7145 "kudu" 0x00007f39b81a6d50 in ?? ()
2 LWP 7146 "kudu" 0x00007f39b34e2680 in ?? ()
3 LWP 7147 "kudu" 0x00007f39b81a2fb9 in ?? ()
4 LWP 7148 "kudu" 0x00007f39b81a2fb9 in ?? ()
5 LWP 7149 "kudu" 0x00007f39b81a2fb9 in ?? ()
6 LWP 7150 "kernel-watcher-" 0x00007f39b81a2fb9 in ?? ()
7 LWP 7157 "ntp client-7157" 0x00007f39b81a69e2 in ?? ()
8 LWP 7158 "file cache-evic" 0x00007f39b81a2fb9 in ?? ()
9 LWP 7159 "sq_acceptor" 0x00007f39b3512bb9 in ?? ()
10 LWP 7162 "rpc reactor-716" 0x00007f39b351f947 in ?? ()
11 LWP 7163 "rpc reactor-716" 0x00007f39b351f947 in ?? ()
12 LWP 7164 "rpc reactor-716" 0x00007f39b351f947 in ?? ()
13 LWP 7165 "rpc reactor-716" 0x00007f39b351f947 in ?? ()
14 LWP 7166 "MaintenanceMgr " 0x00007f39b81a2ad3 in ?? ()
15 LWP 7167 "txn-status-mana" 0x00007f39b81a2fb9 in ?? ()
16 LWP 7168 "collect_and_rem" 0x00007f39b81a2fb9 in ?? ()
17 LWP 7169 "tc-session-exp-" 0x00007f39b81a2fb9 in ?? ()
18 LWP 7170 "rpc worker-7170" 0x00007f39b81a2ad3 in ?? ()
19 LWP 7171 "rpc worker-7171" 0x00007f39b81a2ad3 in ?? ()
20 LWP 7172 "rpc worker-7172" 0x00007f39b81a2ad3 in ?? ()
21 LWP 7173 "rpc worker-7173" 0x00007f39b81a2ad3 in ?? ()
22 LWP 7174 "rpc worker-7174" 0x00007f39b81a2ad3 in ?? ()
23 LWP 7175 "rpc worker-7175" 0x00007f39b81a2ad3 in ?? ()
24 LWP 7176 "rpc worker-7176" 0x00007f39b81a2ad3 in ?? ()
25 LWP 7177 "rpc worker-7177" 0x00007f39b81a2ad3 in ?? ()
26 LWP 7178 "rpc worker-7178" 0x00007f39b81a2ad3 in ?? ()
27 LWP 7179 "rpc worker-7179" 0x00007f39b81a2ad3 in ?? ()
28 LWP 7180 "rpc worker-7180" 0x00007f39b81a2ad3 in ?? ()
29 LWP 7181 "rpc worker-7181" 0x00007f39b81a2ad3 in ?? ()
30 LWP 7182 "rpc worker-7182" 0x00007f39b81a2ad3 in ?? ()
31 LWP 7183 "rpc worker-7183" 0x00007f39b81a2ad3 in ?? ()
32 LWP 7184 "rpc worker-7184" 0x00007f39b81a2ad3 in ?? ()
33 LWP 7185 "rpc worker-7185" 0x00007f39b81a2ad3 in ?? ()
34 LWP 7186 "rpc worker-7186" 0x00007f39b81a2ad3 in ?? ()
35 LWP 7187 "rpc worker-7187" 0x00007f39b81a2ad3 in ?? ()
36 LWP 7188 "rpc worker-7188" 0x00007f39b81a2ad3 in ?? ()
37 LWP 7189 "rpc worker-7189" 0x00007f39b81a2ad3 in ?? ()
38 LWP 7190 "rpc worker-7190" 0x00007f39b81a2ad3 in ?? ()
39 LWP 7191 "rpc worker-7191" 0x00007f39b81a2ad3 in ?? ()
40 LWP 7192 "rpc worker-7192" 0x00007f39b81a2ad3 in ?? ()
41 LWP 7193 "rpc worker-7193" 0x00007f39b81a2ad3 in ?? ()
42 LWP 7194 "rpc worker-7194" 0x00007f39b81a2ad3 in ?? ()
43 LWP 7195 "rpc worker-7195" 0x00007f39b81a2ad3 in ?? ()
44 LWP 7196 "rpc worker-7196" 0x00007f39b81a2ad3 in ?? ()
45 LWP 7197 "rpc worker-7197" 0x00007f39b81a2ad3 in ?? ()
46 LWP 7198 "rpc worker-7198" 0x00007f39b81a2ad3 in ?? ()
47 LWP 7199 "rpc worker-7199" 0x00007f39b81a2ad3 in ?? ()
48 LWP 7200 "rpc worker-7200" 0x00007f39b81a2ad3 in ?? ()
49 LWP 7201 "rpc worker-7201" 0x00007f39b81a2ad3 in ?? ()
50 LWP 7202 "rpc worker-7202" 0x00007f39b81a2ad3 in ?? ()
51 LWP 7203 "rpc worker-7203" 0x00007f39b81a2ad3 in ?? ()
52 LWP 7204 "rpc worker-7204" 0x00007f39b81a2ad3 in ?? ()
53 LWP 7205 "rpc worker-7205" 0x00007f39b81a2ad3 in ?? ()
54 LWP 7206 "rpc worker-7206" 0x00007f39b81a2ad3 in ?? ()
55 LWP 7207 "rpc worker-7207" 0x00007f39b81a2ad3 in ?? ()
56 LWP 7208 "rpc worker-7208" 0x00007f39b81a2ad3 in ?? ()
57 LWP 7209 "rpc worker-7209" 0x00007f39b81a2ad3 in ?? ()
58 LWP 7210 "rpc worker-7210" 0x00007f39b81a2ad3 in ?? ()
59 LWP 7211 "rpc worker-7211" 0x00007f39b81a2ad3 in ?? ()
60 LWP 7212 "rpc worker-7212" 0x00007f39b81a2ad3 in ?? ()
61 LWP 7213 "rpc worker-7213" 0x00007f39b81a2ad3 in ?? ()
62 LWP 7214 "rpc worker-7214" 0x00007f39b81a2ad3 in ?? ()
63 LWP 7215 "rpc worker-7215" 0x00007f39b81a2ad3 in ?? ()
64 LWP 7216 "rpc worker-7216" 0x00007f39b81a2ad3 in ?? ()
65 LWP 7217 "rpc worker-7217" 0x00007f39b81a2ad3 in ?? ()
66 LWP 7218 "rpc worker-7218" 0x00007f39b81a2ad3 in ?? ()
67 LWP 7219 "rpc worker-7219" 0x00007f39b81a2ad3 in ?? ()
68 LWP 7220 "rpc worker-7220" 0x00007f39b81a2ad3 in ?? ()
69 LWP 7221 "rpc worker-7221" 0x00007f39b81a2ad3 in ?? ()
70 LWP 7222 "rpc worker-7222" 0x00007f39b81a2ad3 in ?? ()
71 LWP 7223 "rpc worker-7223" 0x00007f39b81a2ad3 in ?? ()
72 LWP 7224 "rpc worker-7224" 0x00007f39b81a2ad3 in ?? ()
73 LWP 7225 "rpc worker-7225" 0x00007f39b81a2ad3 in ?? ()
74 LWP 7226 "rpc worker-7226" 0x00007f39b81a2ad3 in ?? ()
75 LWP 7227 "rpc worker-7227" 0x00007f39b81a2ad3 in ?? ()
76 LWP 7228 "rpc worker-7228" 0x00007f39b81a2ad3 in ?? ()
77 LWP 7229 "rpc worker-7229" 0x00007f39b81a2ad3 in ?? ()
78 LWP 7230 "rpc worker-7230" 0x00007f39b81a2ad3 in ?? ()
79 LWP 7231 "rpc worker-7231" 0x00007f39b81a2ad3 in ?? ()
80 LWP 7232 "rpc worker-7232" 0x00007f39b81a2ad3 in ?? ()
81 LWP 7233 "rpc worker-7233" 0x00007f39b81a2ad3 in ?? ()
82 LWP 7234 "rpc worker-7234" 0x00007f39b81a2ad3 in ?? ()
83 LWP 7235 "rpc worker-7235" 0x00007f39b81a2ad3 in ?? ()
84 LWP 7236 "rpc worker-7236" 0x00007f39b81a2ad3 in ?? ()
85 LWP 7237 "rpc worker-7237" 0x00007f39b81a2ad3 in ?? ()
86 LWP 7238 "rpc worker-7238" 0x00007f39b81a2ad3 in ?? ()
87 LWP 7239 "rpc worker-7239" 0x00007f39b81a2ad3 in ?? ()
88 LWP 7240 "rpc worker-7240" 0x00007f39b81a2ad3 in ?? ()
89 LWP 7241 "rpc worker-7241" 0x00007f39b81a2ad3 in ?? ()
90 LWP 7242 "rpc worker-7242" 0x00007f39b81a2ad3 in ?? ()
91 LWP 7243 "rpc worker-7243" 0x00007f39b81a2ad3 in ?? ()
92 LWP 7244 "rpc worker-7244" 0x00007f39b81a2ad3 in ?? ()
93 LWP 7245 "rpc worker-7245" 0x00007f39b81a2ad3 in ?? ()
94 LWP 7246 "rpc worker-7246" 0x00007f39b81a2ad3 in ?? ()
95 LWP 7247 "rpc worker-7247" 0x00007f39b81a2ad3 in ?? ()
96 LWP 7248 "rpc worker-7248" 0x00007f39b81a2ad3 in ?? ()
97 LWP 7249 "rpc worker-7249" 0x00007f39b81a2ad3 in ?? ()
98 LWP 7250 "rpc worker-7250" 0x00007f39b81a2ad3 in ?? ()
99 LWP 7251 "rpc worker-7251" 0x00007f39b81a2ad3 in ?? ()
100 LWP 7252 "rpc worker-7252" 0x00007f39b81a2ad3 in ?? ()
101 LWP 7253 "rpc worker-7253" 0x00007f39b81a2ad3 in ?? ()
102 LWP 7254 "rpc worker-7254" 0x00007f39b81a2ad3 in ?? ()
103 LWP 7255 "rpc worker-7255" 0x00007f39b81a2ad3 in ?? ()
104 LWP 7256 "rpc worker-7256" 0x00007f39b81a2ad3 in ?? ()
105 LWP 7257 "rpc worker-7257" 0x00007f39b81a2ad3 in ?? ()
106 LWP 7258 "rpc worker-7258" 0x00007f39b81a2ad3 in ?? ()
107 LWP 7259 "rpc worker-7259" 0x00007f39b81a2ad3 in ?? ()
108 LWP 7260 "rpc worker-7260" 0x00007f39b81a2ad3 in ?? ()
109 LWP 7261 "rpc worker-7261" 0x00007f39b81a2ad3 in ?? ()
110 LWP 7262 "rpc worker-7262" 0x00007f39b81a2ad3 in ?? ()
111 LWP 7263 "rpc worker-7263" 0x00007f39b81a2ad3 in ?? ()
112 LWP 7264 "rpc worker-7264" 0x00007f39b81a2ad3 in ?? ()
113 LWP 7265 "rpc worker-7265" 0x00007f39b81a2ad3 in ?? ()
114 LWP 7266 "rpc worker-7266" 0x00007f39b81a2ad3 in ?? ()
115 LWP 7267 "rpc worker-7267" 0x00007f39b81a2ad3 in ?? ()
116 LWP 7268 "rpc worker-7268" 0x00007f39b81a2ad3 in ?? ()
117 LWP 7269 "rpc worker-7269" 0x00007f39b81a2ad3 in ?? ()
118 LWP 7270 "diag-logger-727" 0x00007f39b81a2fb9 in ?? ()
119 LWP 7271 "result-tracker-" 0x00007f39b81a2fb9 in ?? ()
120 LWP 7272 "excess-log-dele" 0x00007f39b81a2fb9 in ?? ()
121 LWP 7273 "acceptor-7273" 0x00007f39b3520fc7 in ?? ()
122 LWP 7274 "heartbeat-7274" 0x00007f39b81a2fb9 in ?? ()
123 LWP 7275 "maintenance_sch" 0x00007f39b81a2fb9 in ?? ()
Thread 123 (LWP 7275):
#0 0x00007f39b81a2fb9 in ?? ()
#1 0x00007b0100000000 in ?? ()
#2 0x00000000000001ef in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007f396c3b96d0 in ?? ()
#6 0x00000000000003de in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 7274):
#0 0x00007f39b81a2fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 7273):
#0 0x00007f39b3520fc7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 7272):
#0 0x00007f39b81a2fb9 in ?? ()
#1 0x00007f396dbbc950 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffffc4632c0 in ?? ()
#5 0x00007f396dbbc7c0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 7271):
#0 0x00007f39b81a2fb9 in ?? ()
#1 0x00007b34000016c0 in ?? ()
#2 0x000000000000007d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001758 in ?? ()
#5 0x00007f396e3bd810 in ?? ()
#6 0x00000000000000fa in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 7270):
#0 0x00007f39b81a2fb9 in ?? ()
#1 0x00007f39b1488008 in ?? ()
#2 0x000000000000006e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000990 in ?? ()
#5 0x00007f396ebbe760 in ?? ()
#6 0x00000000000000dc in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 7269):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 7268):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 7267):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 7266):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 7265):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 7264):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 7263):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 7262):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 7261):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 7260):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 7259):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 7258):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 7257):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 7256):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 7255):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 7254):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 7253):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 7252):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 7251):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 7250):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 7249):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 7248):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 7247):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 7246):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 7245):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 7244):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 7243):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 7242):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 7241):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 7240):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 7239):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 7238):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 7237):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 7236):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 7235):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 7234):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 7233):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 7232):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 7231):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 7230):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 7229):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000798 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b24001147c8 in ?? ()
#4 0x00007f3983dba730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f3983dba750 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 76 (LWP 7228):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x00000000000005c2 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240010ffc8 in ?? ()
#4 0x00007f39845bb730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f39845bb750 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 7227):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240010d7dc in ?? ()
#4 0x00007f3984dbc730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f3984dbc750 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x0000000000000000 in ?? ()
Thread 74 (LWP 7226):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 7225):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 7224):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 7223):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 7222):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 7221):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 7220):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 7219):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 7218):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 7217):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 7216):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 7215):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 7214):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 7213):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 7212):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 7211):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 7210):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 7209):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000b902c in ?? ()
#4 0x00007f398e1bc730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f398e1bc750 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x007f0400000026b3 in ?? ()
#9 0x00007f39b81a2770 in ?? ()
#10 0x00007f398e1bc750 in ?? ()
#11 0x0002008300000dfe in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 7208):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 7207):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 7206):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 7205):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 7204):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 7203):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 7202):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 7201):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 7200):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 7199):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 7198):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 7197):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 7196):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 7195):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 7194):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 7193):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 7192):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 7191):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 7190):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 7189):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000004 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240005ffe8 in ?? ()
#4 0x00007f39985be730 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f39985be750 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 36 (LWP 7188):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 7187):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 7186):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 7185):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 7184):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 7183):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 7182):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 7181):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 7180):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 7179):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 7178):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 7177):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 7176):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 7175):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 7174):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 7173):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 7172):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 7171):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 7170):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 7169):
#0 0x00007f39b81a2fb9 in ?? ()
#1 0x00007f39a2b94ab8 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800004000 in ?? ()
#5 0x00007f39a2b94710 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 7168):
#0 0x00007f39b81a2fb9 in ?? ()
#1 0x00007f39a33959b8 in ?? ()
#2 0x0000000000000019 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000345d8 in ?? ()
#5 0x00007f39a3395850 in ?? ()
#6 0x0000000000000032 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 7167):
#0 0x00007f39b81a2fb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007f39a3b96420 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 7166):
#0 0x00007f39b81a2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 7165):
#0 0x00007f39b351f947 in ?? ()
#1 0x00007f39a4b98560 in ?? ()
#2 0x0048e000000e09b2 in ?? ()
#3 0x00007f39a4b98510 in ?? ()
#4 0x00007f39a4b99b80 in ?? ()
#5 0x00007f39a4b98510 in ?? ()
#6 0x0000000000000014 in ?? ()
#7 0x00007b5800001b00 in ?? ()
#8 0x000000000049bd45 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f39b0c70000 in ?? ()
#10 0x000000000049bc49 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f39a4b99b80 in ?? ()
#12 0x00007f39b5f78069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f39bb79e4c0 in ?? ()
#15 0x00007b4c000034d0 in ?? ()
#16 0x00007b4c000034d8 in ?? ()
#17 0x00007f39a4b987b0 in ?? ()
#18 0x00007b4400035890 in ?? ()
#19 0x00007f39a4b98ce0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 12 (LWP 7164):
#0 0x00007f39b351f947 in ?? ()
#1 0x00007f39a5399560 in ?? ()
#2 0x0044e0000008ffb9 in ?? ()
#3 0x00007f39a5399510 in ?? ()
#4 0x00007f39a539ab80 in ?? ()
#5 0x00007f39a5399510 in ?? ()
#6 0x0000000000000013 in ?? ()
#7 0x00007b5800001800 in ?? ()
#8 0x000000000049bd45 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f39b0c82000 in ?? ()
#10 0x000000000049bc49 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f39a539ab80 in ?? ()
#12 0x00007f39b5f78069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f39bb79e4c0 in ?? ()
#15 0x00007b4c00002f90 in ?? ()
#16 0x00007b4c00002f98 in ?? ()
#17 0x00007f39a53997b0 in ?? ()
#18 0x00007b44000368d0 in ?? ()
#19 0x00007f39a5399ce0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 11 (LWP 7163):
#0 0x00007f39b351f947 in ?? ()
#1 0x00007f39a5b9a560 in ?? ()
#2 0x0040e000000a53b7 in ?? ()
#3 0x00007f39a5b9a510 in ?? ()
#4 0x00007f39a5b9bb80 in ?? ()
#5 0x00007f39a5b9a510 in ?? ()
#6 0x0000000000000012 in ?? ()
#7 0x00007b5800001200 in ?? ()
#8 0x000000000049bd45 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f39b0c94000 in ?? ()
#10 0x000000000049bc49 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f39a5b9bb80 in ?? ()
#12 0x00007f39b5f78069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f39bb79e4c0 in ?? ()
#15 0x00007b4c000026d0 in ?? ()
#16 0x00007b4c000026d8 in ?? ()
#17 0x00007f39a5b9a7b0 in ?? ()
#18 0x00007b4400035c50 in ?? ()
#19 0x00007f39a5b9ace0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 10 (LWP 7162):
#0 0x00007f39b351f947 in ?? ()
#1 0x00007f39a8bbe560 in ?? ()
#2 0x003ce000016f8eff in ?? ()
#3 0x00007f39a8bbe510 in ?? ()
#4 0x00007f39a8bbfb80 in ?? ()
#5 0x00007f39a8bbe510 in ?? ()
#6 0x000000000000000f in ?? ()
#7 0x00007b5800000f00 in ?? ()
#8 0x000000000049bd45 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f39b0ca6000 in ?? ()
#10 0x000000000049bc49 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f39a8bbfb80 in ?? ()
#12 0x00007f39b5f78069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f39bb79e4c0 in ?? ()
#15 0x00007b4c00002c10 in ?? ()
#16 0x00007b4c00002c18 in ?? ()
#17 0x00007f39a8bbe7b0 in ?? ()
#18 0x00007b4400037050 in ?? ()
#19 0x00007f39a8bbece0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 9 (LWP 7159):
#0 0x00007f39b3512bb9 in ?? ()
#1 0x00007f39ac3bcbf0 in ?? ()
#2 0x00007b0400009810 in ?? ()
#3 0x00007f39ac3bdb80 in ?? ()
#4 0x00007f39ac3bcbf0 in ?? ()
#5 0x00007b0400009810 in ?? ()
#6 0x000000000049bf53 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007f39b0e3a000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007f39ac3bdb80 in ?? ()
#10 0x00007f39bcf92f2d in ?? ()
#11 0x00007f39bcf90000 in ?? ()
#12 0x00000000004a975d in __sanitizer::internal_alloc_placeholder ()
#13 0x0000000000000000 in ?? ()
Thread 8 (LWP 7158):
#0 0x00007f39b81a2fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 7 (LWP 7157):
#0 0x00007f39b81a69e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 7150):
#0 0x00007f39b81a2fb9 in ?? ()
#1 0x00007f39ad3bea50 in ?? ()
#2 0x0000000000000267 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400037718 in ?? ()
#5 0x00007f39ad3be5e0 in ?? ()
#6 0x00000000000004ce in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 7149):
#0 0x00007f39b81a2fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 7148):
#0 0x00007f39b81a2fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 7147):
#0 0x00007f39b81a2fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 7146):
#0 0x00007f39b34e2680 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 7145):
#0 0x00007f39b81a6d50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x000000000047b1db in __sanitizer::internal_alloc_placeholder ()
#3 0x00007f39b2740cc0 in ?? ()
#4 0x00007f39b2740cc0 in ?? ()
#5 0x00007ffffc4630d0 in ?? ()
#6 0x000000000049e5a4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000b0ad3c in ?? ()
#9 0x00007f39b2740cc0 in ?? ()
#10 0x00007f39b6646f0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20260505 13:59:58.597390 6116 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu with pid 6878
I20260505 13:59:58.657459 6116 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu with pid 7012
I20260505 13:59:58.731506 6116 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu with pid 7145
I20260505 13:59:58.796723 6116 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskiEg_5k/build/tsan/bin/kudu with pid 6786
2026-05-05T13:59:58Z chronyd exiting
I20260505 13:59:58.876755 6116 test_util.cc:182] -----------------------------------------------
I20260505 13:59:58.876987 6116 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskiEg_5k/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1777989435273980-6116-0
[ FAILED ] TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate (135963 ms)
[----------] 4 tests from TabletCopyITest (163399 ms total)
[----------] 1 test from FaultFlags/BadTabletCopyITest
[ RUN ] FaultFlags/BadTabletCopyITest.TestBadCopy/1
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:1510: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] FaultFlags/BadTabletCopyITest.TestBadCopy/1 (8 ms)
[----------] 1 test from FaultFlags/BadTabletCopyITest (8 ms total)
[----------] Global test environment tear-down
[==========] 5 tests from 2 test suites ran. (163409 ms total)
[ PASSED ] 1 test.
[ SKIPPED ] 3 tests, listed below:
[ SKIPPED ] TabletCopyITest.TestRejectRogueLeader
[ SKIPPED ] TabletCopyITest.TestDeleteLeaderDuringTabletCopyStressTest
[ SKIPPED ] FaultFlags/BadTabletCopyITest.TestBadCopy/1
[ FAILED ] 1 test, listed below:
[ FAILED ] TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate
1 FAILED TEST