Note: This is test shard 6 of 6.
[==========] Running 4 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 3 tests from TabletCopyITest
[ RUN ] TabletCopyITest.TestConcurrentTabletCopys
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:620: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] TabletCopyITest.TestConcurrentTabletCopys (11 ms)
[ RUN ] TabletCopyITest.TestTabletCopyClearsLastLoggedOpId
2025-06-24T19:57:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T19:57:54Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250624 19:57:54.440048 27399 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
/tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.193.254:40859
--webserver_interface=127.26.193.254
--webserver_port=0
--builtin_ntp_servers=127.26.193.212:36561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.193.254:40859
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250624 19:57:54.745715 27408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 19:57:54.746307 27408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 19:57:54.746732 27408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 19:57:54.779160 27408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 19:57:54.779460 27408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 19:57:54.779690 27408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 19:57:54.779896 27408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 19:57:54.816550 27408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.193.212:36561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.26.193.254:40859
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.193.254:40859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.193.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 19:43:22 UTC on 5fd53c4cbb9d
build id 6753
TSAN enabled
I20250624 19:57:54.817973 27408 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 19:57:54.819824 27408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 19:57:54.836360 27415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 19:57:54.837424 27408 server_base.cc:1048] running on GCE node
W20250624 19:57:54.836403 27417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:57:54.836419 27414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 19:57:56.026022 27408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 19:57:56.029450 27408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 19:57:56.031003 27408 hybrid_clock.cc:648] HybridClock initialized: now 1750795076030950 us; error 72 us; skew 500 ppm
I20250624 19:57:56.031870 27408 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 19:57:56.046029 27408 webserver.cc:469] Webserver started at http://127.26.193.254:38991/ using document root <none> and password file <none>
I20250624 19:57:56.047135 27408 fs_manager.cc:362] Metadata directory not provided
I20250624 19:57:56.047422 27408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 19:57:56.047959 27408 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 19:57:56.052783 27408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/data/instance:
uuid: "a330f784374f4d769314556698d7e3aa"
format_stamp: "Formatted at 2025-06-24 19:57:56 on dist-test-slave-gtkq"
I20250624 19:57:56.054119 27408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/wal/instance:
uuid: "a330f784374f4d769314556698d7e3aa"
format_stamp: "Formatted at 2025-06-24 19:57:56 on dist-test-slave-gtkq"
I20250624 19:57:56.062135 27408 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250624 19:57:56.069100 27424 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 19:57:56.070370 27408 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20250624 19:57:56.070763 27408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/data,/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/wal
uuid: "a330f784374f4d769314556698d7e3aa"
format_stamp: "Formatted at 2025-06-24 19:57:56 on dist-test-slave-gtkq"
I20250624 19:57:56.071111 27408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 19:57:56.132079 27408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 19:57:56.133693 27408 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 19:57:56.134168 27408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 19:57:56.211637 27408 rpc_server.cc:307] RPC server started. Bound to: 127.26.193.254:40859
I20250624 19:57:56.211715 27475 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.193.254:40859 every 8 connection(s)
I20250624 19:57:56.214633 27408 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/data/info.pb
I20250624 19:57:56.220453 27476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 19:57:56.221535 27399 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu as pid 27408
I20250624 19:57:56.221989 27399 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/master-0/wal/instance
I20250624 19:57:56.242759 27476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa: Bootstrap starting.
I20250624 19:57:56.248934 27476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa: Neither blocks nor log segments found. Creating new log.
I20250624 19:57:56.250959 27476 log.cc:826] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa: Log is configured to *not* fsync() on all Append() calls
I20250624 19:57:56.256368 27476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa: No bootstrap required, opened a new log
I20250624 19:57:56.275815 27476 raft_consensus.cc:357] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a330f784374f4d769314556698d7e3aa" member_type: VOTER last_known_addr { host: "127.26.193.254" port: 40859 } }
I20250624 19:57:56.276587 27476 raft_consensus.cc:383] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 19:57:56.276916 27476 raft_consensus.cc:738] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a330f784374f4d769314556698d7e3aa, State: Initialized, Role: FOLLOWER
I20250624 19:57:56.277788 27476 consensus_queue.cc:260] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a330f784374f4d769314556698d7e3aa" member_type: VOTER last_known_addr { host: "127.26.193.254" port: 40859 } }
I20250624 19:57:56.278414 27476 raft_consensus.cc:397] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 19:57:56.278724 27476 raft_consensus.cc:491] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 19:57:56.279067 27476 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [term 0 FOLLOWER]: Advancing to term 1
I20250624 19:57:56.283476 27476 raft_consensus.cc:513] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a330f784374f4d769314556698d7e3aa" member_type: VOTER last_known_addr { host: "127.26.193.254" port: 40859 } }
I20250624 19:57:56.284314 27476 leader_election.cc:304] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a330f784374f4d769314556698d7e3aa; no voters:
I20250624 19:57:56.286770 27476 leader_election.cc:290] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 19:57:56.287565 27481 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [term 1 FOLLOWER]: Leader election won for term 1
I20250624 19:57:56.290092 27481 raft_consensus.cc:695] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [term 1 LEADER]: Becoming Leader. State: Replica: a330f784374f4d769314556698d7e3aa, State: Running, Role: LEADER
I20250624 19:57:56.291086 27481 consensus_queue.cc:237] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a330f784374f4d769314556698d7e3aa" member_type: VOTER last_known_addr { host: "127.26.193.254" port: 40859 } }
I20250624 19:57:56.292737 27476 sys_catalog.cc:564] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [sys.catalog]: configured and running, proceeding with master startup.
I20250624 19:57:56.304728 27482 sys_catalog.cc:455] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a330f784374f4d769314556698d7e3aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a330f784374f4d769314556698d7e3aa" member_type: VOTER last_known_addr { host: "127.26.193.254" port: 40859 } } }
I20250624 19:57:56.305645 27482 sys_catalog.cc:458] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [sys.catalog]: This master's current role is: LEADER
I20250624 19:57:56.305534 27483 sys_catalog.cc:455] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader a330f784374f4d769314556698d7e3aa. Latest consensus state: current_term: 1 leader_uuid: "a330f784374f4d769314556698d7e3aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a330f784374f4d769314556698d7e3aa" member_type: VOTER last_known_addr { host: "127.26.193.254" port: 40859 } } }
I20250624 19:57:56.306363 27483 sys_catalog.cc:458] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa [sys.catalog]: This master's current role is: LEADER
I20250624 19:57:56.308642 27491 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 19:57:56.324267 27491 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 19:57:56.341938 27491 catalog_manager.cc:1349] Generated new cluster ID: 37c92c38237445b6a3f597ac5e13c1b7
I20250624 19:57:56.342304 27491 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 19:57:56.372823 27491 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 19:57:56.375049 27491 catalog_manager.cc:1506] Loading token signing keys...
I20250624 19:57:56.399451 27491 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a330f784374f4d769314556698d7e3aa: Generated new TSK 0
I20250624 19:57:56.400805 27491 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 19:57:56.421566 27399 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
/tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.193.193:0
--local_ip_for_outbound_sockets=127.26.193.193
--webserver_interface=127.26.193.193
--webserver_port=0
--tserver_master_addrs=127.26.193.254:40859
--builtin_ntp_servers=127.26.193.212:36561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250624 19:57:56.740443 27500 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250624 19:57:56.741207 27500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 19:57:56.741511 27500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 19:57:56.742051 27500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 19:57:56.775820 27500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 19:57:56.776929 27500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.193.193
I20250624 19:57:56.814882 27500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.193.212:36561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.193.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.193.193
--webserver_port=0
--tserver_master_addrs=127.26.193.254:40859
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.193.193
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 19:43:22 UTC on 5fd53c4cbb9d
build id 6753
TSAN enabled
I20250624 19:57:56.816386 27500 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 19:57:56.818408 27500 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 19:57:56.837826 27507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:57:58.237919 27505 debug-util.cc:398] Leaking SignalData structure 0x7b0800015620 after lost signal to thread 27500
W20250624 19:57:58.338490 27505 kernel_stack_watchdog.cc:198] Thread 27500 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250624 19:57:56.838888 27506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:57:58.340868 27500 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.505s user 0.527s sys 0.925s
W20250624 19:57:58.341372 27500 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.505s user 0.527s sys 0.926s
W20250624 19:57:58.341492 27509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:57:58.346446 27508 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1509 milliseconds
I20250624 19:57:58.346462 27500 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 19:57:58.347846 27500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 19:57:58.350207 27500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 19:57:58.351667 27500 hybrid_clock.cc:648] HybridClock initialized: now 1750795078351596 us; error 89 us; skew 500 ppm
I20250624 19:57:58.352531 27500 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 19:57:58.359758 27500 webserver.cc:469] Webserver started at http://127.26.193.193:33685/ using document root <none> and password file <none>
I20250624 19:57:58.360931 27500 fs_manager.cc:362] Metadata directory not provided
I20250624 19:57:58.361183 27500 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 19:57:58.361693 27500 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 19:57:58.366470 27500 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/data/instance:
uuid: "0cdfbac95f3a41ed948315dd30b5e8a8"
format_stamp: "Formatted at 2025-06-24 19:57:58 on dist-test-slave-gtkq"
I20250624 19:57:58.367776 27500 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/wal/instance:
uuid: "0cdfbac95f3a41ed948315dd30b5e8a8"
format_stamp: "Formatted at 2025-06-24 19:57:58 on dist-test-slave-gtkq"
I20250624 19:57:58.376888 27500 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20250624 19:57:58.388074 27516 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 19:57:58.389927 27500 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.006s sys 0.000s
I20250624 19:57:58.390403 27500 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/wal
uuid: "0cdfbac95f3a41ed948315dd30b5e8a8"
format_stamp: "Formatted at 2025-06-24 19:57:58 on dist-test-slave-gtkq"
I20250624 19:57:58.390921 27500 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 19:57:58.457140 27500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 19:57:58.459271 27500 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 19:57:58.459913 27500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 19:57:58.463433 27500 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 19:57:58.469507 27500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 19:57:58.469853 27500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 19:57:58.470165 27500 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 19:57:58.470384 27500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 19:57:58.650588 27500 rpc_server.cc:307] RPC server started. Bound to: 127.26.193.193:38887
I20250624 19:57:58.650722 27628 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.193.193:38887 every 8 connection(s)
I20250624 19:57:58.653571 27500 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/data/info.pb
I20250624 19:57:58.659803 27399 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu as pid 27500
I20250624 19:57:58.660288 27399 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/wal/instance
I20250624 19:57:58.669647 27399 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
/tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.193.194:0
--local_ip_for_outbound_sockets=127.26.193.194
--webserver_interface=127.26.193.194
--webserver_port=0
--tserver_master_addrs=127.26.193.254:40859
--builtin_ntp_servers=127.26.193.212:36561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250624 19:57:58.685374 27629 heartbeater.cc:344] Connected to a master server at 127.26.193.254:40859
I20250624 19:57:58.686064 27629 heartbeater.cc:461] Registering TS with master...
I20250624 19:57:58.687958 27629 heartbeater.cc:507] Master 127.26.193.254:40859 requested a full tablet report, sending...
I20250624 19:57:58.692028 27441 ts_manager.cc:194] Registered new tserver with Master: 0cdfbac95f3a41ed948315dd30b5e8a8 (127.26.193.193:38887)
I20250624 19:57:58.695127 27441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.193.193:42671
W20250624 19:57:58.991634 27633 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250624 19:57:58.992311 27633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 19:57:58.992643 27633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 19:57:58.993160 27633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 19:57:59.026638 27633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 19:57:59.027585 27633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.193.194
I20250624 19:57:59.064863 27633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.193.212:36561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.193.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.193.194
--webserver_port=0
--tserver_master_addrs=127.26.193.254:40859
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.193.194
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 19:43:22 UTC on 5fd53c4cbb9d
build id 6753
TSAN enabled
I20250624 19:57:59.066323 27633 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 19:57:59.068163 27633 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 19:57:59.087075 27639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 19:57:59.700028 27629 heartbeater.cc:499] Master 127.26.193.254:40859 was elected leader, sending a full tablet report...
W20250624 19:57:59.087109 27640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:57:59.087175 27642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 19:57:59.089114 27633 server_base.cc:1048] running on GCE node
I20250624 19:58:00.274748 27633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 19:58:00.277158 27633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 19:58:00.278596 27633 hybrid_clock.cc:648] HybridClock initialized: now 1750795080278535 us; error 68 us; skew 500 ppm
I20250624 19:58:00.279443 27633 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 19:58:00.286218 27633 webserver.cc:469] Webserver started at http://127.26.193.194:38415/ using document root <none> and password file <none>
I20250624 19:58:00.287214 27633 fs_manager.cc:362] Metadata directory not provided
I20250624 19:58:00.287423 27633 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 19:58:00.287878 27633 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 19:58:00.292778 27633 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/data/instance:
uuid: "cc019455aed34bcfa1d89817aee20ea8"
format_stamp: "Formatted at 2025-06-24 19:58:00 on dist-test-slave-gtkq"
I20250624 19:58:00.294119 27633 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/wal/instance:
uuid: "cc019455aed34bcfa1d89817aee20ea8"
format_stamp: "Formatted at 2025-06-24 19:58:00 on dist-test-slave-gtkq"
I20250624 19:58:00.302350 27633 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20250624 19:58:00.308754 27649 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:00.310106 27633 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250624 19:58:00.310492 27633 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/wal
uuid: "cc019455aed34bcfa1d89817aee20ea8"
format_stamp: "Formatted at 2025-06-24 19:58:00 on dist-test-slave-gtkq"
I20250624 19:58:00.310858 27633 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 19:58:00.368202 27633 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 19:58:00.369825 27633 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 19:58:00.370323 27633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 19:58:00.373020 27633 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 19:58:00.377427 27633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 19:58:00.377661 27633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:00.377969 27633 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 19:58:00.378125 27633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:00.525552 27633 rpc_server.cc:307] RPC server started. Bound to: 127.26.193.194:37557
I20250624 19:58:00.525703 27761 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.193.194:37557 every 8 connection(s)
I20250624 19:58:00.528277 27633 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/data/info.pb
I20250624 19:58:00.535171 27399 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu as pid 27633
I20250624 19:58:00.535773 27399 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-1/wal/instance
I20250624 19:58:00.542392 27399 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
/tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.193.195:0
--local_ip_for_outbound_sockets=127.26.193.195
--webserver_interface=127.26.193.195
--webserver_port=0
--tserver_master_addrs=127.26.193.254:40859
--builtin_ntp_servers=127.26.193.212:36561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250624 19:58:00.552139 27762 heartbeater.cc:344] Connected to a master server at 127.26.193.254:40859
I20250624 19:58:00.552826 27762 heartbeater.cc:461] Registering TS with master...
I20250624 19:58:00.554535 27762 heartbeater.cc:507] Master 127.26.193.254:40859 requested a full tablet report, sending...
I20250624 19:58:00.557907 27441 ts_manager.cc:194] Registered new tserver with Master: cc019455aed34bcfa1d89817aee20ea8 (127.26.193.194:37557)
I20250624 19:58:00.560081 27441 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.193.194:36181
W20250624 19:58:00.859171 27766 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250624 19:58:00.859874 27766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 19:58:00.860162 27766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 19:58:00.860723 27766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 19:58:00.893589 27766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 19:58:00.894536 27766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.193.195
I20250624 19:58:00.931627 27766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.193.212:36561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.193.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.193.195
--webserver_port=0
--tserver_master_addrs=127.26.193.254:40859
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.193.195
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 19:43:22 UTC on 5fd53c4cbb9d
build id 6753
TSAN enabled
I20250624 19:58:00.933099 27766 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 19:58:00.935262 27766 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 19:58:00.953176 27772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 19:58:01.564989 27762 heartbeater.cc:499] Master 127.26.193.254:40859 was elected leader, sending a full tablet report...
W20250624 19:58:00.955807 27775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:58:00.953184 27773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:58:02.168586 27774 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1210 milliseconds
I20250624 19:58:02.168754 27766 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 19:58:02.170063 27766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 19:58:02.173003 27766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 19:58:02.174521 27766 hybrid_clock.cc:648] HybridClock initialized: now 1750795082174465 us; error 78 us; skew 500 ppm
I20250624 19:58:02.175397 27766 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 19:58:02.186059 27766 webserver.cc:469] Webserver started at http://127.26.193.195:39183/ using document root <none> and password file <none>
I20250624 19:58:02.187094 27766 fs_manager.cc:362] Metadata directory not provided
I20250624 19:58:02.187319 27766 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 19:58:02.187820 27766 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 19:58:02.192576 27766 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/data/instance:
uuid: "09d5340a609f4fc7b1e9967be5561ddf"
format_stamp: "Formatted at 2025-06-24 19:58:02 on dist-test-slave-gtkq"
I20250624 19:58:02.194060 27766 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/wal/instance:
uuid: "09d5340a609f4fc7b1e9967be5561ddf"
format_stamp: "Formatted at 2025-06-24 19:58:02 on dist-test-slave-gtkq"
I20250624 19:58:02.204424 27766 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.006s sys 0.004s
I20250624 19:58:02.211488 27782 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:02.212841 27766 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20250624 19:58:02.213222 27766 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/wal
uuid: "09d5340a609f4fc7b1e9967be5561ddf"
format_stamp: "Formatted at 2025-06-24 19:58:02 on dist-test-slave-gtkq"
I20250624 19:58:02.213564 27766 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 19:58:02.275219 27766 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 19:58:02.276880 27766 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 19:58:02.277364 27766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 19:58:02.280110 27766 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 19:58:02.284549 27766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 19:58:02.284834 27766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250624 19:58:02.285151 27766 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 19:58:02.285305 27766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:02.429919 27766 rpc_server.cc:307] RPC server started. Bound to: 127.26.193.195:35113
I20250624 19:58:02.430042 27894 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.193.195:35113 every 8 connection(s)
I20250624 19:58:02.432657 27766 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/data/info.pb
I20250624 19:58:02.437981 27399 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu as pid 27766
I20250624 19:58:02.438550 27399 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-2/wal/instance
I20250624 19:58:02.455564 27895 heartbeater.cc:344] Connected to a master server at 127.26.193.254:40859
I20250624 19:58:02.456085 27895 heartbeater.cc:461] Registering TS with master...
I20250624 19:58:02.457413 27895 heartbeater.cc:507] Master 127.26.193.254:40859 requested a full tablet report, sending...
I20250624 19:58:02.460084 27440 ts_manager.cc:194] Registered new tserver with Master: 09d5340a609f4fc7b1e9967be5561ddf (127.26.193.195:35113)
I20250624 19:58:02.461594 27440 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.193.195:44783
I20250624 19:58:02.475044 27399 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 19:58:02.510911 27399 test_util.cc:276] Using random seed: -1091063261
I20250624 19:58:02.561884 27440 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59204:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250624 19:58:02.564999 27440 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 19:58:02.630259 27564 tablet_service.cc:1468] Processing CreateTablet for tablet e78e9c3d89d444dbba658c02276bb4ca (DEFAULT_TABLE table=test-workload [id=d034f483f39146a990a680e7fc90f82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 19:58:02.632588 27564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e78e9c3d89d444dbba658c02276bb4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 19:58:02.641116 27697 tablet_service.cc:1468] Processing CreateTablet for tablet e78e9c3d89d444dbba658c02276bb4ca (DEFAULT_TABLE table=test-workload [id=d034f483f39146a990a680e7fc90f82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 19:58:02.643504 27697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e78e9c3d89d444dbba658c02276bb4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 19:58:02.646167 27830 tablet_service.cc:1468] Processing CreateTablet for tablet e78e9c3d89d444dbba658c02276bb4ca (DEFAULT_TABLE table=test-workload [id=d034f483f39146a990a680e7fc90f82d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 19:58:02.648784 27830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e78e9c3d89d444dbba658c02276bb4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 19:58:02.678706 27919 tablet_bootstrap.cc:492] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: Bootstrap starting.
I20250624 19:58:02.688694 27920 tablet_bootstrap.cc:492] T e78e9c3d89d444dbba658c02276bb4ca P cc019455aed34bcfa1d89817aee20ea8: Bootstrap starting.
I20250624 19:58:02.690201 27921 tablet_bootstrap.cc:492] T e78e9c3d89d444dbba658c02276bb4ca P 09d5340a609f4fc7b1e9967be5561ddf: Bootstrap starting.
I20250624 19:58:02.694384 27919 tablet_bootstrap.cc:654] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: Neither blocks nor log segments found. Creating new log.
I20250624 19:58:02.696720 27919 log.cc:826] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: Log is configured to *not* fsync() on all Append() calls
I20250624 19:58:02.697242 27920 tablet_bootstrap.cc:654] T e78e9c3d89d444dbba658c02276bb4ca P cc019455aed34bcfa1d89817aee20ea8: Neither blocks nor log segments found. Creating new log.
I20250624 19:58:02.698720 27921 tablet_bootstrap.cc:654] T e78e9c3d89d444dbba658c02276bb4ca P 09d5340a609f4fc7b1e9967be5561ddf: Neither blocks nor log segments found. Creating new log.
I20250624 19:58:02.699911 27920 log.cc:826] T e78e9c3d89d444dbba658c02276bb4ca P cc019455aed34bcfa1d89817aee20ea8: Log is configured to *not* fsync() on all Append() calls
I20250624 19:58:02.701428 27921 log.cc:826] T e78e9c3d89d444dbba658c02276bb4ca P 09d5340a609f4fc7b1e9967be5561ddf: Log is configured to *not* fsync() on all Append() calls
I20250624 19:58:02.705389 27919 tablet_bootstrap.cc:492] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: No bootstrap required, opened a new log
I20250624 19:58:02.706241 27919 ts_tablet_manager.cc:1397] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.003s
I20250624 19:58:02.708899 27921 tablet_bootstrap.cc:492] T e78e9c3d89d444dbba658c02276bb4ca P 09d5340a609f4fc7b1e9967be5561ddf: No bootstrap required, opened a new log
I20250624 19:58:02.708899 27920 tablet_bootstrap.cc:492] T e78e9c3d89d444dbba658c02276bb4ca P cc019455aed34bcfa1d89817aee20ea8: No bootstrap required, opened a new log
I20250624 19:58:02.709512 27920 ts_tablet_manager.cc:1397] T e78e9c3d89d444dbba658c02276bb4ca P cc019455aed34bcfa1d89817aee20ea8: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.007s
I20250624 19:58:02.709561 27921 ts_tablet_manager.cc:1397] T e78e9c3d89d444dbba658c02276bb4ca P 09d5340a609f4fc7b1e9967be5561ddf: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.008s
I20250624 19:58:02.729796 27919 raft_consensus.cc:357] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdfbac95f3a41ed948315dd30b5e8a8" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38887 } } peers { permanent_uuid: "09d5340a609f4fc7b1e9967be5561ddf" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 35113 } } peers { permanent_uuid: "cc019455aed34bcfa1d89817aee20ea8" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 37557 } }
I20250624 19:58:02.730677 27919 raft_consensus.cc:738] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cdfbac95f3a41ed948315dd30b5e8a8, State: Initialized, Role: FOLLOWER
I20250624 19:58:02.731397 27919 consensus_queue.cc:260] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdfbac95f3a41ed948315dd30b5e8a8" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38887 } } peers { permanent_uuid: "09d5340a609f4fc7b1e9967be5561ddf" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 35113 } } peers { permanent_uuid: "cc019455aed34bcfa1d89817aee20ea8" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 37557 } }
I20250624 19:58:02.736101 27919 ts_tablet_manager.cc:1428] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: Time spent starting tablet: real 0.030s user 0.028s sys 0.001s
I20250624 19:58:02.738775 27921 raft_consensus.cc:357] T e78e9c3d89d444dbba658c02276bb4ca P 09d5340a609f4fc7b1e9967be5561ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdfbac95f3a41ed948315dd30b5e8a8" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38887 } } peers { permanent_uuid: "09d5340a609f4fc7b1e9967be5561ddf" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 35113 } } peers { permanent_uuid: "cc019455aed34bcfa1d89817aee20ea8" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 37557 } }
I20250624 19:58:02.739781 27921 raft_consensus.cc:738] T e78e9c3d89d444dbba658c02276bb4ca P 09d5340a609f4fc7b1e9967be5561ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09d5340a609f4fc7b1e9967be5561ddf, State: Initialized, Role: FOLLOWER
I20250624 19:58:02.740419 27921 consensus_queue.cc:260] T e78e9c3d89d444dbba658c02276bb4ca P 09d5340a609f4fc7b1e9967be5561ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdfbac95f3a41ed948315dd30b5e8a8" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38887 } } peers { permanent_uuid: "09d5340a609f4fc7b1e9967be5561ddf" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 35113 } } peers { permanent_uuid: "cc019455aed34bcfa1d89817aee20ea8" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 37557 } }
I20250624 19:58:02.741104 27920 raft_consensus.cc:357] T e78e9c3d89d444dbba658c02276bb4ca P cc019455aed34bcfa1d89817aee20ea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdfbac95f3a41ed948315dd30b5e8a8" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38887 } } peers { permanent_uuid: "09d5340a609f4fc7b1e9967be5561ddf" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 35113 } } peers { permanent_uuid: "cc019455aed34bcfa1d89817aee20ea8" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 37557 } }
I20250624 19:58:02.742218 27920 raft_consensus.cc:738] T e78e9c3d89d444dbba658c02276bb4ca P cc019455aed34bcfa1d89817aee20ea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc019455aed34bcfa1d89817aee20ea8, State: Initialized, Role: FOLLOWER
I20250624 19:58:02.743124 27920 consensus_queue.cc:260] T e78e9c3d89d444dbba658c02276bb4ca P cc019455aed34bcfa1d89817aee20ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdfbac95f3a41ed948315dd30b5e8a8" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38887 } } peers { permanent_uuid: "09d5340a609f4fc7b1e9967be5561ddf" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 35113 } } peers { permanent_uuid: "cc019455aed34bcfa1d89817aee20ea8" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 37557 } }
I20250624 19:58:02.747329 27895 heartbeater.cc:499] Master 127.26.193.254:40859 was elected leader, sending a full tablet report...
I20250624 19:58:02.747738 27921 ts_tablet_manager.cc:1428] T e78e9c3d89d444dbba658c02276bb4ca P 09d5340a609f4fc7b1e9967be5561ddf: Time spent starting tablet: real 0.038s user 0.031s sys 0.004s
I20250624 19:58:02.748315 27920 ts_tablet_manager.cc:1428] T e78e9c3d89d444dbba658c02276bb4ca P cc019455aed34bcfa1d89817aee20ea8: Time spent starting tablet: real 0.038s user 0.030s sys 0.004s
W20250624 19:58:02.785859 27763 tablet.cc:2378] T e78e9c3d89d444dbba658c02276bb4ca P cc019455aed34bcfa1d89817aee20ea8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 19:58:02.814370 27584 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e78e9c3d89d444dbba658c02276bb4ca"
dest_uuid: "0cdfbac95f3a41ed948315dd30b5e8a8"
from {username='slave'} at 127.0.0.1:39262
I20250624 19:58:02.814970 27584 raft_consensus.cc:491] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250624 19:58:02.815274 27584 raft_consensus.cc:3058] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [term 0 FOLLOWER]: Advancing to term 1
I20250624 19:58:02.819788 27584 raft_consensus.cc:513] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdfbac95f3a41ed948315dd30b5e8a8" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38887 } } peers { permanent_uuid: "09d5340a609f4fc7b1e9967be5561ddf" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 35113 } } peers { permanent_uuid: "cc019455aed34bcfa1d89817aee20ea8" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 37557 } }
I20250624 19:58:02.822268 27584 leader_election.cc:290] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [CANDIDATE]: Term 1 election: Requested vote from peers 09d5340a609f4fc7b1e9967be5561ddf (127.26.193.195:35113), cc019455aed34bcfa1d89817aee20ea8 (127.26.193.194:37557)
I20250624 19:58:02.841459 27850 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e78e9c3d89d444dbba658c02276bb4ca" candidate_uuid: "0cdfbac95f3a41ed948315dd30b5e8a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "09d5340a609f4fc7b1e9967be5561ddf"
I20250624 19:58:02.842486 27850 raft_consensus.cc:3058] T e78e9c3d89d444dbba658c02276bb4ca P 09d5340a609f4fc7b1e9967be5561ddf [term 0 FOLLOWER]: Advancing to term 1
I20250624 19:58:02.842571 27717 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e78e9c3d89d444dbba658c02276bb4ca" candidate_uuid: "0cdfbac95f3a41ed948315dd30b5e8a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cc019455aed34bcfa1d89817aee20ea8"
I20250624 19:58:02.843442 27717 raft_consensus.cc:3058] T e78e9c3d89d444dbba658c02276bb4ca P cc019455aed34bcfa1d89817aee20ea8 [term 0 FOLLOWER]: Advancing to term 1
I20250624 19:58:02.852456 27717 raft_consensus.cc:2466] T e78e9c3d89d444dbba658c02276bb4ca P cc019455aed34bcfa1d89817aee20ea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdfbac95f3a41ed948315dd30b5e8a8 in term 1.
I20250624 19:58:02.854451 27518 leader_election.cc:304] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cdfbac95f3a41ed948315dd30b5e8a8, cc019455aed34bcfa1d89817aee20ea8; no voters:
I20250624 19:58:02.855831 27925 raft_consensus.cc:2802] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 19:58:02.858950 27925 raft_consensus.cc:695] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [term 1 LEADER]: Becoming Leader. State: Replica: 0cdfbac95f3a41ed948315dd30b5e8a8, State: Running, Role: LEADER
I20250624 19:58:02.860306 27925 consensus_queue.cc:237] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdfbac95f3a41ed948315dd30b5e8a8" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38887 } } peers { permanent_uuid: "09d5340a609f4fc7b1e9967be5561ddf" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 35113 } } peers { permanent_uuid: "cc019455aed34bcfa1d89817aee20ea8" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 37557 } }
I20250624 19:58:02.863699 27850 raft_consensus.cc:2466] T e78e9c3d89d444dbba658c02276bb4ca P 09d5340a609f4fc7b1e9967be5561ddf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cdfbac95f3a41ed948315dd30b5e8a8 in term 1.
I20250624 19:58:02.882792 27439 catalog_manager.cc:5582] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cdfbac95f3a41ed948315dd30b5e8a8 (127.26.193.193). New cstate: current_term: 1 leader_uuid: "0cdfbac95f3a41ed948315dd30b5e8a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cdfbac95f3a41ed948315dd30b5e8a8" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09d5340a609f4fc7b1e9967be5561ddf" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 35113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc019455aed34bcfa1d89817aee20ea8" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 37557 } health_report { overall_health: UNKNOWN } } }
W20250624 19:58:02.914033 27630 tablet.cc:2378] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 19:58:02.938154 27896 tablet.cc:2378] T e78e9c3d89d444dbba658c02276bb4ca P 09d5340a609f4fc7b1e9967be5561ddf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250624 19:58:03.252156 27935 consensus_queue.cc:1035] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09d5340a609f4fc7b1e9967be5561ddf" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 35113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250624 19:58:03.329886 27925 consensus_queue.cc:1035] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc019455aed34bcfa1d89817aee20ea8" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 37557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250624 19:58:04.540163 27519 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.193.195:35113, user_credentials={real_user=slave}} blocked reactor thread for 45823.2us
W20250624 19:58:04.541200 27518 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.193.194:37557, user_credentials={real_user=slave}} blocked reactor thread for 47982.1us
I20250624 19:58:04.737037 27564 tablet_service.cc:1515] Processing DeleteTablet for tablet e78e9c3d89d444dbba658c02276bb4ca with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:39262
I20250624 19:58:04.745043 27967 tablet_replica.cc:331] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: stopping tablet replica
I20250624 19:58:04.745817 27967 raft_consensus.cc:2241] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [term 1 LEADER]: Raft consensus shutting down.
I20250624 19:58:04.746722 27967 raft_consensus.cc:2270] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250624 19:58:04.750293 27967 ts_tablet_manager.cc:1905] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250624 19:58:04.760989 27967 ts_tablet_manager.cc:1918] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.15
I20250624 19:58:04.761370 27967 log.cc:1199] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: Deleting WAL directory at /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestTabletCopyClearsLastLoggedOpId.1750795074187602-27399-0/minicluster-data/ts-0/wal/wals/e78e9c3d89d444dbba658c02276bb4ca
I20250624 19:58:04.770226 27968 ts_tablet_manager.cc:927] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: Initiating tablet copy from peer cc019455aed34bcfa1d89817aee20ea8 (127.26.193.194:37557)
I20250624 19:58:04.772317 27968 tablet_copy_client.cc:323] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: tablet copy: Beginning tablet copy session from remote peer at address 127.26.193.194:37557
I20250624 19:58:04.774278 27737 tablet_copy_service.cc:140] P cc019455aed34bcfa1d89817aee20ea8: Received BeginTabletCopySession request for tablet e78e9c3d89d444dbba658c02276bb4ca from peer 0cdfbac95f3a41ed948315dd30b5e8a8 ({username='slave'} at 127.26.193.193:59651)
I20250624 19:58:04.774792 27737 tablet_copy_service.cc:161] P cc019455aed34bcfa1d89817aee20ea8: Beginning new tablet copy session on tablet e78e9c3d89d444dbba658c02276bb4ca from peer 0cdfbac95f3a41ed948315dd30b5e8a8 at {username='slave'} at 127.26.193.193:59651: session id = 0cdfbac95f3a41ed948315dd30b5e8a8-e78e9c3d89d444dbba658c02276bb4ca
I20250624 19:58:04.780848 27737 tablet_copy_source_session.cc:215] T e78e9c3d89d444dbba658c02276bb4ca P cc019455aed34bcfa1d89817aee20ea8: Tablet Copy: opened 0 blocks and 1 log segments
I20250624 19:58:04.784070 27968 ts_tablet_manager.cc:1905] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250624 19:58:04.796231 27968 ts_tablet_manager.cc:1918] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.15
I20250624 19:58:04.797029 27968 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e78e9c3d89d444dbba658c02276bb4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 19:58:04.802702 27968 tablet_copy_client.cc:806] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: tablet copy: Starting download of 0 data blocks...
I20250624 19:58:04.803309 27968 tablet_copy_client.cc:670] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: tablet copy: Starting download of 1 WAL segments...
I20250624 19:58:04.808460 27968 tablet_copy_client.cc:538] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250624 19:58:04.815222 27968 tablet_bootstrap.cc:492] T e78e9c3d89d444dbba658c02276bb4ca P 0cdfbac95f3a41ed948315dd30b5e8a8: Bootstrap starting.
I20250624 19:58:04.824421 27399 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu with pid 27500
I20250624 19:58:04.855800 27399 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu with pid 27633
I20250624 19:58:04.888085 27399 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu with pid 27766
I20250624 19:58:04.919131 27399 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu with pid 27408
2025-06-24T19:58:04Z chronyd exiting
[ OK ] TabletCopyITest.TestTabletCopyClearsLastLoggedOpId (10595 ms)
[ RUN ] TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert
2025-06-24T19:58:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-24T19:58:04Z Disabled control of system clock
I20250624 19:58:05.001811 27399 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
/tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.193.254:33835
--webserver_interface=127.26.193.254
--webserver_port=0
--builtin_ntp_servers=127.26.193.212:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.26.193.254:33835 with env {}
W20250624 19:58:05.311631 27973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 19:58:05.312206 27973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 19:58:05.312660 27973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 19:58:05.344662 27973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250624 19:58:05.344967 27973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 19:58:05.345171 27973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250624 19:58:05.345384 27973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250624 19:58:05.386662 27973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.193.212:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.26.193.254:33835
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.26.193.254:33835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.26.193.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 19:43:22 UTC on 5fd53c4cbb9d
build id 6753
TSAN enabled
I20250624 19:58:05.388029 27973 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 19:58:05.389882 27973 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 19:58:05.407333 27980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:58:05.407325 27979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:58:05.409126 27982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:58:06.601269 27981 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 19:58:06.601363 27973 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 19:58:06.606269 27973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 19:58:06.609131 27973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 19:58:06.610576 27973 hybrid_clock.cc:648] HybridClock initialized: now 1750795086610525 us; error 59 us; skew 500 ppm
I20250624 19:58:06.611445 27973 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 19:58:06.619020 27973 webserver.cc:469] Webserver started at http://127.26.193.254:32963/ using document root <none> and password file <none>
I20250624 19:58:06.620095 27973 fs_manager.cc:362] Metadata directory not provided
I20250624 19:58:06.620347 27973 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 19:58:06.620862 27973 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 19:58:06.625931 27973 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/data/instance:
uuid: "17b3ce70c07a4c749bf499df35f87ec7"
format_stamp: "Formatted at 2025-06-24 19:58:06 on dist-test-slave-gtkq"
I20250624 19:58:06.627183 27973 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/wal/instance:
uuid: "17b3ce70c07a4c749bf499df35f87ec7"
format_stamp: "Formatted at 2025-06-24 19:58:06 on dist-test-slave-gtkq"
I20250624 19:58:06.635166 27973 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.004s
I20250624 19:58:06.641423 27989 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:06.642650 27973 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250624 19:58:06.643015 27973 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/data,/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/wal
uuid: "17b3ce70c07a4c749bf499df35f87ec7"
format_stamp: "Formatted at 2025-06-24 19:58:06 on dist-test-slave-gtkq"
I20250624 19:58:06.643388 27973 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 19:58:06.699735 27973 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 19:58:06.701377 27973 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 19:58:06.701858 27973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 19:58:06.776806 27973 rpc_server.cc:307] RPC server started. Bound to: 127.26.193.254:33835
I20250624 19:58:06.776880 28040 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.193.254:33835 every 8 connection(s)
I20250624 19:58:06.779771 27973 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/data/info.pb
I20250624 19:58:06.785503 28041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 19:58:06.788076 27399 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu as pid 27973
I20250624 19:58:06.788630 27399 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/master-0/wal/instance
I20250624 19:58:06.809444 28041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7: Bootstrap starting.
I20250624 19:58:06.816417 28041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7: Neither blocks nor log segments found. Creating new log.
I20250624 19:58:06.818449 28041 log.cc:826] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7: Log is configured to *not* fsync() on all Append() calls
I20250624 19:58:06.823421 28041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7: No bootstrap required, opened a new log
I20250624 19:58:06.841924 28041 raft_consensus.cc:357] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b3ce70c07a4c749bf499df35f87ec7" member_type: VOTER last_known_addr { host: "127.26.193.254" port: 33835 } }
I20250624 19:58:06.842682 28041 raft_consensus.cc:383] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 19:58:06.842893 28041 raft_consensus.cc:738] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b3ce70c07a4c749bf499df35f87ec7, State: Initialized, Role: FOLLOWER
I20250624 19:58:06.843634 28041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b3ce70c07a4c749bf499df35f87ec7" member_type: VOTER last_known_addr { host: "127.26.193.254" port: 33835 } }
I20250624 19:58:06.844262 28041 raft_consensus.cc:397] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250624 19:58:06.844569 28041 raft_consensus.cc:491] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250624 19:58:06.844957 28041 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [term 0 FOLLOWER]: Advancing to term 1
I20250624 19:58:06.849272 28041 raft_consensus.cc:513] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b3ce70c07a4c749bf499df35f87ec7" member_type: VOTER last_known_addr { host: "127.26.193.254" port: 33835 } }
I20250624 19:58:06.850076 28041 leader_election.cc:304] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17b3ce70c07a4c749bf499df35f87ec7; no voters:
I20250624 19:58:06.851902 28041 leader_election.cc:290] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250624 19:58:06.852702 28046 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 19:58:06.855283 28046 raft_consensus.cc:695] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [term 1 LEADER]: Becoming Leader. State: Replica: 17b3ce70c07a4c749bf499df35f87ec7, State: Running, Role: LEADER
I20250624 19:58:06.856151 28046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b3ce70c07a4c749bf499df35f87ec7" member_type: VOTER last_known_addr { host: "127.26.193.254" port: 33835 } }
I20250624 19:58:06.857196 28041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [sys.catalog]: configured and running, proceeding with master startup.
I20250624 19:58:06.866820 28048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17b3ce70c07a4c749bf499df35f87ec7. Latest consensus state: current_term: 1 leader_uuid: "17b3ce70c07a4c749bf499df35f87ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b3ce70c07a4c749bf499df35f87ec7" member_type: VOTER last_known_addr { host: "127.26.193.254" port: 33835 } } }
I20250624 19:58:06.867589 28048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [sys.catalog]: This master's current role is: LEADER
I20250624 19:58:06.866726 28047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17b3ce70c07a4c749bf499df35f87ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b3ce70c07a4c749bf499df35f87ec7" member_type: VOTER last_known_addr { host: "127.26.193.254" port: 33835 } } }
I20250624 19:58:06.868494 28047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7 [sys.catalog]: This master's current role is: LEADER
I20250624 19:58:06.871481 28054 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250624 19:58:06.885272 28054 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250624 19:58:06.902925 28054 catalog_manager.cc:1349] Generated new cluster ID: 2351ff0261824501b55bb83f187dff4c
I20250624 19:58:06.903213 28054 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250624 19:58:06.922741 28054 catalog_manager.cc:1372] Generated new certificate authority record
I20250624 19:58:06.924353 28054 catalog_manager.cc:1506] Loading token signing keys...
I20250624 19:58:06.941462 28054 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 17b3ce70c07a4c749bf499df35f87ec7: Generated new TSK 0
I20250624 19:58:06.942449 28054 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250624 19:58:06.956173 27399 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
/tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.193.193:0
--local_ip_for_outbound_sockets=127.26.193.193
--webserver_interface=127.26.193.193
--webserver_port=0
--tserver_master_addrs=127.26.193.254:33835
--builtin_ntp_servers=127.26.193.212:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
W20250624 19:58:07.312079 28065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 19:58:07.312765 28065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 19:58:07.313474 28065 flags.cc:425] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20250624 19:58:07.313690 28065 flags.cc:425] Enabled unsafe flag: --enable_flush_memrowset=false
W20250624 19:58:07.314081 28065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 19:58:07.347257 28065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 19:58:07.348273 28065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.193.193
I20250624 19:58:07.392042 28065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.193.212:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.193.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.26.193.193
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.26.193.254:33835
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.193.193
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 19:43:22 UTC on 5fd53c4cbb9d
build id 6753
TSAN enabled
I20250624 19:58:07.393503 28065 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 19:58:07.395378 28065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 19:58:07.414325 28072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 19:58:07.423225 28065 server_base.cc:1048] running on GCE node
W20250624 19:58:07.422349 28071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:58:07.422693 28074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 19:58:08.622485 28065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 19:58:08.625636 28065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 19:58:08.627244 28065 hybrid_clock.cc:648] HybridClock initialized: now 1750795088627193 us; error 43 us; skew 500 ppm
I20250624 19:58:08.628432 28065 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 19:58:08.642024 28065 webserver.cc:469] Webserver started at http://127.26.193.193:43745/ using document root <none> and password file <none>
I20250624 19:58:08.643404 28065 fs_manager.cc:362] Metadata directory not provided
I20250624 19:58:08.643726 28065 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 19:58:08.644345 28065 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 19:58:08.651371 28065 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/data/instance:
uuid: "7d9253b8899d4db8b7cc997c18404393"
format_stamp: "Formatted at 2025-06-24 19:58:08 on dist-test-slave-gtkq"
I20250624 19:58:08.653092 28065 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/wal/instance:
uuid: "7d9253b8899d4db8b7cc997c18404393"
format_stamp: "Formatted at 2025-06-24 19:58:08 on dist-test-slave-gtkq"
I20250624 19:58:08.663663 28065 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.005s sys 0.004s
I20250624 19:58:08.672084 28081 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:08.673597 28065 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20250624 19:58:08.674072 28065 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/wal
uuid: "7d9253b8899d4db8b7cc997c18404393"
format_stamp: "Formatted at 2025-06-24 19:58:08 on dist-test-slave-gtkq"
I20250624 19:58:08.674680 28065 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 19:58:08.763872 28065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 19:58:08.765545 28065 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 19:58:08.766033 28065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 19:58:08.768633 28065 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 19:58:08.773113 28065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 19:58:08.773386 28065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:08.773674 28065 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 19:58:08.773840 28065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:08.916524 28065 rpc_server.cc:307] RPC server started. Bound to: 127.26.193.193:38569
I20250624 19:58:08.916713 28193 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.193.193:38569 every 8 connection(s)
I20250624 19:58:08.919494 28065 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/data/info.pb
I20250624 19:58:08.924923 27399 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu as pid 28065
I20250624 19:58:08.925380 27399 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-0/wal/instance
I20250624 19:58:08.932382 27399 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
/tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.193.194:0
--local_ip_for_outbound_sockets=127.26.193.194
--webserver_interface=127.26.193.194
--webserver_port=0
--tserver_master_addrs=127.26.193.254:33835
--builtin_ntp_servers=127.26.193.212:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
I20250624 19:58:08.946142 28194 heartbeater.cc:344] Connected to a master server at 127.26.193.254:33835
I20250624 19:58:08.946656 28194 heartbeater.cc:461] Registering TS with master...
I20250624 19:58:08.947871 28194 heartbeater.cc:507] Master 127.26.193.254:33835 requested a full tablet report, sending...
I20250624 19:58:08.951296 28006 ts_manager.cc:194] Registered new tserver with Master: 7d9253b8899d4db8b7cc997c18404393 (127.26.193.193:38569)
I20250624 19:58:08.954700 28006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.193.193:47203
W20250624 19:58:09.245463 28198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 19:58:09.245947 28198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 19:58:09.246305 28198 flags.cc:425] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20250624 19:58:09.246493 28198 flags.cc:425] Enabled unsafe flag: --enable_flush_memrowset=false
W20250624 19:58:09.246754 28198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 19:58:09.279215 28198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 19:58:09.280177 28198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.193.194
I20250624 19:58:09.315954 28198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.193.212:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.193.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.26.193.194
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.26.193.254:33835
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.193.194
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 19:43:22 UTC on 5fd53c4cbb9d
build id 6753
TSAN enabled
I20250624 19:58:09.317446 28198 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 19:58:09.319224 28198 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 19:58:09.336005 28205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 19:58:09.959444 28194 heartbeater.cc:499] Master 127.26.193.254:33835 was elected leader, sending a full tablet report...
W20250624 19:58:09.335992 28204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:58:10.541702 28207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:58:10.543880 28206 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1205 milliseconds
I20250624 19:58:10.543989 28198 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 19:58:10.545238 28198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 19:58:10.548111 28198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 19:58:10.549674 28198 hybrid_clock.cc:648] HybridClock initialized: now 1750795090549618 us; error 62 us; skew 500 ppm
I20250624 19:58:10.550539 28198 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 19:58:10.557773 28198 webserver.cc:469] Webserver started at http://127.26.193.194:40863/ using document root <none> and password file <none>
I20250624 19:58:10.558888 28198 fs_manager.cc:362] Metadata directory not provided
I20250624 19:58:10.559119 28198 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 19:58:10.559583 28198 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 19:58:10.564234 28198 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/data/instance:
uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5"
format_stamp: "Formatted at 2025-06-24 19:58:10 on dist-test-slave-gtkq"
I20250624 19:58:10.565478 28198 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/wal/instance:
uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5"
format_stamp: "Formatted at 2025-06-24 19:58:10 on dist-test-slave-gtkq"
I20250624 19:58:10.573113 28198 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250624 19:58:10.580055 28214 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:10.581377 28198 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20250624 19:58:10.581723 28198 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/wal
uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5"
format_stamp: "Formatted at 2025-06-24 19:58:10 on dist-test-slave-gtkq"
I20250624 19:58:10.582046 28198 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 19:58:10.637022 28198 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 19:58:10.638550 28198 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 19:58:10.638998 28198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 19:58:10.641585 28198 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 19:58:10.646015 28198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 19:58:10.646275 28198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:10.646492 28198 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 19:58:10.646637 28198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:10.793056 28198 rpc_server.cc:307] RPC server started. Bound to: 127.26.193.194:46567
I20250624 19:58:10.793182 28326 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.193.194:46567 every 8 connection(s)
I20250624 19:58:10.795887 28198 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/data/info.pb
I20250624 19:58:10.800058 27399 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu as pid 28198
I20250624 19:58:10.800573 27399 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-1/wal/instance
I20250624 19:58:10.807682 27399 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
/tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.26.193.195:0
--local_ip_for_outbound_sockets=127.26.193.195
--webserver_interface=127.26.193.195
--webserver_port=0
--tserver_master_addrs=127.26.193.254:33835
--builtin_ntp_servers=127.26.193.212:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
I20250624 19:58:10.818560 28327 heartbeater.cc:344] Connected to a master server at 127.26.193.254:33835
I20250624 19:58:10.819169 28327 heartbeater.cc:461] Registering TS with master...
I20250624 19:58:10.820750 28327 heartbeater.cc:507] Master 127.26.193.254:33835 requested a full tablet report, sending...
I20250624 19:58:10.823892 28006 ts_manager.cc:194] Registered new tserver with Master: db7bdfbb301b4d0a99e1a6b61e0b1ad5 (127.26.193.194:46567)
I20250624 19:58:10.825268 28006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.193.194:58241
W20250624 19:58:11.120204 28331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250624 19:58:11.120790 28331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250624 19:58:11.121224 28331 flags.cc:425] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20250624 19:58:11.121417 28331 flags.cc:425] Enabled unsafe flag: --enable_flush_memrowset=false
W20250624 19:58:11.121773 28331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250624 19:58:11.154299 28331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250624 19:58:11.155311 28331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.26.193.195
I20250624 19:58:11.191814 28331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.26.193.212:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.26.193.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.26.193.195
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.26.193.254:33835
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.26.193.195
--log_dir=/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 24 Jun 2025 19:43:22 UTC on 5fd53c4cbb9d
build id 6753
TSAN enabled
I20250624 19:58:11.193285 28331 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250624 19:58:11.195145 28331 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250624 19:58:11.211870 28338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250624 19:58:11.829396 28327 heartbeater.cc:499] Master 127.26.193.254:33835 was elected leader, sending a full tablet report...
W20250624 19:58:11.211934 28337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:58:11.217351 28340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250624 19:58:12.406937 28339 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250624 19:58:12.407034 28331 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250624 19:58:12.411773 28331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250624 19:58:12.414896 28331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250624 19:58:12.416502 28331 hybrid_clock.cc:648] HybridClock initialized: now 1750795092416442 us; error 71 us; skew 500 ppm
I20250624 19:58:12.417546 28331 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250624 19:58:12.424944 28331 webserver.cc:469] Webserver started at http://127.26.193.195:42225/ using document root <none> and password file <none>
I20250624 19:58:12.426036 28331 fs_manager.cc:362] Metadata directory not provided
I20250624 19:58:12.426313 28331 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250624 19:58:12.426796 28331 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250624 19:58:12.432116 28331 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/data/instance:
uuid: "07bfec0a5d354cf58ddb356ca8b26e46"
format_stamp: "Formatted at 2025-06-24 19:58:12 on dist-test-slave-gtkq"
I20250624 19:58:12.433660 28331 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/wal/instance:
uuid: "07bfec0a5d354cf58ddb356ca8b26e46"
format_stamp: "Formatted at 2025-06-24 19:58:12 on dist-test-slave-gtkq"
I20250624 19:58:12.443856 28331 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20250624 19:58:12.450330 28347 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:12.451539 28331 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250624 19:58:12.451917 28331 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/wal
uuid: "07bfec0a5d354cf58ddb356ca8b26e46"
format_stamp: "Formatted at 2025-06-24 19:58:12 on dist-test-slave-gtkq"
I20250624 19:58:12.452291 28331 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250624 19:58:12.503962 28331 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250624 19:58:12.505550 28331 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250624 19:58:12.506038 28331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250624 19:58:12.508646 28331 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250624 19:58:12.513695 28331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250624 19:58:12.513936 28331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:12.514199 28331 ts_tablet_manager.cc:610] Registered 0 tablets
I20250624 19:58:12.514369 28331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250624 19:58:12.658998 28331 rpc_server.cc:307] RPC server started. Bound to: 127.26.193.195:44151
I20250624 19:58:12.659104 28459 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.26.193.195:44151 every 8 connection(s)
I20250624 19:58:12.661772 28331 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/data/info.pb
I20250624 19:58:12.665700 27399 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu as pid 28331
I20250624 19:58:12.666240 27399 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0/minicluster-data/ts-2/wal/instance
I20250624 19:58:12.685958 28460 heartbeater.cc:344] Connected to a master server at 127.26.193.254:33835
I20250624 19:58:12.686532 28460 heartbeater.cc:461] Registering TS with master...
I20250624 19:58:12.687898 28460 heartbeater.cc:507] Master 127.26.193.254:33835 requested a full tablet report, sending...
I20250624 19:58:12.690380 28006 ts_manager.cc:194] Registered new tserver with Master: 07bfec0a5d354cf58ddb356ca8b26e46 (127.26.193.195:44151)
I20250624 19:58:12.691709 28006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.26.193.195:44745
I20250624 19:58:12.703159 27399 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250624 19:58:12.734633 27399 test_util.cc:276] Using random seed: -1080839533
I20250624 19:58:12.778250 28006 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35680:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250624 19:58:12.780921 28006 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250624 19:58:12.857785 28262 tablet_service.cc:1468] Processing CreateTablet for tablet ce5e782ad40747bf92261df05e75bd86 (DEFAULT_TABLE table=test-workload [id=fc7d122d54f648a7be7da7cae73340bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 19:58:12.857045 28129 tablet_service.cc:1468] Processing CreateTablet for tablet ce5e782ad40747bf92261df05e75bd86 (DEFAULT_TABLE table=test-workload [id=fc7d122d54f648a7be7da7cae73340bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 19:58:12.859737 28262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce5e782ad40747bf92261df05e75bd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 19:58:12.859951 28129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce5e782ad40747bf92261df05e75bd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 19:58:12.861848 28395 tablet_service.cc:1468] Processing CreateTablet for tablet ce5e782ad40747bf92261df05e75bd86 (DEFAULT_TABLE table=test-workload [id=fc7d122d54f648a7be7da7cae73340bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250624 19:58:12.864104 28395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce5e782ad40747bf92261df05e75bd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250624 19:58:12.886637 28484 tablet_bootstrap.cc:492] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5: Bootstrap starting.
I20250624 19:58:12.891986 28485 tablet_bootstrap.cc:492] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46: Bootstrap starting.
I20250624 19:58:12.894695 28486 tablet_bootstrap.cc:492] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393: Bootstrap starting.
I20250624 19:58:12.895462 28484 tablet_bootstrap.cc:654] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5: Neither blocks nor log segments found. Creating new log.
I20250624 19:58:12.897666 28484 log.cc:826] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5: Log is configured to *not* fsync() on all Append() calls
I20250624 19:58:12.903014 28486 tablet_bootstrap.cc:654] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393: Neither blocks nor log segments found. Creating new log.
I20250624 19:58:12.903101 28485 tablet_bootstrap.cc:654] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46: Neither blocks nor log segments found. Creating new log.
I20250624 19:58:12.903132 28484 tablet_bootstrap.cc:492] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5: No bootstrap required, opened a new log
I20250624 19:58:12.903643 28484 ts_tablet_manager.cc:1397] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20250624 19:58:12.905730 28485 log.cc:826] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46: Log is configured to *not* fsync() on all Append() calls
I20250624 19:58:12.905730 28486 log.cc:826] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393: Log is configured to *not* fsync() on all Append() calls
I20250624 19:58:12.911726 28486 tablet_bootstrap.cc:492] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393: No bootstrap required, opened a new log
I20250624 19:58:12.912403 28486 ts_tablet_manager.cc:1397] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.005s
I20250624 19:58:12.912757 28485 tablet_bootstrap.cc:492] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46: No bootstrap required, opened a new log
I20250624 19:58:12.913378 28485 ts_tablet_manager.cc:1397] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.010s
I20250624 19:58:12.931151 28484 raft_consensus.cc:357] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9253b8899d4db8b7cc997c18404393" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38569 } } peers { permanent_uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 46567 } } peers { permanent_uuid: "07bfec0a5d354cf58ddb356ca8b26e46" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 44151 } }
I20250624 19:58:12.932358 28484 raft_consensus.cc:383] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 19:58:12.932767 28484 raft_consensus.cc:738] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7bdfbb301b4d0a99e1a6b61e0b1ad5, State: Initialized, Role: FOLLOWER
I20250624 19:58:12.933933 28484 consensus_queue.cc:260] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9253b8899d4db8b7cc997c18404393" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38569 } } peers { permanent_uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 46567 } } peers { permanent_uuid: "07bfec0a5d354cf58ddb356ca8b26e46" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 44151 } }
I20250624 19:58:12.944558 28486 raft_consensus.cc:357] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9253b8899d4db8b7cc997c18404393" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38569 } } peers { permanent_uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 46567 } } peers { permanent_uuid: "07bfec0a5d354cf58ddb356ca8b26e46" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 44151 } }
I20250624 19:58:12.945813 28486 raft_consensus.cc:383] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 19:58:12.946190 28486 raft_consensus.cc:738] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d9253b8899d4db8b7cc997c18404393, State: Initialized, Role: FOLLOWER
I20250624 19:58:12.947136 28485 raft_consensus.cc:357] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9253b8899d4db8b7cc997c18404393" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38569 } } peers { permanent_uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 46567 } } peers { permanent_uuid: "07bfec0a5d354cf58ddb356ca8b26e46" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 44151 } }
I20250624 19:58:12.948141 28485 raft_consensus.cc:383] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250624 19:58:12.948495 28485 raft_consensus.cc:738] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bfec0a5d354cf58ddb356ca8b26e46, State: Initialized, Role: FOLLOWER
I20250624 19:58:12.949115 28484 ts_tablet_manager.cc:1428] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5: Time spent starting tablet: real 0.045s user 0.039s sys 0.008s
I20250624 19:58:12.949658 28485 consensus_queue.cc:260] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9253b8899d4db8b7cc997c18404393" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38569 } } peers { permanent_uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 46567 } } peers { permanent_uuid: "07bfec0a5d354cf58ddb356ca8b26e46" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 44151 } }
I20250624 19:58:12.948944 28486 consensus_queue.cc:260] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9253b8899d4db8b7cc997c18404393" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38569 } } peers { permanent_uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 46567 } } peers { permanent_uuid: "07bfec0a5d354cf58ddb356ca8b26e46" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 44151 } }
I20250624 19:58:12.975150 28460 heartbeater.cc:499] Master 127.26.193.254:33835 was elected leader, sending a full tablet report...
I20250624 19:58:12.975126 28493 raft_consensus.cc:491] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250624 19:58:12.976341 28485 ts_tablet_manager.cc:1428] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46: Time spent starting tablet: real 0.063s user 0.035s sys 0.022s
I20250624 19:58:12.976724 28493 raft_consensus.cc:513] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9253b8899d4db8b7cc997c18404393" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38569 } } peers { permanent_uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 46567 } } peers { permanent_uuid: "07bfec0a5d354cf58ddb356ca8b26e46" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 44151 } }
I20250624 19:58:12.977243 28486 ts_tablet_manager.cc:1428] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393: Time spent starting tablet: real 0.064s user 0.038s sys 0.018s
I20250624 19:58:12.983716 28493 leader_election.cc:290] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db7bdfbb301b4d0a99e1a6b61e0b1ad5 (127.26.193.194:46567), 07bfec0a5d354cf58ddb356ca8b26e46 (127.26.193.195:44151)
I20250624 19:58:12.997589 28282 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce5e782ad40747bf92261df05e75bd86" candidate_uuid: "7d9253b8899d4db8b7cc997c18404393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5" is_pre_election: true
I20250624 19:58:12.997653 28415 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce5e782ad40747bf92261df05e75bd86" candidate_uuid: "7d9253b8899d4db8b7cc997c18404393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bfec0a5d354cf58ddb356ca8b26e46" is_pre_election: true
I20250624 19:58:12.998373 28282 raft_consensus.cc:2466] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9253b8899d4db8b7cc997c18404393 in term 0.
I20250624 19:58:12.998375 28415 raft_consensus.cc:2466] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d9253b8899d4db8b7cc997c18404393 in term 0.
I20250624 19:58:12.999663 28083 leader_election.cc:304] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07bfec0a5d354cf58ddb356ca8b26e46, 7d9253b8899d4db8b7cc997c18404393; no voters:
I20250624 19:58:13.000460 28493 raft_consensus.cc:2802] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250624 19:58:13.000846 28493 raft_consensus.cc:491] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250624 19:58:13.001076 28493 raft_consensus.cc:3058] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [term 0 FOLLOWER]: Advancing to term 1
I20250624 19:58:13.005579 28493 raft_consensus.cc:513] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9253b8899d4db8b7cc997c18404393" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38569 } } peers { permanent_uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 46567 } } peers { permanent_uuid: "07bfec0a5d354cf58ddb356ca8b26e46" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 44151 } }
I20250624 19:58:13.007006 28493 leader_election.cc:290] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [CANDIDATE]: Term 1 election: Requested vote from peers db7bdfbb301b4d0a99e1a6b61e0b1ad5 (127.26.193.194:46567), 07bfec0a5d354cf58ddb356ca8b26e46 (127.26.193.195:44151)
I20250624 19:58:13.008008 28282 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce5e782ad40747bf92261df05e75bd86" candidate_uuid: "7d9253b8899d4db8b7cc997c18404393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5"
I20250624 19:58:13.008145 28415 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce5e782ad40747bf92261df05e75bd86" candidate_uuid: "7d9253b8899d4db8b7cc997c18404393" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07bfec0a5d354cf58ddb356ca8b26e46"
I20250624 19:58:13.008452 28282 raft_consensus.cc:3058] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5 [term 0 FOLLOWER]: Advancing to term 1
I20250624 19:58:13.008697 28415 raft_consensus.cc:3058] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46 [term 0 FOLLOWER]: Advancing to term 1
I20250624 19:58:13.013306 28282 raft_consensus.cc:2466] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9253b8899d4db8b7cc997c18404393 in term 1.
I20250624 19:58:13.013312 28415 raft_consensus.cc:2466] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d9253b8899d4db8b7cc997c18404393 in term 1.
I20250624 19:58:13.014609 28085 leader_election.cc:304] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d9253b8899d4db8b7cc997c18404393, db7bdfbb301b4d0a99e1a6b61e0b1ad5; no voters:
I20250624 19:58:13.015367 28493 raft_consensus.cc:2802] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [term 1 FOLLOWER]: Leader election won for term 1
I20250624 19:58:13.015769 28493 raft_consensus.cc:695] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [term 1 LEADER]: Becoming Leader. State: Replica: 7d9253b8899d4db8b7cc997c18404393, State: Running, Role: LEADER
I20250624 19:58:13.016705 28493 consensus_queue.cc:237] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9253b8899d4db8b7cc997c18404393" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38569 } } peers { permanent_uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 46567 } } peers { permanent_uuid: "07bfec0a5d354cf58ddb356ca8b26e46" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 44151 } }
I20250624 19:58:13.028502 28005 catalog_manager.cc:5582] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d9253b8899d4db8b7cc997c18404393 (127.26.193.193). New cstate: current_term: 1 leader_uuid: "7d9253b8899d4db8b7cc997c18404393" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d9253b8899d4db8b7cc997c18404393" member_type: VOTER last_known_addr { host: "127.26.193.193" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 46567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07bfec0a5d354cf58ddb356ca8b26e46" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 44151 } health_report { overall_health: UNKNOWN } } }
W20250624 19:58:13.053077 28328 tablet.cc:2378] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 19:58:13.053565 28328 tablet_replica_mm_ops.cc:240] Deltamemstore flush is disabled (check --enable_flush_deltamemstores)
W20250624 19:58:13.053722 28328 tablet_replica_mm_ops.cc:163] Memrowset flush is disabled (check --enable_flush_memrowset)
I20250624 19:58:13.099134 28149 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ce5e782ad40747bf92261df05e75bd86"
dest_uuid: "7d9253b8899d4db8b7cc997c18404393"
from {username='slave'} at 127.0.0.1:51652
I20250624 19:58:13.099725 28149 raft_consensus.cc:474] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [term 1 LEADER]: Not starting forced leader election -- already a leader
W20250624 19:58:13.166918 28461 tablet.cc:2378] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 19:58:13.167546 28461 tablet_replica_mm_ops.cc:240] Deltamemstore flush is disabled (check --enable_flush_deltamemstores)
W20250624 19:58:13.167791 28461 tablet_replica_mm_ops.cc:163] Memrowset flush is disabled (check --enable_flush_memrowset)
W20250624 19:58:13.178750 28195 tablet.cc:2378] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250624 19:58:13.179404 28195 tablet_replica_mm_ops.cc:240] Deltamemstore flush is disabled (check --enable_flush_deltamemstores)
W20250624 19:58:13.179639 28195 tablet_replica_mm_ops.cc:163] Memrowset flush is disabled (check --enable_flush_memrowset)
I20250624 19:58:13.298125 28415 raft_consensus.cc:1273] T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46 [term 1 FOLLOWER]: Refusing update from remote peer 7d9253b8899d4db8b7cc997c18404393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 19:58:13.298523 28282 raft_consensus.cc:1273] T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5 [term 1 FOLLOWER]: Refusing update from remote peer 7d9253b8899d4db8b7cc997c18404393: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250624 19:58:13.300209 28492 consensus_queue.cc:1035] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bfec0a5d354cf58ddb356ca8b26e46" member_type: VOTER last_known_addr { host: "127.26.193.195" port: 44151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250624 19:58:13.301004 28493 consensus_queue.cc:1035] T ce5e782ad40747bf92261df05e75bd86 P 7d9253b8899d4db8b7cc997c18404393 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7bdfbb301b4d0a99e1a6b61e0b1ad5" member_type: VOTER last_known_addr { host: "127.26.193.194" port: 46567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250624 19:58:13.350402 28510 mvcc.cc:204] Tried to move back new op lower bound from 7171256702132072448 to 7171256701004972032. Current Snapshot: MvccSnapshot[applied={T|T < 7171256702132072448}]
I20250624 19:58:13.422830 28508 mvcc.cc:204] Tried to move back new op lower bound from 7171256702290644992 to 7171256701004972032. Current Snapshot: MvccSnapshot[applied={T|T < 7171256702205177856}]
W20250624 19:58:13.584344 28470 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.26.193.193:38569, user_credentials={real_user=slave}} blocked reactor thread for 43525.3us
W20250624 19:58:16.031018 28502 meta_cache.cc:1261] Time spent looking up entry by key: real 0.067s user 0.003s sys 0.000s
W20250624 19:58:17.613243 28498 meta_cache.cc:1261] Time spent looking up entry by key: real 0.078s user 0.003s sys 0.000s
W20250624 19:58:18.052074 28498 meta_cache.cc:1261] Time spent looking up entry by key: real 0.065s user 0.009s sys 0.000s
W20250624 19:58:18.052506 28504 meta_cache.cc:1261] Time spent looking up entry by key: real 0.072s user 0.002s sys 0.015s
W20250624 19:58:18.052906 28500 meta_cache.cc:1261] Time spent looking up entry by key: real 0.076s user 0.006s sys 0.012s
W20250624 19:58:18.052194 28505 meta_cache.cc:1261] Time spent looking up entry by key: real 0.052s user 0.004s sys 0.004s
W20250624 19:58:18.064904 28502 meta_cache.cc:1261] Time spent looking up entry by key: real 0.070s user 0.022s sys 0.005s
W20250624 19:58:18.052707 28503 meta_cache.cc:1261] Time spent looking up entry by key: real 0.084s user 0.006s sys 0.000s
W20250624 19:58:19.405758 28498 meta_cache.cc:1261] Time spent looking up entry by key: real 0.057s user 0.004s sys 0.000s
W20250624 19:58:20.239781 28504 meta_cache.cc:1261] Time spent looking up entry by key: real 0.081s user 0.000s sys 0.000s
W20250624 19:58:20.248880 28498 meta_cache.cc:1261] Time spent looking up entry by key: real 0.078s user 0.001s sys 0.000s
W20250624 19:58:20.760493 28498 meta_cache.cc:1261] Time spent looking up entry by key: real 0.073s user 0.000s sys 0.000s
W20250624 19:58:21.971153 28501 meta_cache.cc:1261] Time spent looking up entry by key: real 0.058s user 0.004s sys 0.000s
W20250624 19:58:23.052042 28470 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.26.193.193:38569, user_credentials={real_user=slave}} blocked reactor thread for 48521.3us
W20250624 19:58:24.168560 28502 meta_cache.cc:1261] Time spent looking up entry by key: real 0.072s user 0.000s sys 0.000s
W20250624 19:58:24.214550 28504 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s user 0.000s sys 0.000s
W20250624 19:58:25.359248 28501 meta_cache.cc:1261] Time spent looking up entry by key: real 0.067s user 0.002s sys 0.000s
W20250624 19:58:26.251339 28500 meta_cache.cc:1261] Time spent looking up entry by key: real 0.063s user 0.004s sys 0.000s
W20250624 19:58:30.472818 28498 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s user 0.001s sys 0.000s
W20250624 19:58:30.525597 28470 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.26.193.193:38569, user_credentials={real_user=slave}} blocked reactor thread for 66930.6us
W20250624 19:58:30.993170 28511 log.cc:927] Time spent T ce5e782ad40747bf92261df05e75bd86 P 07bfec0a5d354cf58ddb356ca8b26e46: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
W20250624 19:58:31.701634 28501 meta_cache.cc:1261] Time spent looking up entry by key: real 0.085s user 0.003s sys 0.000s
W20250624 19:58:33.375159 28502 meta_cache.cc:1261] Time spent looking up entry by key: real 0.065s user 0.009s sys 0.007s
W20250624 19:58:33.733107 28083 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.193.195:44151, user_credentials={real_user=slave}} blocked reactor thread for 58865.9us
W20250624 19:58:34.551723 28498 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s user 0.004s sys 0.000s
W20250624 19:58:35.704823 28504 meta_cache.cc:1261] Time spent looking up entry by key: real 0.056s user 0.001s sys 0.000s
W20250624 19:58:36.996701 28504 meta_cache.cc:1261] Time spent looking up entry by key: real 0.093s user 0.003s sys 0.000s
W20250624 19:58:38.038112 28512 log.cc:927] Time spent T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
W20250624 19:58:38.662976 28470 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.26.193.193:38569, user_credentials={real_user=slave}} blocked reactor thread for 61265.7us
W20250624 19:58:41.969933 28498 meta_cache.cc:1261] Time spent looking up entry by key: real 0.060s user 0.001s sys 0.000s
W20250624 19:58:44.212682 28498 meta_cache.cc:1261] Time spent looking up entry by key: real 0.098s user 0.002s sys 0.000s
W20250624 19:58:44.462743 28500 meta_cache.cc:1261] Time spent looking up entry by key: real 0.050s user 0.004s sys 0.000s
W20250624 19:58:44.966176 28499 meta_cache.cc:1261] Time spent looking up entry by key: real 0.089s user 0.000s sys 0.000s
W20250624 19:58:44.966640 28498 meta_cache.cc:1261] Time spent looking up entry by key: real 0.089s user 0.000s sys 0.000s
W20250624 19:58:44.975420 28501 meta_cache.cc:1261] Time spent looking up entry by key: real 0.094s user 0.000s sys 0.001s
W20250624 19:58:45.000847 28504 meta_cache.cc:1261] Time spent looking up entry by key: real 0.124s user 0.015s sys 0.046s
W20250624 19:58:45.160938 28323 debug-util.cc:398] Leaking SignalData structure 0x7b08000c1be0 after lost signal to thread 28199
W20250624 19:58:45.162147 28323 debug-util.cc:398] Leaking SignalData structure 0x7b0800081520 after lost signal to thread 28326
W20250624 19:58:46.500890 28502 meta_cache.cc:1261] Time spent looking up entry by key: real 0.055s user 0.000s sys 0.000s
W20250624 19:58:48.390103 28502 meta_cache.cc:1261] Time spent looking up entry by key: real 0.076s user 0.002s sys 0.000s
W20250624 19:58:48.643580 28470 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.26.193.193:38569, user_credentials={real_user=slave}} blocked reactor thread for 62359.8us
W20250624 19:58:48.717217 28470 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.26.193.193:38569, user_credentials={real_user=slave}} blocked reactor thread for 65342.8us
W20250624 19:58:49.534250 28470 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.26.193.193:38569, user_credentials={real_user=slave}} blocked reactor thread for 45319.4us
W20250624 19:58:50.881415 28504 meta_cache.cc:1261] Time spent looking up entry by key: real 0.067s user 0.001s sys 0.000s
W20250624 19:58:54.277228 28504 meta_cache.cc:1261] Time spent looking up entry by key: real 0.054s user 0.002s sys 0.000s
W20250624 19:58:55.061208 28470 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.26.193.193:38569, user_credentials={real_user=slave}} blocked reactor thread for 70830.8us
W20250624 19:58:55.128785 28502 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s user 0.001s sys 0.000s
W20250624 19:58:55.652822 28498 meta_cache.cc:1261] Time spent looking up entry by key: real 0.072s user 0.004s sys 0.000s
W20250624 19:58:58.364806 28502 meta_cache.cc:1261] Time spent looking up entry by key: real 0.068s user 0.000s sys 0.000s
W20250624 19:58:59.003151 28085 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.193.194:46567, user_credentials={real_user=slave}} blocked reactor thread for 50489.1us
W20250624 19:58:59.849165 28085 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.193.194:46567, user_credentials={real_user=slave}} blocked reactor thread for 48161.4us
W20250624 19:58:59.851502 28083 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.26.193.195:44151, user_credentials={real_user=slave}} blocked reactor thread for 46037.8us
W20250624 19:59:00.126883 28504 meta_cache.cc:1261] Time spent looking up entry by key: real 0.078s user 0.000s sys 0.000s
W20250624 19:59:00.969693 28470 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.26.193.193:38569, user_credentials={real_user=slave}} blocked reactor thread for 60320.8us
W20250624 19:59:01.640714 28498 meta_cache.cc:1261] Time spent looking up entry by key: real 0.071s user 0.002s sys 0.000s
W20250624 19:59:01.645474 28505 meta_cache.cc:1261] Time spent looking up entry by key: real 0.081s user 0.004s sys 0.000s
W20250624 19:59:03.291949 28190 debug-util.cc:398] Leaking SignalData structure 0x7b080005b440 after lost signal to thread 28066
W20250624 19:59:03.293171 28190 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5be0 after lost signal to thread 28193
W20250624 19:59:03.741001 28499 meta_cache.cc:1261] Time spent looking up entry by key: real 0.069s user 0.003s sys 0.000s
W20250624 19:59:03.918298 28499 meta_cache.cc:1261] Time spent looking up entry by key: real 0.054s user 0.004s sys 0.000s
W20250624 19:59:04.020869 28503 meta_cache.cc:1261] Time spent looking up entry by key: real 0.096s user 0.000s sys 0.000s
W20250624 19:59:04.177081 28500 meta_cache.cc:1261] Time spent looking up entry by key: real 0.059s user 0.001s sys 0.000s
W20250624 19:59:05.453204 28512 log.cc:927] Time spent T ce5e782ad40747bf92261df05e75bd86 P db7bdfbb301b4d0a99e1a6b61e0b1ad5: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
W20250624 19:59:06.128275 28502 meta_cache.cc:1261] Time spent looking up entry by key: real 0.055s user 0.004s sys 0.000s
W20250624 19:59:08.556861 28505 meta_cache.cc:1261] Time spent looking up entry by key: real 0.064s user 0.003s sys 0.000s
W20250624 19:59:08.565491 28498 meta_cache.cc:1261] Time spent looking up entry by key: real 0.053s user 0.002s sys 0.000s
W20250624 19:59:10.332734 28505 meta_cache.cc:1261] Time spent looking up entry by key: real 0.052s user 0.000s sys 0.000s
W20250624 19:59:11.143926 28498 meta_cache.cc:1261] Time spent looking up entry by key: real 0.057s user 0.003s sys 0.000s
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:2074: Failure
Failed
Bad status: Timed out: Timed out waiting for number of WAL segments on tablet ce5e782ad40747bf92261df05e75bd86 on TS 0 to be 6. Found 5
I20250624 19:59:13.276324 27399 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20250624 19:59:13.276774 27399 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 7d9253b8899d4db8b7cc997c18404393 and pid 28065
************************ BEGIN STACKS **************************
[New LWP 28066]
[New LWP 28067]
[New LWP 28068]
[New LWP 28069]
[New LWP 28070]
[New LWP 28077]
[New LWP 28078]
[New LWP 28079]
[New LWP 28082]
[New LWP 28083]
[New LWP 28084]
[New LWP 28085]
[New LWP 28086]
[New LWP 28087]
[New LWP 28088]
[New LWP 28089]
[New LWP 28090]
[New LWP 28091]
[New LWP 28092]
[New LWP 28093]
[New LWP 28094]
[New LWP 28095]
[New LWP 28096]
[New LWP 28097]
[New LWP 28098]
[New LWP 28099]
[New LWP 28100]
[New LWP 28101]
[New LWP 28102]
[New LWP 28103]
[New LWP 28104]
[New LWP 28105]
[New LWP 28106]
[New LWP 28107]
[New LWP 28108]
[New LWP 28109]
[New LWP 28110]
[New LWP 28111]
[New LWP 28112]
[New LWP 28113]
[New LWP 28114]
[New LWP 28115]
[New LWP 28116]
[New LWP 28117]
[New LWP 28118]
[New LWP 28119]
[New LWP 28120]
[New LWP 28121]
[New LWP 28122]
[New LWP 28123]
[New LWP 28124]
[New LWP 28125]
[New LWP 28126]
[New LWP 28127]
[New LWP 28128]
[New LWP 28129]
[New LWP 28130]
[New LWP 28131]
[New LWP 28132]
[New LWP 28133]
[New LWP 28134]
[New LWP 28135]
[New LWP 28136]
[New LWP 28137]
[New LWP 28138]
[New LWP 28139]
[New LWP 28140]
[New LWP 28141]
[New LWP 28142]
[New LWP 28143]
[New LWP 28144]
[New LWP 28145]
[New LWP 28146]
[New LWP 28147]
[New LWP 28148]
[New LWP 28149]
[New LWP 28150]
[New LWP 28151]
[New LWP 28152]
[New LWP 28153]
[New LWP 28154]
[New LWP 28155]
[New LWP 28156]
[New LWP 28157]
[New LWP 28158]
[New LWP 28159]
[New LWP 28160]
[New LWP 28161]
[New LWP 28162]
[New LWP 28163]
[New LWP 28164]
[New LWP 28165]
[New LWP 28166]
[New LWP 28167]
[New LWP 28168]
[New LWP 28169]
[New LWP 28170]
[New LWP 28171]
[New LWP 28172]
[New LWP 28173]
[New LWP 28174]
[New LWP 28175]
[New LWP 28176]
[New LWP 28177]
[New LWP 28178]
[New LWP 28179]
[New LWP 28180]
[New LWP 28181]
[New LWP 28182]
[New LWP 28183]
[New LWP 28184]
[New LWP 28185]
[New LWP 28186]
[New LWP 28187]
[New LWP 28188]
[New LWP 28189]
[New LWP 28190]
[New LWP 28191]
[New LWP 28192]
[New LWP 28193]
[New LWP 28194]
[New LWP 28195]
[New LWP 28496]
[New LWP 28685]
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
0x00007fd361b78d50 in ?? ()
Id Target Id Frame
* 1 LWP 28065 "kudu" 0x00007fd361b78d50 in ?? ()
2 LWP 28066 "kudu" 0x00007fd35cf3c7a0 in ?? ()
3 LWP 28067 "kudu" 0x00007fd361b74fb9 in ?? ()
4 LWP 28068 "kudu" 0x00007fd361b74fb9 in ?? ()
5 LWP 28069 "kudu" 0x00007fd361b74fb9 in ?? ()
6 LWP 28070 "kernel-watcher-" 0x00007fd361b74fb9 in ?? ()
7 LWP 28077 "ntp client-2807" 0x00007fd361b789e2 in ?? ()
8 LWP 28078 "file cache-evic" 0x00007fd361b74fb9 in ?? ()
9 LWP 28079 "sq_acceptor" 0x00007fd35cf6ccb9 in ?? ()
10 LWP 28082 "rpc reactor-280" 0x00007fd35cf79a47 in ?? ()
11 LWP 28083 "rpc reactor-280" 0x00007fd35cf79a47 in ?? ()
12 LWP 28084 "rpc reactor-280" 0x00007fd35cf79a47 in ?? ()
13 LWP 28085 "rpc reactor-280" 0x00007fd35cf79a47 in ?? ()
14 LWP 28086 "MaintenanceMgr " 0x00007fd361b74ad3 in ?? ()
15 LWP 28087 "txn-status-mana" 0x00007fd361b74fb9 in ?? ()
16 LWP 28088 "collect_and_rem" 0x00007fd361b74fb9 in ?? ()
17 LWP 28089 "tc-session-exp-" 0x00007fd361b74fb9 in ?? ()
18 LWP 28090 "rpc worker-2809" 0x00007fd361b74ad3 in ?? ()
19 LWP 28091 "rpc worker-2809" 0x00007fd361b74ad3 in ?? ()
20 LWP 28092 "rpc worker-2809" 0x00007fd361b74ad3 in ?? ()
21 LWP 28093 "rpc worker-2809" 0x00007fd361b74ad3 in ?? ()
22 LWP 28094 "rpc worker-2809" 0x00007fd361b74ad3 in ?? ()
23 LWP 28095 "rpc worker-2809" 0x00007fd361b74ad3 in ?? ()
24 LWP 28096 "rpc worker-2809" 0x00007fd361b74ad3 in ?? ()
25 LWP 28097 "rpc worker-2809" 0x00007fd361b74ad3 in ?? ()
26 LWP 28098 "rpc worker-2809" 0x00007fd361b74ad3 in ?? ()
27 LWP 28099 "rpc worker-2809" 0x00007fd361b74ad3 in ?? ()
28 LWP 28100 "rpc worker-2810" 0x00007fd361b74ad3 in ?? ()
29 LWP 28101 "rpc worker-2810" 0x00007fd361b74ad3 in ?? ()
30 LWP 28102 "rpc worker-2810" 0x00007fd361b74ad3 in ?? ()
31 LWP 28103 "rpc worker-2810" 0x00007fd361b74ad3 in ?? ()
32 LWP 28104 "rpc worker-2810" 0x00007fd361b74ad3 in ?? ()
33 LWP 28105 "rpc worker-2810" 0x00007fd361b74ad3 in ?? ()
34 LWP 28106 "rpc worker-2810" 0x00007fd361b74ad3 in ?? ()
35 LWP 28107 "rpc worker-2810" 0x00007fd361b74ad3 in ?? ()
36 LWP 28108 "rpc worker-2810" 0x00007fd361b74ad3 in ?? ()
37 LWP 28109 "rpc worker-2810" 0x00007fd361b74ad3 in ?? ()
38 LWP 28110 "rpc worker-2811" 0x00007fd361b74ad3 in ?? ()
39 LWP 28111 "rpc worker-2811" 0x00007fd361b74ad3 in ?? ()
40 LWP 28112 "rpc worker-2811" 0x00007fd361b74ad3 in ?? ()
41 LWP 28113 "rpc worker-2811" 0x00007fd361b74ad3 in ?? ()
42 LWP 28114 "rpc worker-2811" 0x00007fd361b74ad3 in ?? ()
43 LWP 28115 "rpc worker-2811" 0x00007fd361b74ad3 in ?? ()
44 LWP 28116 "rpc worker-2811" 0x00007fd361b74ad3 in ?? ()
45 LWP 28117 "rpc worker-2811" 0x00007fd361b74ad3 in ?? ()
46 LWP 28118 "rpc worker-2811" 0x00007fd361b74ad3 in ?? ()
47 LWP 28119 "rpc worker-2811" 0x00007fd361b74ad3 in ?? ()
48 LWP 28120 "rpc worker-2812" 0x00007fd361b74ad3 in ?? ()
49 LWP 28121 "rpc worker-2812" 0x00007fd361b74ad3 in ?? ()
50 LWP 28122 "rpc worker-2812" 0x00007fd361b74ad3 in ?? ()
51 LWP 28123 "rpc worker-2812" 0x00007fd361b74ad3 in ?? ()
52 LWP 28124 "rpc worker-2812" 0x00007fd361b74ad3 in ?? ()
53 LWP 28125 "rpc worker-2812" 0x00007fd361b74ad3 in ?? ()
54 LWP 28126 "rpc worker-2812" 0x00007fd361b74ad3 in ?? ()
55 LWP 28127 "rpc worker-2812" 0x00007fd361b74ad3 in ?? ()
56 LWP 28128 "rpc worker-2812" 0x00007fd361b74ad3 in ?? ()
57 LWP 28129 "rpc worker-2812" 0x00007fd361b74ad3 in ?? ()
58 LWP 28130 "rpc worker-2813" 0x00007fd361b74ad3 in ?? ()
59 LWP 28131 "rpc worker-2813" 0x00007fd361b74ad3 in ?? ()
60 LWP 28132 "rpc worker-2813" 0x00007fd361b74ad3 in ?? ()
61 LWP 28133 "rpc worker-2813" 0x00007fd361b74ad3 in ?? ()
62 LWP 28134 "rpc worker-2813" 0x00007fd361b74ad3 in ?? ()
63 LWP 28135 "rpc worker-2813" 0x00007fd361b74ad3 in ?? ()
64 LWP 28136 "rpc worker-2813" 0x00007fd361b74ad3 in ?? ()
65 LWP 28137 "rpc worker-2813" 0x00007fd361b74ad3 in ?? ()
66 LWP 28138 "rpc worker-2813" 0x00007fd361b74ad3 in ?? ()
67 LWP 28139 "rpc worker-2813" 0x00007fd361b74ad3 in ?? ()
68 LWP 28140 "rpc worker-2814" 0x00007fd361b74ad3 in ?? ()
69 LWP 28141 "rpc worker-2814" 0x00007fd361b74ad3 in ?? ()
70 LWP 28142 "rpc worker-2814" 0x00007fd361b74ad3 in ?? ()
71 LWP 28143 "rpc worker-2814" 0x00007fd361b74ad3 in ?? ()
72 LWP 28144 "rpc worker-2814" 0x00007fd361b74ad3 in ?? ()
73 LWP 28145 "rpc worker-2814" 0x00007fd361b74ad3 in ?? ()
74 LWP 28146 "rpc worker-2814" 0x00007fd361b74ad3 in ?? ()
75 LWP 28147 "rpc worker-2814" 0x00007fd361b74ad3 in ?? ()
76 LWP 28148 "rpc worker-2814" 0x00007fd361b74ad3 in ?? ()
77 LWP 28149 "rpc worker-2814" 0x00007fd361b74ad3 in ?? ()
78 LWP 28150 "rpc worker-2815" 0x00007fd361b74ad3 in ?? ()
79 LWP 28151 "rpc worker-2815" 0x00007fd361b74ad3 in ?? ()
80 LWP 28152 "rpc worker-2815" 0x00007fd361b74ad3 in ?? ()
81 LWP 28153 "rpc worker-2815" 0x00007fd361b74ad3 in ?? ()
82 LWP 28154 "rpc worker-2815" 0x00007fd361b74ad3 in ?? ()
83 LWP 28155 "rpc worker-2815" 0x00007fd361b74ad3 in ?? ()
84 LWP 28156 "rpc worker-2815" 0x00007fd361b74ad3 in ?? ()
85 LWP 28157 "rpc worker-2815" 0x00007fd361b74ad3 in ?? ()
86 LWP 28158 "rpc worker-2815" 0x00007fd361b74ad3 in ?? ()
87 LWP 28159 "rpc worker-2815" 0x00007fd361b74ad3 in ?? ()
88 LWP 28160 "rpc worker-2816" 0x00007fd361b74ad3 in ?? ()
89 LWP 28161 "rpc worker-2816" 0x00007fd361b74ad3 in ?? ()
90 LWP 28162 "rpc worker-2816" 0x00007fd361b74ad3 in ?? ()
91 LWP 28163 "rpc worker-2816" 0x00007fd361b74ad3 in ?? ()
92 LWP 28164 "rpc worker-2816" 0x00007fd361b74ad3 in ?? ()
93 LWP 28165 "rpc worker-2816" 0x00007fd361b74ad3 in ?? ()
94 LWP 28166 "rpc worker-2816" 0x00007fd361b74ad3 in ?? ()
95 LWP 28167 "rpc worker-2816" 0x00007fd361b74ad3 in ?? ()
96 LWP 28168 "rpc worker-2816" 0x00007fd361b74ad3 in ?? ()
97 LWP 28169 "rpc worker-2816" 0x00007fd361b74ad3 in ?? ()
98 LWP 28170 "rpc worker-2817" 0x00007fd361b74ad3 in ?? ()
99 LWP 28171 "rpc worker-2817" 0x00007fd361b74ad3 in ?? ()
100 LWP 28172 "rpc worker-2817" 0x00007fd361b74ad3 in ?? ()
101 LWP 28173 "rpc worker-2817" 0x00007fd361b74ad3 in ?? ()
102 LWP 28174 "rpc worker-2817" 0x00007fd361b74ad3 in ?? ()
103 LWP 28175 "rpc worker-2817" 0x00007fd361b74ad3 in ?? ()
104 LWP 28176 "rpc worker-2817" 0x00007fd361b74ad3 in ?? ()
105 LWP 28177 "rpc worker-2817" 0x00007fd361b74ad3 in ?? ()
106 LWP 28178 "rpc worker-2817" 0x00007fd361b74ad3 in ?? ()
107 LWP 28179 "rpc worker-2817" 0x00007fd361b74ad3 in ?? ()
108 LWP 28180 "rpc worker-2818" 0x00007fd361b74ad3 in ?? ()
109 LWP 28181 "rpc worker-2818" 0x00007fd361b74ad3 in ?? ()
110 LWP 28182 "rpc worker-2818" 0x00007fd361b74ad3 in ?? ()
111 LWP 28183 "rpc worker-2818" 0x00007fd361b74ad3 in ?? ()
112 LWP 28184 "rpc worker-2818" 0x00007fd361b74ad3 in ?? ()
113 LWP 28185 "rpc worker-2818" 0x00007fd361b74ad3 in ?? ()
114 LWP 28186 "rpc worker-2818" 0x00007fd361b74ad3 in ?? ()
115 LWP 28187 "rpc worker-2818" 0x00007fd361b74ad3 in ?? ()
116 LWP 28188 "rpc worker-2818" 0x00007fd361b74ad3 in ?? ()
117 LWP 28189 "rpc worker-2818" 0x00007fd361b74ad3 in ?? ()
118 LWP 28190 "diag-logger-281" 0x00007fd361b74fb9 in ?? ()
119 LWP 28191 "result-tracker-" 0x00007fd361b74fb9 in ?? ()
120 LWP 28192 "excess-log-dele" 0x00007fd361b74fb9 in ?? ()
121 LWP 28193 "acceptor-28193" 0x00007fd35cf7b0c7 in ?? ()
122 LWP 28194 "heartbeat-28194" 0x00007fd361b74fb9 in ?? ()
123 LWP 28195 "maintenance_sch" 0x00007fd361b74fb9 in ?? ()
124 LWP 28496 "wal-append [wor" 0x00007fd361b74fb9 in ?? ()
125 LWP 28685 "raft [worker]-2" 0x00007fd361b74fb9 in ?? ()
Thread 125 (LWP 28685):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 124 (LWP 28496):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x00007b10000563f0 in ?? ()
#2 0x0000000000000f58 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b6400060018 in ?? ()
#5 0x00007fd3123bc440 in ?? ()
#6 0x0000000000001eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 123 (LWP 28195):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x00007b0100000000 in ?? ()
#2 0x0000000000000103 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007fd315db96c0 in ?? ()
#6 0x0000000000000206 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 28194):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 28193):
#0 0x00007fd35cf7b0c7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 28192):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x00007fd3175bc940 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffe5d824560 in ?? ()
#5 0x00007fd3175bc7b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 28191):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x0000000085352f90 in ?? ()
#2 0x0000000000000041 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001008 in ?? ()
#5 0x00007fd317dbd800 in ?? ()
#6 0x0000000000000082 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 28190):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x00007fd35ae88008 in ?? ()
#2 0x0000000000000039 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000c90 in ?? ()
#5 0x00007fd3185be750 in ?? ()
#6 0x0000000000000072 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 28189):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 28188):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 28187):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 28186):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 28185):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 28184):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 28183):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 28182):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 28181):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 28180):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 28179):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 28178):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 28177):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 28176):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 28175):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 28174):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 28173):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 28172):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 28171):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 28170):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 28169):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 28168):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 28167):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 28166):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 28165):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 28164):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 28163):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 28162):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 28161):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 28160):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 28159):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 28158):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 28157):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 28156):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 28155):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 28154):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 28153):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 28152):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 28151):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 28150):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 28149):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24001147cc in ?? ()
#4 0x00007fd32d7ba710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd32d7ba730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x0000000000000000 in ?? ()
Thread 76 (LWP 28148):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 75 (LWP 28147):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 28146):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 28145):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 28144):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 28143):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 28142):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 28141):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 28140):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 28139):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 28138):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 28137):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 28136):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 28135):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 28134):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 28133):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 28132):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 28131):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 28130):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 28129):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000b902c in ?? ()
#4 0x00007fd337bbc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd337bbc730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x007f0400000026c3 in ?? ()
#9 0x00007fd361b74770 in ?? ()
#10 0x00007fd337bbc730 in ?? ()
#11 0x0002008300000dfe in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 28128):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 28127):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 28126):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 28125):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 28124):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 28123):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 28122):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 28121):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 28120):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 28119):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 28118):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 28117):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 28116):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 28115):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 28114):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 28113):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 28112):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 28111):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 28110):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 28109):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000068 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240005ffe8 in ?? ()
#4 0x00007fd341fbe710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd341fbe730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 36 (LWP 28108):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000259 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240005d7fc in ?? ()
#4 0x00007fd3429b6710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd3429b6730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007fd361b74770 in ?? ()
#10 0x00007fd3429b6730 in ?? ()
#11 0x00007fd313c276b0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 28107):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x00000000000001f3 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b2400058ffc in ?? ()
#4 0x00007fd3431b7710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd3431b7730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007fd361b74770 in ?? ()
#10 0x00007fd3431b7730 in ?? ()
#11 0x00007fd30e458cb0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 34 (LWP 28106):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x00000000000002da in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b24000547f8 in ?? ()
#4 0x00007fd3439b8710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd3439b8730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 33 (LWP 28105):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000030 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240004fff8 in ?? ()
#4 0x00007fd3441b9710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd3441b9730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 32 (LWP 28104):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x000000000000013b in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240004900c in ?? ()
#4 0x00007fd3449ba710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd3449ba730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007fd361b74770 in ?? ()
#10 0x00007fd3449ba730 in ?? ()
#11 0x00007fd313aeccb0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 31 (LWP 28103):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 28102):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 28101):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 28100):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 28099):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 28098):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 28097):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 28096):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 28095):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 28094):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 28093):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 28092):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 28091):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 28090):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 28089):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x0000000017a335c8 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800003a00 in ?? ()
#5 0x00007fd34c58e700 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 28088):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x00007fd34cd8f9a8 in ?? ()
#2 0x000000000000000d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000372d8 in ?? ()
#5 0x00007fd34cd8f840 in ?? ()
#6 0x000000000000001a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 28087):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007fd34d590410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 28086):
#0 0x00007fd361b74ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 28085):
#0 0x00007fd35cf79a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 28084):
#0 0x00007fd35cf79a47 in ?? ()
#1 0x00007b2800025088 in ?? ()
#2 0x0044e00000083cf1 in ?? ()
#3 0x00007fd34ed93500 in ?? ()
#4 0x00007fd34ed94b80 in ?? ()
#5 0x00007fd34ed93500 in ?? ()
#6 0x0000000000000011 in ?? ()
#7 0x00007b5800001800 in ?? ()
#8 0x0000000000488695 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007fd35a6ee000 in ?? ()
#10 0x0000000000488599 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007fd34ed94b80 in ?? ()
#12 0x00007fd35f9d2069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007fd3651531a0 in ?? ()
#15 0x00007b4c00002f90 in ?? ()
#16 0x00007b4c00002f98 in ?? ()
#17 0x00007fd34ed937a0 in ?? ()
#18 0x00007b4400036500 in ?? ()
#19 0x00007fd34ed93cd0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 11 (LWP 28083):
#0 0x00007fd35cf79a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 10 (LWP 28082):
#0 0x00007fd35cf79a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 9 (LWP 28079):
#0 0x00007fd35cf6ccb9 in ?? ()
#1 0x00007fd355dbcc10 in ?? ()
#2 0x00007b0400009010 in ?? ()
#3 0x00007fd355dbdb80 in ?? ()
#4 0x00007fd355dbcc10 in ?? ()
#5 0x00007b0400009010 in ?? ()
#6 0x00000000004888a3 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007fd35a8b0000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007fd355dbdb80 in ?? ()
#10 0x00007fd366951908 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 8 (LWP 28078):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x0000600000000000 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400034018 in ?? ()
#5 0x00007fd3555bb7f0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 28077):
#0 0x00007fd361b789e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 28070):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x00007fd356dbea40 in ?? ()
#2 0x000000000000014b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400035b98 in ?? ()
#5 0x00007fd356dbe5d0 in ?? ()
#6 0x0000000000000296 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 28069):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 28068):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 28067):
#0 0x00007fd361b74fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 28066):
#0 0x00007fd35cf3c7a0 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 28065):
#0 0x00007fd361b78d50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x0000000000467b2b in __sanitizer::internal_alloc_placeholder ()
#3 0x00007fd35c19acc0 in ?? ()
#4 0x00007fd35c19acc0 in ?? ()
#5 0x00007ffe5d824370 in ?? ()
#6 0x000000000048aef4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000a9d3c9 in ?? ()
#9 0x00007fd35c19acc0 in ?? ()
#10 0x00007fd3600a0f0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250624 19:59:14.299085 27399 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID db7bdfbb301b4d0a99e1a6b61e0b1ad5 and pid 28198
************************ BEGIN STACKS **************************
[New LWP 28199]
[New LWP 28200]
[New LWP 28201]
[New LWP 28202]
[New LWP 28203]
[New LWP 28210]
[New LWP 28211]
[New LWP 28212]
[New LWP 28215]
[New LWP 28216]
[New LWP 28217]
[New LWP 28218]
[New LWP 28219]
[New LWP 28220]
[New LWP 28221]
[New LWP 28222]
[New LWP 28223]
[New LWP 28224]
[New LWP 28225]
[New LWP 28226]
[New LWP 28227]
[New LWP 28228]
[New LWP 28229]
[New LWP 28230]
[New LWP 28231]
[New LWP 28232]
[New LWP 28233]
[New LWP 28234]
[New LWP 28235]
[New LWP 28236]
[New LWP 28237]
[New LWP 28238]
[New LWP 28239]
[New LWP 28240]
[New LWP 28241]
[New LWP 28242]
[New LWP 28243]
[New LWP 28244]
[New LWP 28245]
[New LWP 28246]
[New LWP 28247]
[New LWP 28248]
[New LWP 28249]
[New LWP 28250]
[New LWP 28251]
[New LWP 28252]
[New LWP 28253]
[New LWP 28254]
[New LWP 28255]
[New LWP 28256]
[New LWP 28257]
[New LWP 28258]
[New LWP 28259]
[New LWP 28260]
[New LWP 28261]
[New LWP 28262]
[New LWP 28263]
[New LWP 28264]
[New LWP 28265]
[New LWP 28266]
[New LWP 28267]
[New LWP 28268]
[New LWP 28269]
[New LWP 28270]
[New LWP 28271]
[New LWP 28272]
[New LWP 28273]
[New LWP 28274]
[New LWP 28275]
[New LWP 28276]
[New LWP 28277]
[New LWP 28278]
[New LWP 28279]
[New LWP 28280]
[New LWP 28281]
[New LWP 28282]
[New LWP 28283]
[New LWP 28284]
[New LWP 28285]
[New LWP 28286]
[New LWP 28287]
[New LWP 28288]
[New LWP 28289]
[New LWP 28290]
[New LWP 28291]
[New LWP 28292]
[New LWP 28293]
[New LWP 28294]
[New LWP 28295]
[New LWP 28296]
[New LWP 28297]
[New LWP 28298]
[New LWP 28299]
[New LWP 28300]
[New LWP 28301]
[New LWP 28302]
[New LWP 28303]
[New LWP 28304]
[New LWP 28305]
[New LWP 28306]
[New LWP 28307]
[New LWP 28308]
[New LWP 28309]
[New LWP 28310]
[New LWP 28311]
[New LWP 28312]
[New LWP 28313]
[New LWP 28314]
[New LWP 28315]
[New LWP 28316]
[New LWP 28317]
[New LWP 28318]
[New LWP 28319]
[New LWP 28320]
[New LWP 28321]
[New LWP 28322]
[New LWP 28323]
[New LWP 28324]
[New LWP 28325]
[New LWP 28326]
[New LWP 28327]
[New LWP 28328]
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
0x00007f8082becd50 in ?? ()
Id Target Id Frame
* 1 LWP 28198 "kudu" 0x00007f8082becd50 in ?? ()
2 LWP 28199 "kudu" 0x00007f807dfb07a0 in ?? ()
3 LWP 28200 "kudu" 0x00007f8082be8fb9 in ?? ()
4 LWP 28201 "kudu" 0x00007f8082be8fb9 in ?? ()
5 LWP 28202 "kudu" 0x00007f8082be8fb9 in ?? ()
6 LWP 28203 "kernel-watcher-" 0x00007f8082be8fb9 in ?? ()
7 LWP 28210 "ntp client-2821" 0x00007f8082bec9e2 in ?? ()
8 LWP 28211 "file cache-evic" 0x00007f8082be8fb9 in ?? ()
9 LWP 28212 "sq_acceptor" 0x00007f807dfe0cb9 in ?? ()
10 LWP 28215 "rpc reactor-282" 0x00007f807dfeda47 in ?? ()
11 LWP 28216 "rpc reactor-282" 0x00007f807dfeda47 in ?? ()
12 LWP 28217 "rpc reactor-282" 0x00007f807dfeda47 in ?? ()
13 LWP 28218 "rpc reactor-282" 0x00007f807dfeda47 in ?? ()
14 LWP 28219 "MaintenanceMgr " 0x00007f8082be8ad3 in ?? ()
15 LWP 28220 "txn-status-mana" 0x00007f8082be8fb9 in ?? ()
16 LWP 28221 "collect_and_rem" 0x00007f8082be8fb9 in ?? ()
17 LWP 28222 "tc-session-exp-" 0x00007f8082be8fb9 in ?? ()
18 LWP 28223 "rpc worker-2822" 0x00007f8082be8ad3 in ?? ()
19 LWP 28224 "rpc worker-2822" 0x00007f8082be8ad3 in ?? ()
20 LWP 28225 "rpc worker-2822" 0x00007f8082be8ad3 in ?? ()
21 LWP 28226 "rpc worker-2822" 0x00007f8082be8ad3 in ?? ()
22 LWP 28227 "rpc worker-2822" 0x00007f8082be8ad3 in ?? ()
23 LWP 28228 "rpc worker-2822" 0x00007f8082be8ad3 in ?? ()
24 LWP 28229 "rpc worker-2822" 0x00007f8082be8ad3 in ?? ()
25 LWP 28230 "rpc worker-2823" 0x00007f8082be8ad3 in ?? ()
26 LWP 28231 "rpc worker-2823" 0x00007f8082be8ad3 in ?? ()
27 LWP 28232 "rpc worker-2823" 0x00007f8082be8ad3 in ?? ()
28 LWP 28233 "rpc worker-2823" 0x00007f8082be8ad3 in ?? ()
29 LWP 28234 "rpc worker-2823" 0x00007f8082be8ad3 in ?? ()
30 LWP 28235 "rpc worker-2823" 0x00007f8082be8ad3 in ?? ()
31 LWP 28236 "rpc worker-2823" 0x00007f8082be8ad3 in ?? ()
32 LWP 28237 "rpc worker-2823" 0x00007f8082be8ad3 in ?? ()
33 LWP 28238 "rpc worker-2823" 0x00007f8082be8ad3 in ?? ()
34 LWP 28239 "rpc worker-2823" 0x00007f8082be8ad3 in ?? ()
35 LWP 28240 "rpc worker-2824" 0x00007f8082be8ad3 in ?? ()
36 LWP 28241 "rpc worker-2824" 0x00007f8082be8ad3 in ?? ()
37 LWP 28242 "rpc worker-2824" 0x00007f8082be8ad3 in ?? ()
38 LWP 28243 "rpc worker-2824" 0x00007f8082be8ad3 in ?? ()
39 LWP 28244 "rpc worker-2824" 0x00007f8082be8ad3 in ?? ()
40 LWP 28245 "rpc worker-2824" 0x00007f8082be8ad3 in ?? ()
41 LWP 28246 "rpc worker-2824" 0x00007f8082be8ad3 in ?? ()
42 LWP 28247 "rpc worker-2824" 0x00007f8082be8ad3 in ?? ()
43 LWP 28248 "rpc worker-2824" 0x00007f8082be8ad3 in ?? ()
44 LWP 28249 "rpc worker-2824" 0x00007f8082be8ad3 in ?? ()
45 LWP 28250 "rpc worker-2825" 0x00007f8082be8ad3 in ?? ()
46 LWP 28251 "rpc worker-2825" 0x00007f8082be8ad3 in ?? ()
47 LWP 28252 "rpc worker-2825" 0x00007f8082be8ad3 in ?? ()
48 LWP 28253 "rpc worker-2825" 0x00007f8082be8ad3 in ?? ()
49 LWP 28254 "rpc worker-2825" 0x00007f8082be8ad3 in ?? ()
50 LWP 28255 "rpc worker-2825" 0x00007f8082be8ad3 in ?? ()
51 LWP 28256 "rpc worker-2825" 0x00007f8082be8ad3 in ?? ()
52 LWP 28257 "rpc worker-2825" 0x00007f8082be8ad3 in ?? ()
53 LWP 28258 "rpc worker-2825" 0x00007f8082be8ad3 in ?? ()
54 LWP 28259 "rpc worker-2825" 0x00007f8082be8ad3 in ?? ()
55 LWP 28260 "rpc worker-2826" 0x00007f8082be8ad3 in ?? ()
56 LWP 28261 "rpc worker-2826" 0x00007f8082be8ad3 in ?? ()
57 LWP 28262 "rpc worker-2826" 0x00007f8082be8ad3 in ?? ()
58 LWP 28263 "rpc worker-2826" 0x00007f8082be8ad3 in ?? ()
59 LWP 28264 "rpc worker-2826" 0x00007f8082be8ad3 in ?? ()
60 LWP 28265 "rpc worker-2826" 0x00007f8082be8ad3 in ?? ()
61 LWP 28266 "rpc worker-2826" 0x00007f8082be8ad3 in ?? ()
62 LWP 28267 "rpc worker-2826" 0x00007f8082be8ad3 in ?? ()
63 LWP 28268 "rpc worker-2826" 0x00007f8082be8ad3 in ?? ()
64 LWP 28269 "rpc worker-2826" 0x00007f8082be8ad3 in ?? ()
65 LWP 28270 "rpc worker-2827" 0x00007f8082be8ad3 in ?? ()
66 LWP 28271 "rpc worker-2827" 0x00007f8082be8ad3 in ?? ()
67 LWP 28272 "rpc worker-2827" 0x00007f8082be8ad3 in ?? ()
68 LWP 28273 "rpc worker-2827" 0x00007f8082be8ad3 in ?? ()
69 LWP 28274 "rpc worker-2827" 0x00007f8082be8ad3 in ?? ()
70 LWP 28275 "rpc worker-2827" 0x00007f8082be8ad3 in ?? ()
71 LWP 28276 "rpc worker-2827" 0x00007f8082be8ad3 in ?? ()
72 LWP 28277 "rpc worker-2827" 0x00007f8082be8ad3 in ?? ()
73 LWP 28278 "rpc worker-2827" 0x00007f8082be8ad3 in ?? ()
74 LWP 28279 "rpc worker-2827" 0x00007f8082be8ad3 in ?? ()
75 LWP 28280 "rpc worker-2828" 0x00007f8082be8ad3 in ?? ()
76 LWP 28281 "rpc worker-2828" 0x00007f8082be8ad3 in ?? ()
77 LWP 28282 "rpc worker-2828" 0x00007f8082be8ad3 in ?? ()
78 LWP 28283 "rpc worker-2828" 0x00007f8082be8ad3 in ?? ()
79 LWP 28284 "rpc worker-2828" 0x00007f8082be8ad3 in ?? ()
80 LWP 28285 "rpc worker-2828" 0x00007f8082be8ad3 in ?? ()
81 LWP 28286 "rpc worker-2828" 0x00007f8082be8ad3 in ?? ()
82 LWP 28287 "rpc worker-2828" 0x00007f8082be8ad3 in ?? ()
83 LWP 28288 "rpc worker-2828" 0x00007f8082be8ad3 in ?? ()
84 LWP 28289 "rpc worker-2828" 0x00007f8082be8ad3 in ?? ()
85 LWP 28290 "rpc worker-2829" 0x00007f8082be8ad3 in ?? ()
86 LWP 28291 "rpc worker-2829" 0x00007f8082be8ad3 in ?? ()
87 LWP 28292 "rpc worker-2829" 0x00007f8082be8ad3 in ?? ()
88 LWP 28293 "rpc worker-2829" 0x00007f8082be8ad3 in ?? ()
89 LWP 28294 "rpc worker-2829" 0x00007f8082be8ad3 in ?? ()
90 LWP 28295 "rpc worker-2829" 0x00007f8082be8ad3 in ?? ()
91 LWP 28296 "rpc worker-2829" 0x00007f8082be8ad3 in ?? ()
92 LWP 28297 "rpc worker-2829" 0x00007f8082be8ad3 in ?? ()
93 LWP 28298 "rpc worker-2829" 0x00007f8082be8ad3 in ?? ()
94 LWP 28299 "rpc worker-2829" 0x00007f8082be8ad3 in ?? ()
95 LWP 28300 "rpc worker-2830" 0x00007f8082be8ad3 in ?? ()
96 LWP 28301 "rpc worker-2830" 0x00007f8082be8ad3 in ?? ()
97 LWP 28302 "rpc worker-2830" 0x00007f8082be8ad3 in ?? ()
98 LWP 28303 "rpc worker-2830" 0x00007f8082be8ad3 in ?? ()
99 LWP 28304 "rpc worker-2830" 0x00007f8082be8ad3 in ?? ()
100 LWP 28305 "rpc worker-2830" 0x00007f8082be8ad3 in ?? ()
101 LWP 28306 "rpc worker-2830" 0x00007f8082be8ad3 in ?? ()
102 LWP 28307 "rpc worker-2830" 0x00007f8082be8ad3 in ?? ()
103 LWP 28308 "rpc worker-2830" 0x00007f8082be8ad3 in ?? ()
104 LWP 28309 "rpc worker-2830" 0x00007f8082be8ad3 in ?? ()
105 LWP 28310 "rpc worker-2831" 0x00007f8082be8ad3 in ?? ()
106 LWP 28311 "rpc worker-2831" 0x00007f8082be8ad3 in ?? ()
107 LWP 28312 "rpc worker-2831" 0x00007f8082be8ad3 in ?? ()
108 LWP 28313 "rpc worker-2831" 0x00007f8082be8ad3 in ?? ()
109 LWP 28314 "rpc worker-2831" 0x00007f8082be8ad3 in ?? ()
110 LWP 28315 "rpc worker-2831" 0x00007f8082be8ad3 in ?? ()
111 LWP 28316 "rpc worker-2831" 0x00007f8082be8ad3 in ?? ()
112 LWP 28317 "rpc worker-2831" 0x00007f8082be8ad3 in ?? ()
113 LWP 28318 "rpc worker-2831" 0x00007f8082be8ad3 in ?? ()
114 LWP 28319 "rpc worker-2831" 0x00007f8082be8ad3 in ?? ()
115 LWP 28320 "rpc worker-2832" 0x00007f8082be8ad3 in ?? ()
116 LWP 28321 "rpc worker-2832" 0x00007f8082be8ad3 in ?? ()
117 LWP 28322 "rpc worker-2832" 0x00007f8082be8ad3 in ?? ()
118 LWP 28323 "diag-logger-283" 0x00007f8082be8fb9 in ?? ()
119 LWP 28324 "result-tracker-" 0x00007f8082be8fb9 in ?? ()
120 LWP 28325 "excess-log-dele" 0x00007f8082be8fb9 in ?? ()
121 LWP 28326 "acceptor-28326" 0x00007f807dfef0c7 in ?? ()
122 LWP 28327 "heartbeat-28327" 0x00007f8082be8fb9 in ?? ()
123 LWP 28328 "maintenance_sch" 0x00007f8082be8fb9 in ?? ()
Thread 123 (LWP 28328):
#0 0x00007f8082be8fb9 in ?? ()
#1 0x00007b0100000000 in ?? ()
#2 0x0000000000000100 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007f8036db96c0 in ?? ()
#6 0x0000000000000200 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 28327):
#0 0x00007f8082be8fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 28326):
#0 0x00007f807dfef0c7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 28325):
#0 0x00007f8082be8fb9 in ?? ()
#1 0x00007f80385bc940 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fffcd25b230 in ?? ()
#5 0x00007f80385bc7b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 28324):
#0 0x00007f8082be8fb9 in ?? ()
#1 0x0000000085352f90 in ?? ()
#2 0x0000000000000040 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001008 in ?? ()
#5 0x00007f8038dbd800 in ?? ()
#6 0x0000000000000080 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 28323):
#0 0x00007f8082be8fb9 in ?? ()
#1 0x00007f807c036008 in ?? ()
#2 0x000000000000003a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000c90 in ?? ()
#5 0x00007f80395be750 in ?? ()
#6 0x0000000000000074 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 28322):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 28321):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 28320):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 28319):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 28318):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 28317):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 28316):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 28315):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 28314):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 28313):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 28312):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 28311):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 28310):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 28309):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 28308):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 28307):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 28306):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 28305):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 28304):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 28303):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 28302):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 28301):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 28300):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 28299):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 28298):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 28297):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 28296):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 28295):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 28294):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 28293):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 28292):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 28291):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 28290):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 28289):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 28288):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 28287):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 28286):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 28285):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 28284):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 28283):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 28282):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000356 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b24001147c8 in ?? ()
#4 0x00007f804e7ba710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f804e7ba730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 76 (LWP 28281):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x00000000000002b8 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240010ffc8 in ?? ()
#4 0x00007f804efbb710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f804efbb730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 28280):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x000000000000022e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240010d7d8 in ?? ()
#4 0x00007f804f7bc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f804f7bc730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 28279):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 28278):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 28277):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 28276):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 28275):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 28274):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 28273):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 28272):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 28271):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 28270):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 28269):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 28268):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 28267):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 28266):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 28265):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 28264):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 28263):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 28262):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000b902c in ?? ()
#4 0x00007f8058bbc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f8058bbc730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f8082be8770 in ?? ()
#10 0x00007f8058bbc730 in ?? ()
#11 0x00007f807ad89c18 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 28261):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 28260):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 28259):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 28258):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 28257):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 28256):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 28255):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 28254):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 28253):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 28252):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 28251):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 28250):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 28249):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 28248):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 28247):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 28246):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 28245):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 28244):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 28243):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 28242):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240005ffec in ?? ()
#4 0x00007f8062fbe710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f8062fbe730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f8082be8770 in ?? ()
#10 0x00007f8062fbe730 in ?? ()
#11 0x00007f807b149c30 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 36 (LWP 28241):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 28240):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 28239):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 28238):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 28237):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 28236):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 28235):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 28234):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 28233):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 28232):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 28231):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 28230):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 28229):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 28228):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 28227):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 28226):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 28225):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 28224):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 28223):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 28222):
#0 0x00007f8082be8fb9 in ?? ()
#1 0x0000000017a335c8 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800003a00 in ?? ()
#5 0x00007f806d58e700 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 28221):
#0 0x00007f8082be8fb9 in ?? ()
#1 0x00007f806dd8f9a8 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000372d8 in ?? ()
#5 0x00007f806dd8f840 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 28220):
#0 0x00007f8082be8fb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007f806e590410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 28219):
#0 0x00007f8082be8ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 28218):
#0 0x00007f807dfeda47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 28217):
#0 0x00007f807dfeda47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 11 (LWP 28216):
#0 0x00007f807dfeda47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 10 (LWP 28215):
#0 0x00007f807dfeda47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 9 (LWP 28212):
#0 0x00007f807dfe0cb9 in ?? ()
#1 0x00007f8076dbcc10 in ?? ()
#2 0x00007b0400009510 in ?? ()
#3 0x00007f8076dbdb80 in ?? ()
#4 0x00007f8076dbcc10 in ?? ()
#5 0x00007b0400009510 in ?? ()
#6 0x00000000004888a3 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007f807ba16000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007f8076dbdb80 in ?? ()
#10 0x00007f80879c5908 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 8 (LWP 28211):
#0 0x00007f8082be8fb9 in ?? ()
#1 0x0000600000000000 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400034018 in ?? ()
#5 0x00007f80765bb7f0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 28210):
#0 0x00007f8082bec9e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 28203):
#0 0x00007f8082be8fb9 in ?? ()
#1 0x00007f8077dbea40 in ?? ()
#2 0x0000000000000146 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400035b98 in ?? ()
#5 0x00007f8077dbe5d0 in ?? ()
#6 0x000000000000028c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 28202):
#0 0x00007f8082be8fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 28201):
#0 0x00007f8082be8fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 28200):
#0 0x00007f8082be8fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 28199):
#0 0x00007f807dfb07a0 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 28198):
#0 0x00007f8082becd50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x0000000000467b2b in __sanitizer::internal_alloc_placeholder ()
#3 0x00007f807d20ecc0 in ?? ()
#4 0x00007f807d20ecc0 in ?? ()
#5 0x00007fffcd25b040 in ?? ()
#6 0x000000000048aef4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000a98ea9 in ?? ()
#9 0x00007f807d20ecc0 in ?? ()
#10 0x00007f8081114f0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250624 19:59:15.298318 27399 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID 07bfec0a5d354cf58ddb356ca8b26e46 and pid 28331
************************ BEGIN STACKS **************************
W20250624 19:59:16.080250 28037 debug-util.cc:398] Leaking SignalData structure 0x7b08000a2620 after lost signal to thread 27974
W20250624 19:59:16.080935 28037 debug-util.cc:398] Leaking SignalData structure 0x7b08000a8380 after lost signal to thread 28040
[New LWP 28332]
[New LWP 28333]
[New LWP 28334]
[New LWP 28335]
[New LWP 28336]
[New LWP 28343]
[New LWP 28344]
[New LWP 28345]
[New LWP 28348]
[New LWP 28349]
[New LWP 28350]
[New LWP 28351]
[New LWP 28352]
[New LWP 28353]
[New LWP 28354]
[New LWP 28355]
[New LWP 28356]
[New LWP 28357]
[New LWP 28358]
[New LWP 28359]
[New LWP 28360]
[New LWP 28361]
[New LWP 28362]
[New LWP 28363]
[New LWP 28364]
[New LWP 28365]
[New LWP 28366]
[New LWP 28367]
[New LWP 28368]
[New LWP 28369]
[New LWP 28370]
[New LWP 28371]
[New LWP 28372]
[New LWP 28373]
[New LWP 28374]
[New LWP 28375]
[New LWP 28376]
[New LWP 28377]
[New LWP 28378]
[New LWP 28379]
[New LWP 28380]
[New LWP 28381]
[New LWP 28382]
[New LWP 28383]
[New LWP 28384]
[New LWP 28385]
[New LWP 28386]
[New LWP 28387]
[New LWP 28388]
[New LWP 28389]
[New LWP 28390]
[New LWP 28391]
[New LWP 28392]
[New LWP 28393]
[New LWP 28394]
[New LWP 28395]
[New LWP 28396]
[New LWP 28397]
[New LWP 28398]
[New LWP 28399]
[New LWP 28400]
[New LWP 28401]
[New LWP 28402]
[New LWP 28403]
[New LWP 28404]
[New LWP 28405]
[New LWP 28406]
[New LWP 28407]
[New LWP 28408]
[New LWP 28409]
[New LWP 28410]
[New LWP 28411]
[New LWP 28412]
[New LWP 28413]
[New LWP 28414]
[New LWP 28415]
[New LWP 28416]
[New LWP 28417]
[New LWP 28418]
[New LWP 28419]
[New LWP 28420]
[New LWP 28421]
[New LWP 28422]
[New LWP 28423]
[New LWP 28424]
[New LWP 28425]
[New LWP 28426]
[New LWP 28427]
[New LWP 28428]
[New LWP 28429]
[New LWP 28430]
[New LWP 28431]
[New LWP 28432]
[New LWP 28433]
[New LWP 28434]
[New LWP 28435]
[New LWP 28436]
[New LWP 28437]
[New LWP 28438]
[New LWP 28439]
[New LWP 28440]
[New LWP 28441]
[New LWP 28442]
[New LWP 28443]
[New LWP 28444]
[New LWP 28445]
[New LWP 28446]
[New LWP 28447]
[New LWP 28448]
[New LWP 28449]
[New LWP 28450]
[New LWP 28451]
[New LWP 28452]
[New LWP 28453]
[New LWP 28454]
[New LWP 28455]
[New LWP 28456]
[New LWP 28457]
[New LWP 28458]
[New LWP 28459]
[New LWP 28460]
[New LWP 28461]
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
0x00007f4fad50bd50 in ?? ()
Id Target Id Frame
* 1 LWP 28331 "kudu" 0x00007f4fad50bd50 in ?? ()
2 LWP 28332 "kudu" 0x00007f4fa88cf7a0 in ?? ()
3 LWP 28333 "kudu" 0x00007f4fad507fb9 in ?? ()
4 LWP 28334 "kudu" 0x00007f4fad507fb9 in ?? ()
5 LWP 28335 "kudu" 0x00007f4fad507fb9 in ?? ()
6 LWP 28336 "kernel-watcher-" 0x00007f4fad507fb9 in ?? ()
7 LWP 28343 "ntp client-2834" 0x00007f4fad50b9e2 in ?? ()
8 LWP 28344 "file cache-evic" 0x00007f4fad507fb9 in ?? ()
9 LWP 28345 "sq_acceptor" 0x00007f4fa88ffcb9 in ?? ()
10 LWP 28348 "rpc reactor-283" 0x00007f4fa890ca47 in ?? ()
11 LWP 28349 "rpc reactor-283" 0x00007f4fa890ca47 in ?? ()
12 LWP 28350 "rpc reactor-283" 0x00007f4fa890ca47 in ?? ()
13 LWP 28351 "rpc reactor-283" 0x00007f4fa890ca47 in ?? ()
14 LWP 28352 "MaintenanceMgr " 0x00007f4fad507ad3 in ?? ()
15 LWP 28353 "txn-status-mana" 0x00007f4fad507fb9 in ?? ()
16 LWP 28354 "collect_and_rem" 0x00007f4fad507fb9 in ?? ()
17 LWP 28355 "tc-session-exp-" 0x00007f4fad507fb9 in ?? ()
18 LWP 28356 "rpc worker-2835" 0x00007f4fad507ad3 in ?? ()
19 LWP 28357 "rpc worker-2835" 0x00007f4fad507ad3 in ?? ()
20 LWP 28358 "rpc worker-2835" 0x00007f4fad507ad3 in ?? ()
21 LWP 28359 "rpc worker-2835" 0x00007f4fad507ad3 in ?? ()
22 LWP 28360 "rpc worker-2836" 0x00007f4fad507ad3 in ?? ()
23 LWP 28361 "rpc worker-2836" 0x00007f4fad507ad3 in ?? ()
24 LWP 28362 "rpc worker-2836" 0x00007f4fad507ad3 in ?? ()
25 LWP 28363 "rpc worker-2836" 0x00007f4fad507ad3 in ?? ()
26 LWP 28364 "rpc worker-2836" 0x00007f4fad507ad3 in ?? ()
27 LWP 28365 "rpc worker-2836" 0x00007f4fad507ad3 in ?? ()
28 LWP 28366 "rpc worker-2836" 0x00007f4fad507ad3 in ?? ()
29 LWP 28367 "rpc worker-2836" 0x00007f4fad507ad3 in ?? ()
30 LWP 28368 "rpc worker-2836" 0x00007f4fad507ad3 in ?? ()
31 LWP 28369 "rpc worker-2836" 0x00007f4fad507ad3 in ?? ()
32 LWP 28370 "rpc worker-2837" 0x00007f4fad507ad3 in ?? ()
33 LWP 28371 "rpc worker-2837" 0x00007f4fad507ad3 in ?? ()
34 LWP 28372 "rpc worker-2837" 0x00007f4fad507ad3 in ?? ()
35 LWP 28373 "rpc worker-2837" 0x00007f4fad507ad3 in ?? ()
36 LWP 28374 "rpc worker-2837" 0x00007f4fad507ad3 in ?? ()
37 LWP 28375 "rpc worker-2837" 0x00007f4fad507ad3 in ?? ()
38 LWP 28376 "rpc worker-2837" 0x00007f4fad507ad3 in ?? ()
39 LWP 28377 "rpc worker-2837" 0x00007f4fad507ad3 in ?? ()
40 LWP 28378 "rpc worker-2837" 0x00007f4fad507ad3 in ?? ()
41 LWP 28379 "rpc worker-2837" 0x00007f4fad507ad3 in ?? ()
42 LWP 28380 "rpc worker-2838" 0x00007f4fad507ad3 in ?? ()
43 LWP 28381 "rpc worker-2838" 0x00007f4fad507ad3 in ?? ()
44 LWP 28382 "rpc worker-2838" 0x00007f4fad507ad3 in ?? ()
45 LWP 28383 "rpc worker-2838" 0x00007f4fad507ad3 in ?? ()
46 LWP 28384 "rpc worker-2838" 0x00007f4fad507ad3 in ?? ()
47 LWP 28385 "rpc worker-2838" 0x00007f4fad507ad3 in ?? ()
48 LWP 28386 "rpc worker-2838" 0x00007f4fad507ad3 in ?? ()
49 LWP 28387 "rpc worker-2838" 0x00007f4fad507ad3 in ?? ()
50 LWP 28388 "rpc worker-2838" 0x00007f4fad507ad3 in ?? ()
51 LWP 28389 "rpc worker-2838" 0x00007f4fad507ad3 in ?? ()
52 LWP 28390 "rpc worker-2839" 0x00007f4fad507ad3 in ?? ()
53 LWP 28391 "rpc worker-2839" 0x00007f4fad507ad3 in ?? ()
54 LWP 28392 "rpc worker-2839" 0x00007f4fad507ad3 in ?? ()
55 LWP 28393 "rpc worker-2839" 0x00007f4fad507ad3 in ?? ()
56 LWP 28394 "rpc worker-2839" 0x00007f4fad507ad3 in ?? ()
57 LWP 28395 "rpc worker-2839" 0x00007f4fad507ad3 in ?? ()
58 LWP 28396 "rpc worker-2839" 0x00007f4fad507ad3 in ?? ()
59 LWP 28397 "rpc worker-2839" 0x00007f4fad507ad3 in ?? ()
60 LWP 28398 "rpc worker-2839" 0x00007f4fad507ad3 in ?? ()
61 LWP 28399 "rpc worker-2839" 0x00007f4fad507ad3 in ?? ()
62 LWP 28400 "rpc worker-2840" 0x00007f4fad507ad3 in ?? ()
63 LWP 28401 "rpc worker-2840" 0x00007f4fad507ad3 in ?? ()
64 LWP 28402 "rpc worker-2840" 0x00007f4fad507ad3 in ?? ()
65 LWP 28403 "rpc worker-2840" 0x00007f4fad507ad3 in ?? ()
66 LWP 28404 "rpc worker-2840" 0x00007f4fad507ad3 in ?? ()
67 LWP 28405 "rpc worker-2840" 0x00007f4fad507ad3 in ?? ()
68 LWP 28406 "rpc worker-2840" 0x00007f4fad507ad3 in ?? ()
69 LWP 28407 "rpc worker-2840" 0x00007f4fad507ad3 in ?? ()
70 LWP 28408 "rpc worker-2840" 0x00007f4fad507ad3 in ?? ()
71 LWP 28409 "rpc worker-2840" 0x00007f4fad507ad3 in ?? ()
72 LWP 28410 "rpc worker-2841" 0x00007f4fad507ad3 in ?? ()
73 LWP 28411 "rpc worker-2841" 0x00007f4fad507ad3 in ?? ()
74 LWP 28412 "rpc worker-2841" 0x00007f4fad507ad3 in ?? ()
75 LWP 28413 "rpc worker-2841" 0x00007f4fad507ad3 in ?? ()
76 LWP 28414 "rpc worker-2841" 0x00007f4fad507ad3 in ?? ()
77 LWP 28415 "rpc worker-2841" 0x00007f4fad507ad3 in ?? ()
78 LWP 28416 "rpc worker-2841" 0x00007f4fad507ad3 in ?? ()
79 LWP 28417 "rpc worker-2841" 0x00007f4fad507ad3 in ?? ()
80 LWP 28418 "rpc worker-2841" 0x00007f4fad507ad3 in ?? ()
81 LWP 28419 "rpc worker-2841" 0x00007f4fad507ad3 in ?? ()
82 LWP 28420 "rpc worker-2842" 0x00007f4fad507ad3 in ?? ()
83 LWP 28421 "rpc worker-2842" 0x00007f4fad507ad3 in ?? ()
84 LWP 28422 "rpc worker-2842" 0x00007f4fad507ad3 in ?? ()
85 LWP 28423 "rpc worker-2842" 0x00007f4fad507ad3 in ?? ()
86 LWP 28424 "rpc worker-2842" 0x00007f4fad507ad3 in ?? ()
87 LWP 28425 "rpc worker-2842" 0x00007f4fad507ad3 in ?? ()
88 LWP 28426 "rpc worker-2842" 0x00007f4fad507ad3 in ?? ()
89 LWP 28427 "rpc worker-2842" 0x00007f4fad507ad3 in ?? ()
90 LWP 28428 "rpc worker-2842" 0x00007f4fad507ad3 in ?? ()
91 LWP 28429 "rpc worker-2842" 0x00007f4fad507ad3 in ?? ()
92 LWP 28430 "rpc worker-2843" 0x00007f4fad507ad3 in ?? ()
93 LWP 28431 "rpc worker-2843" 0x00007f4fad507ad3 in ?? ()
94 LWP 28432 "rpc worker-2843" 0x00007f4fad507ad3 in ?? ()
95 LWP 28433 "rpc worker-2843" 0x00007f4fad507ad3 in ?? ()
96 LWP 28434 "rpc worker-2843" 0x00007f4fad507ad3 in ?? ()
97 LWP 28435 "rpc worker-2843" 0x00007f4fad507ad3 in ?? ()
98 LWP 28436 "rpc worker-2843" 0x00007f4fad507ad3 in ?? ()
99 LWP 28437 "rpc worker-2843" 0x00007f4fad507ad3 in ?? ()
100 LWP 28438 "rpc worker-2843" 0x00007f4fad507ad3 in ?? ()
101 LWP 28439 "rpc worker-2843" 0x00007f4fad507ad3 in ?? ()
102 LWP 28440 "rpc worker-2844" 0x00007f4fad507ad3 in ?? ()
103 LWP 28441 "rpc worker-2844" 0x00007f4fad507ad3 in ?? ()
104 LWP 28442 "rpc worker-2844" 0x00007f4fad507ad3 in ?? ()
105 LWP 28443 "rpc worker-2844" 0x00007f4fad507ad3 in ?? ()
106 LWP 28444 "rpc worker-2844" 0x00007f4fad507ad3 in ?? ()
107 LWP 28445 "rpc worker-2844" 0x00007f4fad507ad3 in ?? ()
108 LWP 28446 "rpc worker-2844" 0x00007f4fad507ad3 in ?? ()
109 LWP 28447 "rpc worker-2844" 0x00007f4fad507ad3 in ?? ()
110 LWP 28448 "rpc worker-2844" 0x00007f4fad507ad3 in ?? ()
111 LWP 28449 "rpc worker-2844" 0x00007f4fad507ad3 in ?? ()
112 LWP 28450 "rpc worker-2845" 0x00007f4fad507ad3 in ?? ()
113 LWP 28451 "rpc worker-2845" 0x00007f4fad507ad3 in ?? ()
114 LWP 28452 "rpc worker-2845" 0x00007f4fad507ad3 in ?? ()
115 LWP 28453 "rpc worker-2845" 0x00007f4fad507ad3 in ?? ()
116 LWP 28454 "rpc worker-2845" 0x00007f4fad507ad3 in ?? ()
117 LWP 28455 "rpc worker-2845" 0x00007f4fad507ad3 in ?? ()
118 LWP 28456 "diag-logger-284" 0x00007f4fad507fb9 in ?? ()
119 LWP 28457 "result-tracker-" 0x00007f4fad507fb9 in ?? ()
120 LWP 28458 "excess-log-dele" 0x00007f4fad507fb9 in ?? ()
121 LWP 28459 "acceptor-28459" 0x00007f4fa890e0c7 in ?? ()
122 LWP 28460 "heartbeat-28460" 0x00007f4fad507fb9 in ?? ()
123 LWP 28461 "maintenance_sch" 0x00007f4fad507fb9 in ?? ()
Thread 123 (LWP 28461):
#0 0x00007f4fad507fb9 in ?? ()
#1 0x00007b0100000000 in ?? ()
#2 0x00000000000000fc in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007f4f617b96c0 in ?? ()
#6 0x00000000000001f8 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 28460):
#0 0x00007f4fad507fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 28459):
#0 0x00007f4fa890e0c7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 28458):
#0 0x00007f4fad507fb9 in ?? ()
#1 0x00007f4f62fbc940 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff18deb770 in ?? ()
#5 0x00007f4f62fbc7b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 28457):
#0 0x00007f4fad507fb9 in ?? ()
#1 0x0000000085352f90 in ?? ()
#2 0x000000000000003f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001008 in ?? ()
#5 0x00007f4f637bd800 in ?? ()
#6 0x000000000000007e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 28456):
#0 0x00007f4fad507fb9 in ?? ()
#1 0x00007f4fa6888008 in ?? ()
#2 0x000000000000003f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000c90 in ?? ()
#5 0x00007f4f63fbe750 in ?? ()
#6 0x000000000000007e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 28455):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 28454):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 28453):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 28452):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 28451):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 28450):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 28449):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 28448):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 28447):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 28446):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 28445):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 28444):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 28443):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 28442):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 28441):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 28440):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 28439):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 28438):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 28437):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 28436):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 28435):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 28434):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 28433):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 28432):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 28431):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 28430):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 28429):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 28428):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 28427):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 28426):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 28425):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 28424):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 28423):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 28422):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 28421):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 28420):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 28419):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 28418):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 28417):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 28416):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 28415):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x00000000000003e8 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b24001147c8 in ?? ()
#4 0x00007f4f791ba710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4f791ba730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 76 (LWP 28414):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x000000000000038a in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240010ffc8 in ?? ()
#4 0x00007f4f799bb710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4f799bb730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 28413):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x00000000000002d4 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240010d7d8 in ?? ()
#4 0x00007f4f7a1bc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4f7a1bc730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 28412):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b2400108fd8 in ?? ()
#4 0x00007f4f7a9bd710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4f7a9bd730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 73 (LWP 28411):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 28410):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 28409):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 28408):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 28407):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 28406):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 28405):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 28404):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 28403):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 28402):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 28401):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 28400):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 28399):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 28398):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 28397):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 28396):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 28395):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000b902c in ?? ()
#4 0x00007f4f835bc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4f835bc730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x007f0400000026c3 in ?? ()
#9 0x00007f4fad507770 in ?? ()
#10 0x00007f4f835bc730 in ?? ()
#11 0x0002008300000dfe in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 28394):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 28393):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 28392):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 28391):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 28390):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 28389):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 28388):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 28387):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 28386):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 28385):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 28384):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 28383):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 28382):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 28381):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 28380):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 28379):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 28378):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 28377):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 28376):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 28375):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240005ffec in ?? ()
#4 0x00007f4f8d9be710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4f8d9be730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f4fad507770 in ?? ()
#10 0x00007f4f8d9be730 in ?? ()
#11 0x00007f4fa5969c30 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 36 (LWP 28374):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 28373):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 28372):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 28371):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 28370):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 28369):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 28368):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 28367):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 28366):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 28365):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 28364):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 28363):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 28362):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 28361):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 28360):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 28359):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 28358):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 28357):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 28356):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 28355):
#0 0x00007f4fad507fb9 in ?? ()
#1 0x0000000017a335c8 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800003a00 in ?? ()
#5 0x00007f4f97f8e700 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 28354):
#0 0x00007f4fad507fb9 in ?? ()
#1 0x00007f4f9878f9a8 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000372d8 in ?? ()
#5 0x00007f4f9878f840 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 28353):
#0 0x00007f4fad507fb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007f4f98f90410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 28352):
#0 0x00007f4fad507ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 28351):
#0 0x00007f4fa890ca47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 28350):
#0 0x00007f4fa890ca47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 11 (LWP 28349):
#0 0x00007f4fa890ca47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 10 (LWP 28348):
#0 0x00007f4fa890ca47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 9 (LWP 28345):
#0 0x00007f4fa88ffcb9 in ?? ()
#1 0x00007f4fa17bcc10 in ?? ()
#2 0x00007b040000a860 in ?? ()
#3 0x00007f4fa17bdb80 in ?? ()
#4 0x00007f4fa17bcc10 in ?? ()
#5 0x00007b040000a860 in ?? ()
#6 0x00000000004888a3 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007f4fa6208000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007f4fa17bdb80 in ?? ()
#10 0x00007f4fb22e4908 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 8 (LWP 28344):
#0 0x00007f4fad507fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 7 (LWP 28343):
#0 0x00007f4fad50b9e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 28336):
#0 0x00007f4fad507fb9 in ?? ()
#1 0x00007f4fa27bea40 in ?? ()
#2 0x0000000000000142 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400035b98 in ?? ()
#5 0x00007f4fa27be5d0 in ?? ()
#6 0x0000000000000284 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 28335):
#0 0x00007f4fad507fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 28334):
#0 0x00007f4fad507fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 28333):
#0 0x00007f4fad507fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 28332):
#0 0x00007f4fa88cf7a0 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 28331):
#0 0x00007f4fad50bd50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x0000000000467b2b in __sanitizer::internal_alloc_placeholder ()
#3 0x00007f4fa7b2dcc0 in ?? ()
#4 0x00007f4fa7b2dcc0 in ?? ()
#5 0x00007fff18deb580 in ?? ()
#6 0x000000000048aef4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000a94636 in ?? ()
#9 0x00007f4fa7b2dcc0 in ?? ()
#10 0x00007f4faba33f0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250624 19:59:16.299504 27399 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu with pid 28065
I20250624 19:59:16.356916 27399 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu with pid 28198
I20250624 19:59:16.420176 27399 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu with pid 28331
I20250624 19:59:16.475961 27399 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskwaEKLl/build/tsan/bin/kudu with pid 27973
2025-06-24T19:59:16Z chronyd exiting
I20250624 19:59:16.536269 27399 test_util.cc:183] -----------------------------------------------
I20250624 19:59:16.536489 27399 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskwaEKLl/test-tmp/tablet_copy-itest.5.TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert.1750795074187602-27399-0
[ FAILED ] TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert (71573 ms)
[----------] 3 tests from TabletCopyITest (82181 ms total)
[----------] 1 test from FaultFlags/BadTabletCopyITest
[ RUN ] FaultFlags/BadTabletCopyITest.TestBadCopy/0
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:1510: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] FaultFlags/BadTabletCopyITest.TestBadCopy/0 (8 ms)
[----------] 1 test from FaultFlags/BadTabletCopyITest (8 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 2 test suites ran. (82190 ms total)
[ PASSED ] 1 test.
[ SKIPPED ] 2 tests, listed below:
[ SKIPPED ] TabletCopyITest.TestConcurrentTabletCopys
[ SKIPPED ] FaultFlags/BadTabletCopyITest.TestBadCopy/0
[ FAILED ] 1 test, listed below:
[ FAILED ] TabletCopyITest.TestDownloadWalInParallelWithHeavyInsert
1 FAILED TEST