Note: This is test shard 1 of 6.
[==========] Running 5 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 4 tests from TabletCopyITest
[ RUN ] TabletCopyITest.TestRejectRogueLeader
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:172: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] TabletCopyITest.TestRejectRogueLeader (11 ms)
[ RUN ] TabletCopyITest.TestDeleteLeaderDuringTabletCopyStressTest
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:727: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] TabletCopyITest.TestDeleteLeaderDuringTabletCopyStressTest (7 ms)
[ RUN ] TabletCopyITest.TestTabletCopyThrottling
2025-06-25T13:57:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T13:57:28Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250625 13:57:28.856084 18577 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
/tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.36.126:42751
--webserver_interface=127.18.36.126
--webserver_port=0
--builtin_ntp_servers=127.18.36.84:33879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.36.126:42751
--master_tombstone_evicted_tablet_replicas=false with env {}
W20250625 13:57:29.203979 18586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:29.204583 18586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:29.205025 18586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:29.245538 18586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:57:29.245890 18586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:29.246114 18586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:57:29.246320 18586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:57:29.284603 18586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.36.84:33879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/wal
--master_tombstone_evicted_tablet_replicas=false
--ipki_ca_key_size=768
--master_addresses=127.18.36.126:42751
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.36.126:42751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.36.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:29.286051 18586 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:29.287884 18586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:29.304917 18595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:29.305087 18593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:29.307053 18586 server_base.cc:1048] running on GCE node
W20250625 13:57:29.305634 18592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:30.518122 18586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:30.521792 18586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:30.523438 18586 hybrid_clock.cc:648] HybridClock initialized: now 1750859850523452 us; error 144 us; skew 500 ppm
I20250625 13:57:30.524485 18586 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:30.533030 18586 webserver.cc:469] Webserver started at http://127.18.36.126:42585/ using document root <none> and password file <none>
I20250625 13:57:30.534502 18586 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:30.534796 18586 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:30.535434 18586 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:30.546355 18586 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data/instance:
uuid: "919e98dfa40245859ff8ff13df312621"
format_stamp: "Formatted at 2025-06-25 13:57:30 on dist-test-slave-690w"
I20250625 13:57:30.547760 18586 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/wal/instance:
uuid: "919e98dfa40245859ff8ff13df312621"
format_stamp: "Formatted at 2025-06-25 13:57:30 on dist-test-slave-690w"
I20250625 13:57:30.556515 18586 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20250625 13:57:30.563004 18602 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:30.564213 18586 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 13:57:30.564579 18586 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/wal
uuid: "919e98dfa40245859ff8ff13df312621"
format_stamp: "Formatted at 2025-06-25 13:57:30 on dist-test-slave-690w"
I20250625 13:57:30.564924 18586 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:30.624179 18586 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:30.625913 18586 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:30.626416 18586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:30.707835 18586 rpc_server.cc:307] RPC server started. Bound to: 127.18.36.126:42751
I20250625 13:57:30.707892 18653 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.36.126:42751 every 8 connection(s)
I20250625 13:57:30.710863 18586 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data/info.pb
I20250625 13:57:30.716647 18654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:30.718387 18577 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu as pid 18586
I20250625 13:57:30.718832 18577 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/wal/instance
I20250625 13:57:30.738539 18654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621: Bootstrap starting.
I20250625 13:57:30.744567 18654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:30.746745 18654 log.cc:826] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:30.752323 18654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621: No bootstrap required, opened a new log
I20250625 13:57:30.774137 18654 raft_consensus.cc:357] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919e98dfa40245859ff8ff13df312621" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 42751 } }
I20250625 13:57:30.775262 18654 raft_consensus.cc:383] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:30.775624 18654 raft_consensus.cc:738] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 919e98dfa40245859ff8ff13df312621, State: Initialized, Role: FOLLOWER
I20250625 13:57:30.776679 18654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919e98dfa40245859ff8ff13df312621" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 42751 } }
I20250625 13:57:30.777495 18654 raft_consensus.cc:397] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:57:30.777942 18654 raft_consensus.cc:491] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:57:30.778437 18654 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:30.786459 18654 raft_consensus.cc:513] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919e98dfa40245859ff8ff13df312621" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 42751 } }
I20250625 13:57:30.787647 18654 leader_election.cc:304] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 919e98dfa40245859ff8ff13df312621; no voters:
I20250625 13:57:30.790350 18654 leader_election.cc:290] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:57:30.791230 18659 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:57:30.796404 18659 raft_consensus.cc:695] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 1 LEADER]: Becoming Leader. State: Replica: 919e98dfa40245859ff8ff13df312621, State: Running, Role: LEADER
I20250625 13:57:30.797490 18654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:57:30.797494 18659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919e98dfa40245859ff8ff13df312621" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 42751 } }
I20250625 13:57:30.809010 18660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "919e98dfa40245859ff8ff13df312621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919e98dfa40245859ff8ff13df312621" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 42751 } } }
I20250625 13:57:30.810163 18660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [sys.catalog]: This master's current role is: LEADER
I20250625 13:57:30.813441 18661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 919e98dfa40245859ff8ff13df312621. Latest consensus state: current_term: 1 leader_uuid: "919e98dfa40245859ff8ff13df312621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919e98dfa40245859ff8ff13df312621" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 42751 } } }
I20250625 13:57:30.814348 18661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [sys.catalog]: This master's current role is: LEADER
I20250625 13:57:30.819730 18666 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:57:30.831568 18666 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:57:30.853170 18666 catalog_manager.cc:1349] Generated new cluster ID: 8be7eb3d54b143d4bafb3d9940c1a4cd
I20250625 13:57:30.853571 18666 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:57:30.873773 18666 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 13:57:30.875341 18666 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:57:30.889467 18666 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621: Generated new TSK 0
I20250625 13:57:30.890710 18666 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 13:57:30.912461 18577 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
/tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.36.65:0
--local_ip_for_outbound_sockets=127.18.36.65
--webserver_interface=127.18.36.65
--webserver_port=0
--tserver_master_addrs=127.18.36.126:42751
--builtin_ntp_servers=127.18.36.84:33879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--num_tablets_to_copy_simultaneously=1 with env {}
W20250625 13:57:31.232672 18678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:31.233198 18678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:31.233840 18678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:31.271070 18678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:31.272311 18678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.36.65
I20250625 13:57:31.309881 18678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.36.84:33879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.36.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.36.65
--webserver_port=0
--tserver_master_addrs=127.18.36.126:42751
--num_tablets_to_copy_simultaneously=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.36.65
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:31.311657 18678 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:31.313585 18678 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:31.333813 18685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:31.337690 18684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:31.334177 18687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:32.543989 18686 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 13:57:32.544034 18678 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:57:32.549235 18678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:32.552341 18678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:32.553848 18678 hybrid_clock.cc:648] HybridClock initialized: now 1750859852553779 us; error 87 us; skew 500 ppm
I20250625 13:57:32.554808 18678 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:32.568492 18678 webserver.cc:469] Webserver started at http://127.18.36.65:38505/ using document root <none> and password file <none>
I20250625 13:57:32.569542 18678 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:32.569799 18678 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:32.570271 18678 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:32.574940 18678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/data/instance:
uuid: "60ae629379bf4f0aa635f1244c320df5"
format_stamp: "Formatted at 2025-06-25 13:57:32 on dist-test-slave-690w"
I20250625 13:57:32.576171 18678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/wal/instance:
uuid: "60ae629379bf4f0aa635f1244c320df5"
format_stamp: "Formatted at 2025-06-25 13:57:32 on dist-test-slave-690w"
I20250625 13:57:32.584686 18678 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20250625 13:57:32.591507 18694 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:32.592763 18678 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20250625 13:57:32.593147 18678 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/wal
uuid: "60ae629379bf4f0aa635f1244c320df5"
format_stamp: "Formatted at 2025-06-25 13:57:32 on dist-test-slave-690w"
I20250625 13:57:32.593550 18678 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:32.656891 18678 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:32.658480 18678 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:32.658979 18678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:32.662214 18678 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:32.666750 18678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:32.666996 18678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:32.667307 18678 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:32.667479 18678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:32.889591 18678 rpc_server.cc:307] RPC server started. Bound to: 127.18.36.65:32959
I20250625 13:57:32.890000 18806 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.36.65:32959 every 8 connection(s)
I20250625 13:57:32.892640 18678 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/data/info.pb
I20250625 13:57:32.893339 18577 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu as pid 18678
I20250625 13:57:32.893745 18577 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-0/wal/instance
I20250625 13:57:32.904040 18577 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
/tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.36.66:0
--local_ip_for_outbound_sockets=127.18.36.66
--webserver_interface=127.18.36.66
--webserver_port=0
--tserver_master_addrs=127.18.36.126:42751
--builtin_ntp_servers=127.18.36.84:33879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--num_tablets_to_copy_simultaneously=1 with env {}
I20250625 13:57:32.932180 18807 heartbeater.cc:344] Connected to a master server at 127.18.36.126:42751
I20250625 13:57:32.932893 18807 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:32.934983 18807 heartbeater.cc:507] Master 127.18.36.126:42751 requested a full tablet report, sending...
I20250625 13:57:32.938974 18619 ts_manager.cc:194] Registered new tserver with Master: 60ae629379bf4f0aa635f1244c320df5 (127.18.36.65:32959)
I20250625 13:57:32.941613 18619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.36.65:55057
W20250625 13:57:33.353458 18811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:33.354107 18811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:33.354682 18811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:33.389679 18811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:33.390719 18811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.36.66
I20250625 13:57:33.431205 18811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.36.84:33879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.36.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.36.66
--webserver_port=0
--tserver_master_addrs=127.18.36.126:42751
--num_tablets_to_copy_simultaneously=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.36.66
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:33.432833 18811 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:33.434938 18811 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:33.463814 18817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:33.947332 18807 heartbeater.cc:499] Master 127.18.36.126:42751 was elected leader, sending a full tablet report...
W20250625 13:57:33.469514 18818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:33.472106 18820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:33.474254 18811 server_base.cc:1048] running on GCE node
I20250625 13:57:34.746320 18811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:34.750710 18811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:34.752250 18811 hybrid_clock.cc:648] HybridClock initialized: now 1750859854752189 us; error 78 us; skew 500 ppm
I20250625 13:57:34.753300 18811 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:34.764669 18811 webserver.cc:469] Webserver started at http://127.18.36.66:39019/ using document root <none> and password file <none>
I20250625 13:57:34.765933 18811 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:34.766203 18811 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:34.766769 18811 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:34.772605 18811 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data/instance:
uuid: "b0eb3498fd904cdaaccb1b99282763e3"
format_stamp: "Formatted at 2025-06-25 13:57:34 on dist-test-slave-690w"
I20250625 13:57:34.774104 18811 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/wal/instance:
uuid: "b0eb3498fd904cdaaccb1b99282763e3"
format_stamp: "Formatted at 2025-06-25 13:57:34 on dist-test-slave-690w"
I20250625 13:57:34.784436 18811 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.001s sys 0.009s
I20250625 13:57:34.793629 18827 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:34.795156 18811 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.002s
I20250625 13:57:34.795586 18811 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/wal
uuid: "b0eb3498fd904cdaaccb1b99282763e3"
format_stamp: "Formatted at 2025-06-25 13:57:34 on dist-test-slave-690w"
I20250625 13:57:34.796039 18811 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:34.872437 18811 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:34.874975 18811 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:34.875557 18811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:34.880498 18811 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:34.887913 18811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:34.888181 18811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:34.888460 18811 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:34.888670 18811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:35.282509 18811 rpc_server.cc:307] RPC server started. Bound to: 127.18.36.66:34459
I20250625 13:57:35.282763 18939 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.36.66:34459 every 8 connection(s)
I20250625 13:57:35.285691 18811 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data/info.pb
I20250625 13:57:35.286191 18577 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu as pid 18811
I20250625 13:57:35.286666 18577 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/wal/instance
I20250625 13:57:35.322292 18940 heartbeater.cc:344] Connected to a master server at 127.18.36.126:42751
I20250625 13:57:35.322961 18940 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:35.324702 18940 heartbeater.cc:507] Master 127.18.36.126:42751 requested a full tablet report, sending...
I20250625 13:57:35.327554 18619 ts_manager.cc:194] Registered new tserver with Master: b0eb3498fd904cdaaccb1b99282763e3 (127.18.36.66:34459)
I20250625 13:57:35.329295 18619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.36.66:51591
I20250625 13:57:35.342449 18577 external_mini_cluster.cc:934] 2 TS(s) registered with all masters
I20250625 13:57:35.390764 18577 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu with pid 18811
I20250625 13:57:35.413967 18577 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu with pid 18586
I20250625 13:57:35.446447 18577 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
/tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.36.126:42751
--webserver_interface=127.18.36.126
--webserver_port=42585
--builtin_ntp_servers=127.18.36.84:33879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.36.126:42751
--master_tombstone_evicted_tablet_replicas=false with env {}
W20250625 13:57:35.963927 18807 heartbeater.cc:646] Failed to heartbeat to 127.18.36.126:42751 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.36.126:42751: connect: Connection refused (error 111)
W20250625 13:57:37.039084 18951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:37.039670 18951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:37.040109 18951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:37.073721 18951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:57:37.074048 18951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:37.074357 18951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:57:37.074620 18951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:57:37.113085 18951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.36.84:33879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/wal
--master_tombstone_evicted_tablet_replicas=false
--ipki_ca_key_size=768
--master_addresses=127.18.36.126:42751
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.36.126:42751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.36.126
--webserver_port=42585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:37.114574 18951 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:37.116469 18951 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:37.132555 18958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:37.133011 18959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:37.135268 18961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:38.336617 18960 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 13:57:38.336678 18951 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:57:38.340839 18951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:38.343823 18951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:38.345338 18951 hybrid_clock.cc:648] HybridClock initialized: now 1750859858345274 us; error 63 us; skew 500 ppm
I20250625 13:57:38.346304 18951 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:38.359652 18951 webserver.cc:469] Webserver started at http://127.18.36.126:42585/ using document root <none> and password file <none>
I20250625 13:57:38.360785 18951 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:38.361058 18951 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:38.370299 18951 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20250625 13:57:38.375919 18969 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:38.377357 18951 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20250625 13:57:38.377780 18951 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/wal
uuid: "919e98dfa40245859ff8ff13df312621"
format_stamp: "Formatted at 2025-06-25 13:57:30 on dist-test-slave-690w"
I20250625 13:57:38.379961 18951 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:38.433902 18951 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:38.435562 18951 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:38.436087 18951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:38.516062 18951 rpc_server.cc:307] RPC server started. Bound to: 127.18.36.126:42751
I20250625 13:57:38.516214 19020 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.36.126:42751 every 8 connection(s)
I20250625 13:57:38.519407 18951 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/master-0/data/info.pb
I20250625 13:57:38.520011 18577 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu as pid 18951
I20250625 13:57:38.537240 19021 sys_catalog.cc:263] Verifying existing consensus state
I20250625 13:57:38.544850 19021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621: Bootstrap starting.
I20250625 13:57:38.587324 19021 log.cc:826] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:38.605587 19021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:57:38.606520 19021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621: Bootstrap complete.
I20250625 13:57:38.628834 19021 raft_consensus.cc:357] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919e98dfa40245859ff8ff13df312621" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 42751 } }
I20250625 13:57:38.631601 19021 raft_consensus.cc:738] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 919e98dfa40245859ff8ff13df312621, State: Initialized, Role: FOLLOWER
I20250625 13:57:38.632404 19021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919e98dfa40245859ff8ff13df312621" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 42751 } }
I20250625 13:57:38.632962 19021 raft_consensus.cc:397] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:57:38.633226 19021 raft_consensus.cc:491] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:57:38.633625 19021 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 1 FOLLOWER]: Advancing to term 2
I20250625 13:57:38.641137 19021 raft_consensus.cc:513] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919e98dfa40245859ff8ff13df312621" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 42751 } }
I20250625 13:57:38.641994 19021 leader_election.cc:304] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 919e98dfa40245859ff8ff13df312621; no voters:
I20250625 13:57:38.643836 19021 leader_election.cc:290] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 13:57:38.644359 19026 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 13:57:38.646813 19026 raft_consensus.cc:695] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [term 2 LEADER]: Becoming Leader. State: Replica: 919e98dfa40245859ff8ff13df312621, State: Running, Role: LEADER
I20250625 13:57:38.647733 19026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919e98dfa40245859ff8ff13df312621" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 42751 } }
I20250625 13:57:38.648886 19021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:57:38.660125 19028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 919e98dfa40245859ff8ff13df312621. Latest consensus state: current_term: 2 leader_uuid: "919e98dfa40245859ff8ff13df312621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919e98dfa40245859ff8ff13df312621" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 42751 } } }
I20250625 13:57:38.661054 19028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [sys.catalog]: This master's current role is: LEADER
I20250625 13:57:38.662060 19027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "919e98dfa40245859ff8ff13df312621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "919e98dfa40245859ff8ff13df312621" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 42751 } } }
I20250625 13:57:38.662894 19027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621 [sys.catalog]: This master's current role is: LEADER
I20250625 13:57:38.665733 19034 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:57:38.687408 19034 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:57:38.692694 19034 catalog_manager.cc:1261] Loaded cluster ID: 8be7eb3d54b143d4bafb3d9940c1a4cd
I20250625 13:57:38.692963 19034 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:57:38.699231 19034 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:57:38.703208 19034 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 919e98dfa40245859ff8ff13df312621: Loaded TSK: 0
I20250625 13:57:38.704649 19034 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 13:57:38.987202 18807 heartbeater.cc:344] Connected to a master server at 127.18.36.126:42751
I20250625 13:57:38.992928 18986 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" instance_seqno: 1750859852799449) as {username='slave'} at 127.18.36.65:53499; Asking this server to re-register.
I20250625 13:57:38.994535 18807 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:38.995136 18807 heartbeater.cc:507] Master 127.18.36.126:42751 requested a full tablet report, sending...
I20250625 13:57:38.997653 18986 ts_manager.cc:194] Registered new tserver with Master: 60ae629379bf4f0aa635f1244c320df5 (127.18.36.65:32959)
I20250625 13:57:39.007889 18577 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250625 13:57:39.008564 18577 test_util.cc:276] Using random seed: -739075051
I20250625 13:57:39.079244 18986 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47688:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 13:57:39.141816 18739 tablet_service.cc:1468] Processing CreateTablet for tablet a0dfec98b40b41839827eb084e6a13b8 (DEFAULT_TABLE table=test-workload [id=05cddba3dcc243eb926ff4f5aebfed79]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250625 13:57:39.140879 18742 tablet_service.cc:1468] Processing CreateTablet for tablet ec8fd40a598d4c2cba01a560404670a3 (DEFAULT_TABLE table=test-workload [id=05cddba3dcc243eb926ff4f5aebfed79]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250625 13:57:39.143399 18739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0dfec98b40b41839827eb084e6a13b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:39.144178 18742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8fd40a598d4c2cba01a560404670a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:39.144371 18741 tablet_service.cc:1468] Processing CreateTablet for tablet 965f316ceb3d434f9c1255a72646a987 (DEFAULT_TABLE table=test-workload [id=05cddba3dcc243eb926ff4f5aebfed79]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250625 13:57:39.140795 18740 tablet_service.cc:1468] Processing CreateTablet for tablet 59bd2bdbe1c042488409306fed5d474b (DEFAULT_TABLE table=test-workload [id=05cddba3dcc243eb926ff4f5aebfed79]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250625 13:57:39.147444 18741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 965f316ceb3d434f9c1255a72646a987. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:39.149412 18740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59bd2bdbe1c042488409306fed5d474b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:39.182652 19054 tablet_bootstrap.cc:492] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5: Bootstrap starting.
I20250625 13:57:39.190296 19054 tablet_bootstrap.cc:654] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:39.192181 19054 log.cc:826] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:39.198441 19054 tablet_bootstrap.cc:492] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5: No bootstrap required, opened a new log
I20250625 13:57:39.198947 19054 ts_tablet_manager.cc:1397] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.003s
I20250625 13:57:39.219658 19054 raft_consensus.cc:357] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.220309 19054 raft_consensus.cc:383] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:39.220590 19054 raft_consensus.cc:738] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ae629379bf4f0aa635f1244c320df5, State: Initialized, Role: FOLLOWER
I20250625 13:57:39.221424 19054 consensus_queue.cc:260] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.221989 19054 raft_consensus.cc:397] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:57:39.222301 19054 raft_consensus.cc:491] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:57:39.222626 19054 raft_consensus.cc:3058] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:39.227357 19054 raft_consensus.cc:513] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.228256 19054 leader_election.cc:304] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60ae629379bf4f0aa635f1244c320df5; no voters:
I20250625 13:57:39.230463 19054 leader_election.cc:290] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:57:39.230944 19056 raft_consensus.cc:2802] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:57:39.234690 19054 ts_tablet_manager.cc:1428] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5: Time spent starting tablet: real 0.035s user 0.028s sys 0.008s
I20250625 13:57:39.235973 19054 tablet_bootstrap.cc:492] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5: Bootstrap starting.
I20250625 13:57:39.236129 19056 raft_consensus.cc:695] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5 [term 1 LEADER]: Becoming Leader. State: Replica: 60ae629379bf4f0aa635f1244c320df5, State: Running, Role: LEADER
I20250625 13:57:39.237150 19056 consensus_queue.cc:237] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.241942 19054 tablet_bootstrap.cc:654] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:39.256963 19054 tablet_bootstrap.cc:492] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5: No bootstrap required, opened a new log
I20250625 13:57:39.257388 19054 ts_tablet_manager.cc:1397] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.005s
I20250625 13:57:39.256690 18986 catalog_manager.cc:5582] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ae629379bf4f0aa635f1244c320df5 (127.18.36.65). New cstate: current_term: 1 leader_uuid: "60ae629379bf4f0aa635f1244c320df5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } health_report { overall_health: HEALTHY } } }
I20250625 13:57:39.259902 19054 raft_consensus.cc:357] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.260553 19054 raft_consensus.cc:383] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:39.260812 19054 raft_consensus.cc:738] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ae629379bf4f0aa635f1244c320df5, State: Initialized, Role: FOLLOWER
I20250625 13:57:39.261451 19054 consensus_queue.cc:260] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.262111 19054 raft_consensus.cc:397] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:57:39.262408 19054 raft_consensus.cc:491] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:57:39.262738 19054 raft_consensus.cc:3058] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:39.268107 19054 raft_consensus.cc:513] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.268883 19054 leader_election.cc:304] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60ae629379bf4f0aa635f1244c320df5; no voters:
I20250625 13:57:39.269523 19054 leader_election.cc:290] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:57:39.269757 19056 raft_consensus.cc:2802] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:57:39.270524 19056 raft_consensus.cc:695] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5 [term 1 LEADER]: Becoming Leader. State: Replica: 60ae629379bf4f0aa635f1244c320df5, State: Running, Role: LEADER
I20250625 13:57:39.271278 19056 consensus_queue.cc:237] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.271687 19054 ts_tablet_manager.cc:1428] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20250625 13:57:39.272591 19054 tablet_bootstrap.cc:492] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5: Bootstrap starting.
I20250625 13:57:39.279695 19054 tablet_bootstrap.cc:654] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:39.280630 18986 catalog_manager.cc:5582] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ae629379bf4f0aa635f1244c320df5 (127.18.36.65). New cstate: current_term: 1 leader_uuid: "60ae629379bf4f0aa635f1244c320df5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } health_report { overall_health: HEALTHY } } }
I20250625 13:57:39.286260 19054 tablet_bootstrap.cc:492] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5: No bootstrap required, opened a new log
I20250625 13:57:39.286664 19054 ts_tablet_manager.cc:1397] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20250625 13:57:39.289312 19054 raft_consensus.cc:357] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.290063 19054 raft_consensus.cc:383] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:39.290406 19054 raft_consensus.cc:738] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ae629379bf4f0aa635f1244c320df5, State: Initialized, Role: FOLLOWER
I20250625 13:57:39.291270 19054 consensus_queue.cc:260] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.292068 19054 raft_consensus.cc:397] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:57:39.292341 19054 raft_consensus.cc:491] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:57:39.292735 19054 raft_consensus.cc:3058] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:39.299088 19054 raft_consensus.cc:513] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.299880 19054 leader_election.cc:304] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60ae629379bf4f0aa635f1244c320df5; no voters:
I20250625 13:57:39.300436 19054 leader_election.cc:290] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:57:39.300592 19056 raft_consensus.cc:2802] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:57:39.301031 19056 raft_consensus.cc:695] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5 [term 1 LEADER]: Becoming Leader. State: Replica: 60ae629379bf4f0aa635f1244c320df5, State: Running, Role: LEADER
I20250625 13:57:39.301807 19056 consensus_queue.cc:237] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.302465 19054 ts_tablet_manager.cc:1428] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20250625 13:57:39.303400 19054 tablet_bootstrap.cc:492] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5: Bootstrap starting.
I20250625 13:57:39.310060 19054 tablet_bootstrap.cc:654] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:39.310256 18986 catalog_manager.cc:5582] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ae629379bf4f0aa635f1244c320df5 (127.18.36.65). New cstate: current_term: 1 leader_uuid: "60ae629379bf4f0aa635f1244c320df5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } health_report { overall_health: HEALTHY } } }
I20250625 13:57:39.317746 19054 tablet_bootstrap.cc:492] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5: No bootstrap required, opened a new log
I20250625 13:57:39.318248 19054 ts_tablet_manager.cc:1397] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20250625 13:57:39.321020 19054 raft_consensus.cc:357] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.321661 19054 raft_consensus.cc:383] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:39.321974 19054 raft_consensus.cc:738] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ae629379bf4f0aa635f1244c320df5, State: Initialized, Role: FOLLOWER
I20250625 13:57:39.322629 19054 consensus_queue.cc:260] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.323233 19054 raft_consensus.cc:397] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:57:39.323545 19054 raft_consensus.cc:491] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:57:39.323889 19054 raft_consensus.cc:3058] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:39.330188 19054 raft_consensus.cc:513] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.330940 19054 leader_election.cc:304] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60ae629379bf4f0aa635f1244c320df5; no voters:
I20250625 13:57:39.331610 19054 leader_election.cc:290] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:57:39.331780 19056 raft_consensus.cc:2802] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:57:39.332827 19056 raft_consensus.cc:695] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5 [term 1 LEADER]: Becoming Leader. State: Replica: 60ae629379bf4f0aa635f1244c320df5, State: Running, Role: LEADER
I20250625 13:57:39.333603 19054 ts_tablet_manager.cc:1428] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20250625 13:57:39.333701 19056 consensus_queue.cc:237] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:39.341953 18986 catalog_manager.cc:5582] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ae629379bf4f0aa635f1244c320df5 (127.18.36.65). New cstate: current_term: 1 leader_uuid: "60ae629379bf4f0aa635f1244c320df5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } health_report { overall_health: HEALTHY } } }
W20250625 13:57:40.791973 19074 meta_cache.cc:1261] Time spent looking up entry by key: real 0.059s user 0.002s sys 0.001s
W20250625 13:57:41.887096 19067 meta_cache.cc:1261] Time spent looking up entry by key: real 0.060s user 0.000s sys 0.000s
I20250625 13:57:45.357128 18577 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
/tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.36.66:34459
--local_ip_for_outbound_sockets=127.18.36.66
--tserver_master_addrs=127.18.36.126:42751
--webserver_port=39019
--webserver_interface=127.18.36.66
--builtin_ntp_servers=127.18.36.84:33879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--num_tablets_to_copy_simultaneously=1 with env {}
W20250625 13:57:45.806603 19091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:45.807243 19091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:45.807845 19091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:45.843994 19091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:45.845036 19091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.36.66
I20250625 13:57:45.889605 19091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.36.84:33879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.36.66:34459
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.36.66
--webserver_port=39019
--tserver_master_addrs=127.18.36.126:42751
--num_tablets_to_copy_simultaneously=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.36.66
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:45.891486 19091 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:45.894089 19091 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:45.916574 19098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:45.916883 19100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:45.917397 19097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:47.195377 19099 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1275 milliseconds
I20250625 13:57:47.195544 19091 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:57:47.196929 19091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:47.199530 19091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:47.200951 19091 hybrid_clock.cc:648] HybridClock initialized: now 1750859867200893 us; error 57 us; skew 500 ppm
I20250625 13:57:47.201894 19091 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:47.209123 19091 webserver.cc:469] Webserver started at http://127.18.36.66:39019/ using document root <none> and password file <none>
I20250625 13:57:47.210431 19091 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:47.210729 19091 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:47.220328 19091 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.005s
I20250625 13:57:47.225677 19108 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:47.226999 19091 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250625 13:57:47.227409 19091 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/wal
uuid: "b0eb3498fd904cdaaccb1b99282763e3"
format_stamp: "Formatted at 2025-06-25 13:57:34 on dist-test-slave-690w"
I20250625 13:57:47.229633 19091 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:47.295848 19091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:47.297869 19091 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:47.298429 19091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:47.301636 19091 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:47.306420 19091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:47.306694 19091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:47.306983 19091 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:47.307157 19091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:47.635563 19091 rpc_server.cc:307] RPC server started. Bound to: 127.18.36.66:34459
I20250625 13:57:47.635914 19220 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.36.66:34459 every 8 connection(s)
I20250625 13:57:47.639159 19091 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750859848574329-18577-0/minicluster-data/ts-1/data/info.pb
I20250625 13:57:47.641541 18577 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu as pid 19091
I20250625 13:57:47.690179 19221 heartbeater.cc:344] Connected to a master server at 127.18.36.126:42751
I20250625 13:57:47.691023 19221 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:47.693140 19221 heartbeater.cc:507] Master 127.18.36.126:42751 requested a full tablet report, sending...
I20250625 13:57:47.693336 19227 ts_tablet_manager.cc:927] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3: Initiating tablet copy from peer 60ae629379bf4f0aa635f1244c320df5 (127.18.36.65:32959)
I20250625 13:57:47.698047 18984 ts_manager.cc:194] Registered new tserver with Master: b0eb3498fd904cdaaccb1b99282763e3 (127.18.36.66:34459)
I20250625 13:57:47.699210 19227 tablet_copy_client.cc:323] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Beginning tablet copy session from remote peer at address 127.18.36.65:32959
I20250625 13:57:47.701757 18984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.36.66:51963
I20250625 13:57:47.714450 18782 tablet_copy_service.cc:140] P 60ae629379bf4f0aa635f1244c320df5: Received BeginTabletCopySession request for tablet 59bd2bdbe1c042488409306fed5d474b from peer b0eb3498fd904cdaaccb1b99282763e3 ({username='slave'} at 127.18.36.66:51389)
I20250625 13:57:47.715235 18782 tablet_copy_service.cc:161] P 60ae629379bf4f0aa635f1244c320df5: Beginning new tablet copy session on tablet 59bd2bdbe1c042488409306fed5d474b from peer b0eb3498fd904cdaaccb1b99282763e3 at {username='slave'} at 127.18.36.66:51389: session id = b0eb3498fd904cdaaccb1b99282763e3-59bd2bdbe1c042488409306fed5d474b
I20250625 13:57:47.723932 18782 tablet_copy_source_session.cc:215] T 59bd2bdbe1c042488409306fed5d474b P 60ae629379bf4f0aa635f1244c320df5: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 13:57:47.730764 19227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59bd2bdbe1c042488409306fed5d474b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:47.761822 19227 tablet_copy_client.cc:806] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Starting download of 0 data blocks...
I20250625 13:57:47.762669 19227 tablet_copy_client.cc:670] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Starting download of 1 WAL segments...
I20250625 13:57:47.787339 19227 tablet_copy_client.cc:538] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 13:57:47.805395 19227 tablet_bootstrap.cc:492] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3: Bootstrap starting.
I20250625 13:57:47.998029 19227 log.cc:826] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:48.707959 19221 heartbeater.cc:499] Master 127.18.36.126:42751 was elected leader, sending a full tablet report...
I20250625 13:57:49.230041 19227 tablet_bootstrap.cc:492] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=211 overwritten=0 applied=211 ignored=0} inserts{seen=2662 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:57:49.230983 19227 tablet_bootstrap.cc:492] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3: Bootstrap complete.
I20250625 13:57:49.231663 19227 ts_tablet_manager.cc:1397] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3: Time spent bootstrapping tablet: real 1.427s user 1.329s sys 0.095s
I20250625 13:57:49.245888 19227 raft_consensus.cc:357] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:49.246711 19227 raft_consensus.cc:738] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: b0eb3498fd904cdaaccb1b99282763e3, State: Initialized, Role: NON_PARTICIPANT
I20250625 13:57:49.247370 19227 consensus_queue.cc:260] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 211, Last appended: 1.211, Last appended by leader: 211, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:49.250702 19227 ts_tablet_manager.cc:1428] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3: Time spent starting tablet: real 0.019s user 0.020s sys 0.000s
I20250625 13:57:49.252463 18782 tablet_copy_service.cc:342] P 60ae629379bf4f0aa635f1244c320df5: Request end of tablet copy session b0eb3498fd904cdaaccb1b99282763e3-59bd2bdbe1c042488409306fed5d474b received from {username='slave'} at 127.18.36.66:51389
I20250625 13:57:49.252959 18782 tablet_copy_service.cc:434] P 60ae629379bf4f0aa635f1244c320df5: ending tablet copy session b0eb3498fd904cdaaccb1b99282763e3-59bd2bdbe1c042488409306fed5d474b on tablet 59bd2bdbe1c042488409306fed5d474b with peer b0eb3498fd904cdaaccb1b99282763e3
W20250625 13:57:49.257834 19227 ts_tablet_manager.cc:726] T 59bd2bdbe1c042488409306fed5d474b P b0eb3498fd904cdaaccb1b99282763e3: Tablet Copy: Invalid argument: Leader has replica of tablet 59bd2bdbe1c042488409306fed5d474b with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
I20250625 13:57:49.264739 19227 ts_tablet_manager.cc:927] T 965f316ceb3d434f9c1255a72646a987 P b0eb3498fd904cdaaccb1b99282763e3: Initiating tablet copy from peer 60ae629379bf4f0aa635f1244c320df5 (127.18.36.65:32959)
I20250625 13:57:49.266698 19227 tablet_copy_client.cc:323] T 965f316ceb3d434f9c1255a72646a987 P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Beginning tablet copy session from remote peer at address 127.18.36.65:32959
I20250625 13:57:49.268389 18782 tablet_copy_service.cc:140] P 60ae629379bf4f0aa635f1244c320df5: Received BeginTabletCopySession request for tablet 965f316ceb3d434f9c1255a72646a987 from peer b0eb3498fd904cdaaccb1b99282763e3 ({username='slave'} at 127.18.36.66:51389)
I20250625 13:57:49.268857 18782 tablet_copy_service.cc:161] P 60ae629379bf4f0aa635f1244c320df5: Beginning new tablet copy session on tablet 965f316ceb3d434f9c1255a72646a987 from peer b0eb3498fd904cdaaccb1b99282763e3 at {username='slave'} at 127.18.36.66:51389: session id = b0eb3498fd904cdaaccb1b99282763e3-965f316ceb3d434f9c1255a72646a987
I20250625 13:57:49.274091 18782 tablet_copy_source_session.cc:215] T 965f316ceb3d434f9c1255a72646a987 P 60ae629379bf4f0aa635f1244c320df5: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 13:57:49.276670 19227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 965f316ceb3d434f9c1255a72646a987. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:49.286479 19227 tablet_copy_client.cc:806] T 965f316ceb3d434f9c1255a72646a987 P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Starting download of 0 data blocks...
I20250625 13:57:49.287060 19227 tablet_copy_client.cc:670] T 965f316ceb3d434f9c1255a72646a987 P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Starting download of 1 WAL segments...
I20250625 13:57:49.300853 19227 tablet_copy_client.cc:538] T 965f316ceb3d434f9c1255a72646a987 P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 13:57:49.375453 19227 tablet_bootstrap.cc:492] T 965f316ceb3d434f9c1255a72646a987 P b0eb3498fd904cdaaccb1b99282763e3: Bootstrap starting.
I20250625 13:57:50.671084 19227 tablet_bootstrap.cc:492] T 965f316ceb3d434f9c1255a72646a987 P b0eb3498fd904cdaaccb1b99282763e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=211 overwritten=0 applied=211 ignored=0} inserts{seen=2579 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:57:50.672102 19227 tablet_bootstrap.cc:492] T 965f316ceb3d434f9c1255a72646a987 P b0eb3498fd904cdaaccb1b99282763e3: Bootstrap complete.
I20250625 13:57:50.672760 19227 ts_tablet_manager.cc:1397] T 965f316ceb3d434f9c1255a72646a987 P b0eb3498fd904cdaaccb1b99282763e3: Time spent bootstrapping tablet: real 1.298s user 1.259s sys 0.036s
I20250625 13:57:50.675380 19227 raft_consensus.cc:357] T 965f316ceb3d434f9c1255a72646a987 P b0eb3498fd904cdaaccb1b99282763e3 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:50.675863 19227 raft_consensus.cc:738] T 965f316ceb3d434f9c1255a72646a987 P b0eb3498fd904cdaaccb1b99282763e3 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: b0eb3498fd904cdaaccb1b99282763e3, State: Initialized, Role: NON_PARTICIPANT
I20250625 13:57:50.676359 19227 consensus_queue.cc:260] T 965f316ceb3d434f9c1255a72646a987 P b0eb3498fd904cdaaccb1b99282763e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 211, Last appended: 1.211, Last appended by leader: 211, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:50.680506 19227 ts_tablet_manager.cc:1428] T 965f316ceb3d434f9c1255a72646a987 P b0eb3498fd904cdaaccb1b99282763e3: Time spent starting tablet: real 0.008s user 0.002s sys 0.008s
I20250625 13:57:50.682929 18782 tablet_copy_service.cc:342] P 60ae629379bf4f0aa635f1244c320df5: Request end of tablet copy session b0eb3498fd904cdaaccb1b99282763e3-965f316ceb3d434f9c1255a72646a987 received from {username='slave'} at 127.18.36.66:51389
I20250625 13:57:50.683408 18782 tablet_copy_service.cc:434] P 60ae629379bf4f0aa635f1244c320df5: ending tablet copy session b0eb3498fd904cdaaccb1b99282763e3-965f316ceb3d434f9c1255a72646a987 on tablet 965f316ceb3d434f9c1255a72646a987 with peer b0eb3498fd904cdaaccb1b99282763e3
I20250625 13:57:50.686750 19227 ts_tablet_manager.cc:927] T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3: Initiating tablet copy from peer 60ae629379bf4f0aa635f1244c320df5 (127.18.36.65:32959)
I20250625 13:57:50.689515 19227 tablet_copy_client.cc:323] T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Beginning tablet copy session from remote peer at address 127.18.36.65:32959
I20250625 13:57:50.691406 18782 tablet_copy_service.cc:140] P 60ae629379bf4f0aa635f1244c320df5: Received BeginTabletCopySession request for tablet a0dfec98b40b41839827eb084e6a13b8 from peer b0eb3498fd904cdaaccb1b99282763e3 ({username='slave'} at 127.18.36.66:51389)
I20250625 13:57:50.692140 18782 tablet_copy_service.cc:161] P 60ae629379bf4f0aa635f1244c320df5: Beginning new tablet copy session on tablet a0dfec98b40b41839827eb084e6a13b8 from peer b0eb3498fd904cdaaccb1b99282763e3 at {username='slave'} at 127.18.36.66:51389: session id = b0eb3498fd904cdaaccb1b99282763e3-a0dfec98b40b41839827eb084e6a13b8
I20250625 13:57:50.699419 18782 tablet_copy_source_session.cc:215] T a0dfec98b40b41839827eb084e6a13b8 P 60ae629379bf4f0aa635f1244c320df5: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 13:57:50.702589 19227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0dfec98b40b41839827eb084e6a13b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:50.713500 19227 tablet_copy_client.cc:806] T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Starting download of 0 data blocks...
I20250625 13:57:50.714027 19227 tablet_copy_client.cc:670] T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Starting download of 1 WAL segments...
I20250625 13:57:50.734417 19227 tablet_copy_client.cc:538] T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 13:57:50.810565 19227 tablet_bootstrap.cc:492] T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3: Bootstrap starting.
W20250625 13:57:52.014951 19227 log.cc:927] Time spent T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3: Append to log took a long time: real 0.075s user 0.003s sys 0.000s
I20250625 13:57:52.177934 19227 tablet_bootstrap.cc:492] T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=211 overwritten=0 applied=211 ignored=0} inserts{seen=2666 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:57:52.178797 19227 tablet_bootstrap.cc:492] T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3: Bootstrap complete.
I20250625 13:57:52.179430 19227 ts_tablet_manager.cc:1397] T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3: Time spent bootstrapping tablet: real 1.369s user 1.252s sys 0.040s
I20250625 13:57:52.181315 19227 raft_consensus.cc:357] T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:52.181711 19227 raft_consensus.cc:738] T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: b0eb3498fd904cdaaccb1b99282763e3, State: Initialized, Role: NON_PARTICIPANT
I20250625 13:57:52.182235 19227 consensus_queue.cc:260] T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 211, Last appended: 1.211, Last appended by leader: 211, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:52.185333 19227 ts_tablet_manager.cc:1428] T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250625 13:57:52.187585 18782 tablet_copy_service.cc:342] P 60ae629379bf4f0aa635f1244c320df5: Request end of tablet copy session b0eb3498fd904cdaaccb1b99282763e3-a0dfec98b40b41839827eb084e6a13b8 received from {username='slave'} at 127.18.36.66:51389
I20250625 13:57:52.188220 18782 tablet_copy_service.cc:434] P 60ae629379bf4f0aa635f1244c320df5: ending tablet copy session b0eb3498fd904cdaaccb1b99282763e3-a0dfec98b40b41839827eb084e6a13b8 on tablet a0dfec98b40b41839827eb084e6a13b8 with peer b0eb3498fd904cdaaccb1b99282763e3
W20250625 13:57:52.193511 19227 ts_tablet_manager.cc:726] T 965f316ceb3d434f9c1255a72646a987 P b0eb3498fd904cdaaccb1b99282763e3: Tablet Copy: Invalid argument: Leader has replica of tablet 965f316ceb3d434f9c1255a72646a987 with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
W20250625 13:57:52.198334 19227 ts_tablet_manager.cc:726] T a0dfec98b40b41839827eb084e6a13b8 P b0eb3498fd904cdaaccb1b99282763e3: Tablet Copy: Invalid argument: Leader has replica of tablet a0dfec98b40b41839827eb084e6a13b8 with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
I20250625 13:57:52.202464 19227 ts_tablet_manager.cc:927] T ec8fd40a598d4c2cba01a560404670a3 P b0eb3498fd904cdaaccb1b99282763e3: Initiating tablet copy from peer 60ae629379bf4f0aa635f1244c320df5 (127.18.36.65:32959)
I20250625 13:57:52.203701 19227 tablet_copy_client.cc:323] T ec8fd40a598d4c2cba01a560404670a3 P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Beginning tablet copy session from remote peer at address 127.18.36.65:32959
I20250625 13:57:52.205157 18782 tablet_copy_service.cc:140] P 60ae629379bf4f0aa635f1244c320df5: Received BeginTabletCopySession request for tablet ec8fd40a598d4c2cba01a560404670a3 from peer b0eb3498fd904cdaaccb1b99282763e3 ({username='slave'} at 127.18.36.66:51389)
I20250625 13:57:52.205679 18782 tablet_copy_service.cc:161] P 60ae629379bf4f0aa635f1244c320df5: Beginning new tablet copy session on tablet ec8fd40a598d4c2cba01a560404670a3 from peer b0eb3498fd904cdaaccb1b99282763e3 at {username='slave'} at 127.18.36.66:51389: session id = b0eb3498fd904cdaaccb1b99282763e3-ec8fd40a598d4c2cba01a560404670a3
I20250625 13:57:52.212384 18782 tablet_copy_source_session.cc:215] T ec8fd40a598d4c2cba01a560404670a3 P 60ae629379bf4f0aa635f1244c320df5: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 13:57:52.215076 19227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8fd40a598d4c2cba01a560404670a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:52.225083 19227 tablet_copy_client.cc:806] T ec8fd40a598d4c2cba01a560404670a3 P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Starting download of 0 data blocks...
I20250625 13:57:52.225661 19227 tablet_copy_client.cc:670] T ec8fd40a598d4c2cba01a560404670a3 P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Starting download of 1 WAL segments...
I20250625 13:57:52.238395 19227 tablet_copy_client.cc:538] T ec8fd40a598d4c2cba01a560404670a3 P b0eb3498fd904cdaaccb1b99282763e3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 13:57:52.244998 19227 tablet_bootstrap.cc:492] T ec8fd40a598d4c2cba01a560404670a3 P b0eb3498fd904cdaaccb1b99282763e3: Bootstrap starting.
I20250625 13:57:53.434983 19227 tablet_bootstrap.cc:492] T ec8fd40a598d4c2cba01a560404670a3 P b0eb3498fd904cdaaccb1b99282763e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=211 overwritten=0 applied=211 ignored=0} inserts{seen=2593 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 13:57:53.435758 19227 tablet_bootstrap.cc:492] T ec8fd40a598d4c2cba01a560404670a3 P b0eb3498fd904cdaaccb1b99282763e3: Bootstrap complete.
I20250625 13:57:53.436412 19227 ts_tablet_manager.cc:1397] T ec8fd40a598d4c2cba01a560404670a3 P b0eb3498fd904cdaaccb1b99282763e3: Time spent bootstrapping tablet: real 1.192s user 1.147s sys 0.044s
I20250625 13:57:53.438424 19227 raft_consensus.cc:357] T ec8fd40a598d4c2cba01a560404670a3 P b0eb3498fd904cdaaccb1b99282763e3 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:53.438844 19227 raft_consensus.cc:738] T ec8fd40a598d4c2cba01a560404670a3 P b0eb3498fd904cdaaccb1b99282763e3 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: b0eb3498fd904cdaaccb1b99282763e3, State: Initialized, Role: NON_PARTICIPANT
I20250625 13:57:53.439381 19227 consensus_queue.cc:260] T ec8fd40a598d4c2cba01a560404670a3 P b0eb3498fd904cdaaccb1b99282763e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 211, Last appended: 1.211, Last appended by leader: 211, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ae629379bf4f0aa635f1244c320df5" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 32959 } }
I20250625 13:57:53.441211 19227 ts_tablet_manager.cc:1428] T ec8fd40a598d4c2cba01a560404670a3 P b0eb3498fd904cdaaccb1b99282763e3: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20250625 13:57:53.443397 18782 tablet_copy_service.cc:342] P 60ae629379bf4f0aa635f1244c320df5: Request end of tablet copy session b0eb3498fd904cdaaccb1b99282763e3-ec8fd40a598d4c2cba01a560404670a3 received from {username='slave'} at 127.18.36.66:51389
I20250625 13:57:53.443789 18782 tablet_copy_service.cc:434] P 60ae629379bf4f0aa635f1244c320df5: ending tablet copy session b0eb3498fd904cdaaccb1b99282763e3-ec8fd40a598d4c2cba01a560404670a3 on tablet ec8fd40a598d4c2cba01a560404670a3 with peer b0eb3498fd904cdaaccb1b99282763e3
W20250625 13:57:53.448752 19227 ts_tablet_manager.cc:726] T ec8fd40a598d4c2cba01a560404670a3 P b0eb3498fd904cdaaccb1b99282763e3: Tablet Copy: Invalid argument: Leader has replica of tablet ec8fd40a598d4c2cba01a560404670a3 with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
I20250625 13:57:53.453871 18577 tablet_copy-itest.cc:1252] Number of Service unavailable responses: 1338
I20250625 13:57:53.454192 18577 tablet_copy-itest.cc:1253] Number of in progress responses: 970
I20250625 13:57:53.457830 18577 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu with pid 18678
I20250625 13:57:53.506883 18577 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu with pid 19091
I20250625 13:57:53.538350 18577 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu with pid 18951
2025-06-25T13:57:53Z chronyd exiting
[ OK ] TabletCopyITest.TestTabletCopyThrottling (24826 ms)
[ RUN ] TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate
2025-06-25T13:57:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T13:57:53Z Disabled control of system clock
I20250625 13:57:53.638722 18577 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
/tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.36.126:46393
--webserver_interface=127.18.36.126
--webserver_port=0
--builtin_ntp_servers=127.18.36.84:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.36.126:46393 with env {}
W20250625 13:57:53.974491 19246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:53.975165 19246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:53.975687 19246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:54.009806 19246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 13:57:54.010203 19246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:54.010494 19246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 13:57:54.010752 19246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 13:57:54.050805 19246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.36.84:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.36.126:46393
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.36.126:46393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.36.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:54.052353 19246 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:54.054380 19246 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:54.073750 19253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:54.073748 19255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:54.073910 19252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:54.076668 19246 server_base.cc:1048] running on GCE node
I20250625 13:57:55.270615 19246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:55.273726 19246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:55.275251 19246 hybrid_clock.cc:648] HybridClock initialized: now 1750859875275201 us; error 45 us; skew 500 ppm
I20250625 13:57:55.276271 19246 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:55.287564 19246 webserver.cc:469] Webserver started at http://127.18.36.126:42821/ using document root <none> and password file <none>
I20250625 13:57:55.288758 19246 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:55.289047 19246 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:55.289647 19246 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:55.294718 19246 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/data/instance:
uuid: "e0ec4cfe37a146848145b7080ad192ce"
format_stamp: "Formatted at 2025-06-25 13:57:55 on dist-test-slave-690w"
I20250625 13:57:55.296080 19246 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/wal/instance:
uuid: "e0ec4cfe37a146848145b7080ad192ce"
format_stamp: "Formatted at 2025-06-25 13:57:55 on dist-test-slave-690w"
I20250625 13:57:55.304636 19246 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.004s
I20250625 13:57:55.311460 19262 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:55.312850 19246 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250625 13:57:55.313243 19246 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/data,/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/wal
uuid: "e0ec4cfe37a146848145b7080ad192ce"
format_stamp: "Formatted at 2025-06-25 13:57:55 on dist-test-slave-690w"
I20250625 13:57:55.313648 19246 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:55.389544 19246 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:55.391229 19246 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:55.391691 19246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:55.478926 19246 rpc_server.cc:307] RPC server started. Bound to: 127.18.36.126:46393
I20250625 13:57:55.479046 19313 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.36.126:46393 every 8 connection(s)
I20250625 13:57:55.482062 19246 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/data/info.pb
I20250625 13:57:55.486857 18577 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu as pid 19246
I20250625 13:57:55.487452 18577 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/master-0/wal/instance
I20250625 13:57:55.489560 19314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:57:55.519016 19314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce: Bootstrap starting.
I20250625 13:57:55.526278 19314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce: Neither blocks nor log segments found. Creating new log.
I20250625 13:57:55.528569 19314 log.cc:826] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce: Log is configured to *not* fsync() on all Append() calls
I20250625 13:57:55.533723 19314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce: No bootstrap required, opened a new log
I20250625 13:57:55.553597 19314 raft_consensus.cc:357] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ec4cfe37a146848145b7080ad192ce" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 46393 } }
I20250625 13:57:55.554418 19314 raft_consensus.cc:383] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:57:55.554752 19314 raft_consensus.cc:738] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ec4cfe37a146848145b7080ad192ce, State: Initialized, Role: FOLLOWER
I20250625 13:57:55.555752 19314 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ec4cfe37a146848145b7080ad192ce" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 46393 } }
I20250625 13:57:55.556380 19314 raft_consensus.cc:397] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 13:57:55.556691 19314 raft_consensus.cc:491] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 13:57:55.557019 19314 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:57:55.561684 19314 raft_consensus.cc:513] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ec4cfe37a146848145b7080ad192ce" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 46393 } }
I20250625 13:57:55.562741 19314 leader_election.cc:304] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0ec4cfe37a146848145b7080ad192ce; no voters:
I20250625 13:57:55.565007 19314 leader_election.cc:290] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 13:57:55.565765 19319 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:57:55.568440 19319 raft_consensus.cc:695] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [term 1 LEADER]: Becoming Leader. State: Replica: e0ec4cfe37a146848145b7080ad192ce, State: Running, Role: LEADER
I20250625 13:57:55.569418 19319 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ec4cfe37a146848145b7080ad192ce" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 46393 } }
I20250625 13:57:55.570482 19314 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [sys.catalog]: configured and running, proceeding with master startup.
I20250625 13:57:55.582259 19321 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0ec4cfe37a146848145b7080ad192ce. Latest consensus state: current_term: 1 leader_uuid: "e0ec4cfe37a146848145b7080ad192ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ec4cfe37a146848145b7080ad192ce" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 46393 } } }
I20250625 13:57:55.583348 19321 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [sys.catalog]: This master's current role is: LEADER
I20250625 13:57:55.588023 19320 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0ec4cfe37a146848145b7080ad192ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ec4cfe37a146848145b7080ad192ce" member_type: VOTER last_known_addr { host: "127.18.36.126" port: 46393 } } }
I20250625 13:57:55.588809 19320 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce [sys.catalog]: This master's current role is: LEADER
I20250625 13:57:55.590309 19329 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 13:57:55.603829 19329 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 13:57:55.622208 19329 catalog_manager.cc:1349] Generated new cluster ID: dec5c6d4643c4de2b34d8bbff6684759
I20250625 13:57:55.622506 19329 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 13:57:55.641949 19329 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 13:57:55.643594 19329 catalog_manager.cc:1506] Loading token signing keys...
I20250625 13:57:55.661674 19329 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e0ec4cfe37a146848145b7080ad192ce: Generated new TSK 0
I20250625 13:57:55.662793 19329 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 13:57:55.680184 18577 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
/tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.36.65:0
--local_ip_for_outbound_sockets=127.18.36.65
--webserver_interface=127.18.36.65
--webserver_port=0
--tserver_master_addrs=127.18.36.126:46393
--builtin_ntp_servers=127.18.36.84:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
W20250625 13:57:56.029577 19338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:56.030176 19338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:56.030639 19338 flags.cc:425] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20250625 13:57:56.030867 19338 flags.cc:425] Enabled unsafe flag: --enable_flush_memrowset=false
W20250625 13:57:56.031217 19338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:56.066586 19338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:56.067693 19338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.36.65
I20250625 13:57:56.108721 19338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.36.84:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.36.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.36.65
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.18.36.126:46393
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.36.65
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:56.110325 19338 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:56.112279 19338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:56.134136 19345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:56.136665 19347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:56.136087 19344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:57.561241 19346 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1424 milliseconds
I20250625 13:57:57.561482 19338 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:57:57.562865 19338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:57.566063 19338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:57.567780 19338 hybrid_clock.cc:648] HybridClock initialized: now 1750859877567711 us; error 74 us; skew 500 ppm
I20250625 13:57:57.568732 19338 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:57.578032 19338 webserver.cc:469] Webserver started at http://127.18.36.65:44709/ using document root <none> and password file <none>
I20250625 13:57:57.579195 19338 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:57.579427 19338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:57.579969 19338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:57.585016 19338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/data/instance:
uuid: "7bfb8feea378419aafa60abbf8ddcb7b"
format_stamp: "Formatted at 2025-06-25 13:57:57 on dist-test-slave-690w"
I20250625 13:57:57.586472 19338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/wal/instance:
uuid: "7bfb8feea378419aafa60abbf8ddcb7b"
format_stamp: "Formatted at 2025-06-25 13:57:57 on dist-test-slave-690w"
I20250625 13:57:57.595662 19338 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20250625 13:57:57.602804 19354 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:57.604275 19338 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20250625 13:57:57.604682 19338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/wal
uuid: "7bfb8feea378419aafa60abbf8ddcb7b"
format_stamp: "Formatted at 2025-06-25 13:57:57 on dist-test-slave-690w"
I20250625 13:57:57.605089 19338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:57.671205 19338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:57.673153 19338 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:57.673766 19338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:57.677309 19338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:57.682937 19338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:57.683212 19338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:57.683481 19338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:57.683645 19338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:57.906049 19338 rpc_server.cc:307] RPC server started. Bound to: 127.18.36.65:41563
I20250625 13:57:57.906283 19466 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.36.65:41563 every 8 connection(s)
I20250625 13:57:57.909332 19338 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/data/info.pb
I20250625 13:57:57.919250 18577 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu as pid 19338
I20250625 13:57:57.919749 18577 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-0/wal/instance
I20250625 13:57:57.926643 18577 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
/tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.36.66:0
--local_ip_for_outbound_sockets=127.18.36.66
--webserver_interface=127.18.36.66
--webserver_port=0
--tserver_master_addrs=127.18.36.126:46393
--builtin_ntp_servers=127.18.36.84:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
I20250625 13:57:57.941205 19467 heartbeater.cc:344] Connected to a master server at 127.18.36.126:46393
I20250625 13:57:57.941895 19467 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:57.943535 19467 heartbeater.cc:507] Master 127.18.36.126:46393 requested a full tablet report, sending...
I20250625 13:57:57.948151 19279 ts_manager.cc:194] Registered new tserver with Master: 7bfb8feea378419aafa60abbf8ddcb7b (127.18.36.65:41563)
I20250625 13:57:57.952446 19279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.36.65:47835
W20250625 13:57:58.278458 19471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:57:58.279040 19471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:57:58.279536 19471 flags.cc:425] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20250625 13:57:58.279727 19471 flags.cc:425] Enabled unsafe flag: --enable_flush_memrowset=false
W20250625 13:57:58.280139 19471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:57:58.315979 19471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:57:58.317679 19471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.36.66
I20250625 13:57:58.357983 19471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.36.84:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.36.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.36.66
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.18.36.126:46393
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.36.66
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:57:58.359421 19471 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:57:58.362063 19471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:57:58.382117 19477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:57:58.958479 19467 heartbeater.cc:499] Master 127.18.36.126:46393 was elected leader, sending a full tablet report...
W20250625 13:57:58.383826 19480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:58.382606 19478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:57:59.592803 19479 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 13:57:59.592862 19471 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:57:59.598363 19471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:57:59.601491 19471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:57:59.603039 19471 hybrid_clock.cc:648] HybridClock initialized: now 1750859879602985 us; error 70 us; skew 500 ppm
I20250625 13:57:59.603994 19471 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:57:59.611649 19471 webserver.cc:469] Webserver started at http://127.18.36.66:34527/ using document root <none> and password file <none>
I20250625 13:57:59.612764 19471 fs_manager.cc:362] Metadata directory not provided
I20250625 13:57:59.613013 19471 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:57:59.613582 19471 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:57:59.618943 19471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/data/instance:
uuid: "935a53707abf4076933a524d32ff50f5"
format_stamp: "Formatted at 2025-06-25 13:57:59 on dist-test-slave-690w"
I20250625 13:57:59.620478 19471 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/wal/instance:
uuid: "935a53707abf4076933a524d32ff50f5"
format_stamp: "Formatted at 2025-06-25 13:57:59 on dist-test-slave-690w"
I20250625 13:57:59.629503 19471 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.011s sys 0.000s
I20250625 13:57:59.636243 19488 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:59.637965 19471 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20250625 13:57:59.638456 19471 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/wal
uuid: "935a53707abf4076933a524d32ff50f5"
format_stamp: "Formatted at 2025-06-25 13:57:59 on dist-test-slave-690w"
I20250625 13:57:59.638840 19471 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:57:59.703783 19471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:57:59.705585 19471 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:57:59.706094 19471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:57:59.709020 19471 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:57:59.713799 19471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:57:59.714053 19471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:59.714401 19471 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:57:59.714591 19471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:57:59.880879 19471 rpc_server.cc:307] RPC server started. Bound to: 127.18.36.66:32807
I20250625 13:57:59.881036 19600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.36.66:32807 every 8 connection(s)
I20250625 13:57:59.883828 19471 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/data/info.pb
I20250625 13:57:59.891080 18577 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu as pid 19471
I20250625 13:57:59.891688 18577 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-1/wal/instance
I20250625 13:57:59.901324 18577 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
/tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.36.67:0
--local_ip_for_outbound_sockets=127.18.36.67
--webserver_interface=127.18.36.67
--webserver_port=0
--tserver_master_addrs=127.18.36.126:46393
--builtin_ntp_servers=127.18.36.84:41361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
I20250625 13:57:59.915836 19601 heartbeater.cc:344] Connected to a master server at 127.18.36.126:46393
I20250625 13:57:59.916795 19601 heartbeater.cc:461] Registering TS with master...
I20250625 13:57:59.918658 19601 heartbeater.cc:507] Master 127.18.36.126:46393 requested a full tablet report, sending...
I20250625 13:57:59.921864 19279 ts_manager.cc:194] Registered new tserver with Master: 935a53707abf4076933a524d32ff50f5 (127.18.36.66:32807)
I20250625 13:57:59.923316 19279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.36.66:33519
W20250625 13:58:00.254451 19605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 13:58:00.255057 19605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 13:58:00.255553 19605 flags.cc:425] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20250625 13:58:00.255752 19605 flags.cc:425] Enabled unsafe flag: --enable_flush_memrowset=false
W20250625 13:58:00.256155 19605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 13:58:00.291621 19605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 13:58:00.292778 19605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.36.67
I20250625 13:58:00.332808 19605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.36.84:41361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.36.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.36.67
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.18.36.126:46393
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.36.67
--log_dir=/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 13:43:23 UTC on 5fd53c4cbb9d
build id 6765
TSAN enabled
I20250625 13:58:00.334439 19605 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 13:58:00.336407 19605 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 13:58:00.357532 19611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 13:58:00.928453 19601 heartbeater.cc:499] Master 127.18.36.126:46393 was elected leader, sending a full tablet report...
W20250625 13:58:00.357617 19614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:00.359087 19612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 13:58:01.578225 19613 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 13:58:01.578337 19605 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 13:58:01.582897 19605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 13:58:01.585558 19605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 13:58:01.587021 19605 hybrid_clock.cc:648] HybridClock initialized: now 1750859881586969 us; error 65 us; skew 500 ppm
I20250625 13:58:01.588076 19605 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 13:58:01.600826 19605 webserver.cc:469] Webserver started at http://127.18.36.67:40649/ using document root <none> and password file <none>
I20250625 13:58:01.602118 19605 fs_manager.cc:362] Metadata directory not provided
I20250625 13:58:01.602380 19605 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 13:58:01.603004 19605 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 13:58:01.608469 19605 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/data/instance:
uuid: "65707482989741b7ba8973d9cf8c7d7c"
format_stamp: "Formatted at 2025-06-25 13:58:01 on dist-test-slave-690w"
I20250625 13:58:01.609920 19605 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/wal/instance:
uuid: "65707482989741b7ba8973d9cf8c7d7c"
format_stamp: "Formatted at 2025-06-25 13:58:01 on dist-test-slave-690w"
I20250625 13:58:01.618726 19605 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20250625 13:58:01.626060 19621 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:01.627421 19605 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20250625 13:58:01.627836 19605 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/wal
uuid: "65707482989741b7ba8973d9cf8c7d7c"
format_stamp: "Formatted at 2025-06-25 13:58:01 on dist-test-slave-690w"
I20250625 13:58:01.628232 19605 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 13:58:01.690558 19605 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 13:58:01.692245 19605 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 13:58:01.692777 19605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 13:58:01.696045 19605 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 13:58:01.700901 19605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 13:58:01.701174 19605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:01.701509 19605 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 13:58:01.701683 19605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 13:58:01.868249 19605 rpc_server.cc:307] RPC server started. Bound to: 127.18.36.67:42459
I20250625 13:58:01.868389 19733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.36.67:42459 every 8 connection(s)
I20250625 13:58:01.871218 19605 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/data/info.pb
I20250625 13:58:01.879607 18577 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu as pid 19605
I20250625 13:58:01.880311 18577 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0/minicluster-data/ts-2/wal/instance
I20250625 13:58:01.904286 19734 heartbeater.cc:344] Connected to a master server at 127.18.36.126:46393
I20250625 13:58:01.904911 19734 heartbeater.cc:461] Registering TS with master...
I20250625 13:58:01.906162 19734 heartbeater.cc:507] Master 127.18.36.126:46393 requested a full tablet report, sending...
I20250625 13:58:01.908793 19279 ts_manager.cc:194] Registered new tserver with Master: 65707482989741b7ba8973d9cf8c7d7c (127.18.36.67:42459)
I20250625 13:58:01.910820 19279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.36.67:34609
I20250625 13:58:01.922104 18577 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 13:58:01.960564 18577 test_util.cc:276] Using random seed: -716123054
I20250625 13:58:02.011152 19279 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42380:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 13:58:02.014662 19279 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 13:58:02.077194 19669 tablet_service.cc:1468] Processing CreateTablet for tablet 17c5459006ae456ca6496d88d53e799e (DEFAULT_TABLE table=test-workload [id=8b6a552b72bd43aebfe059ff0b3b4934]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:58:02.079322 19669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17c5459006ae456ca6496d88d53e799e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:02.079090 19402 tablet_service.cc:1468] Processing CreateTablet for tablet 17c5459006ae456ca6496d88d53e799e (DEFAULT_TABLE table=test-workload [id=8b6a552b72bd43aebfe059ff0b3b4934]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:58:02.079015 19536 tablet_service.cc:1468] Processing CreateTablet for tablet 17c5459006ae456ca6496d88d53e799e (DEFAULT_TABLE table=test-workload [id=8b6a552b72bd43aebfe059ff0b3b4934]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 13:58:02.081213 19536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17c5459006ae456ca6496d88d53e799e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:02.081207 19402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17c5459006ae456ca6496d88d53e799e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 13:58:02.107281 19758 tablet_bootstrap.cc:492] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c: Bootstrap starting.
I20250625 13:58:02.116336 19759 tablet_bootstrap.cc:492] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5: Bootstrap starting.
I20250625 13:58:02.118093 19758 tablet_bootstrap.cc:654] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c: Neither blocks nor log segments found. Creating new log.
I20250625 13:58:02.121127 19758 log.cc:826] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:02.123010 19760 tablet_bootstrap.cc:492] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b: Bootstrap starting.
I20250625 13:58:02.126753 19759 tablet_bootstrap.cc:654] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5: Neither blocks nor log segments found. Creating new log.
I20250625 13:58:02.129621 19759 log.cc:826] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:02.130667 19758 tablet_bootstrap.cc:492] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c: No bootstrap required, opened a new log
I20250625 13:58:02.131476 19758 ts_tablet_manager.cc:1397] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.002s
I20250625 13:58:02.131547 19760 tablet_bootstrap.cc:654] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b: Neither blocks nor log segments found. Creating new log.
I20250625 13:58:02.134781 19760 log.cc:826] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b: Log is configured to *not* fsync() on all Append() calls
I20250625 13:58:02.137087 19759 tablet_bootstrap.cc:492] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5: No bootstrap required, opened a new log
I20250625 13:58:02.137890 19759 ts_tablet_manager.cc:1397] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.008s
I20250625 13:58:02.141849 19760 tablet_bootstrap.cc:492] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b: No bootstrap required, opened a new log
I20250625 13:58:02.142544 19760 ts_tablet_manager.cc:1397] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.008s
I20250625 13:58:02.164098 19758 raft_consensus.cc:357] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 41563 } } peers { permanent_uuid: "65707482989741b7ba8973d9cf8c7d7c" member_type: VOTER last_known_addr { host: "127.18.36.67" port: 42459 } } peers { permanent_uuid: "935a53707abf4076933a524d32ff50f5" member_type: VOTER last_known_addr { host: "127.18.36.66" port: 32807 } }
I20250625 13:58:02.165280 19758 raft_consensus.cc:383] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:58:02.165642 19758 raft_consensus.cc:738] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65707482989741b7ba8973d9cf8c7d7c, State: Initialized, Role: FOLLOWER
I20250625 13:58:02.165351 19760 raft_consensus.cc:357] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 41563 } } peers { permanent_uuid: "65707482989741b7ba8973d9cf8c7d7c" member_type: VOTER last_known_addr { host: "127.18.36.67" port: 42459 } } peers { permanent_uuid: "935a53707abf4076933a524d32ff50f5" member_type: VOTER last_known_addr { host: "127.18.36.66" port: 32807 } }
I20250625 13:58:02.166414 19760 raft_consensus.cc:383] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:58:02.166733 19760 raft_consensus.cc:738] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bfb8feea378419aafa60abbf8ddcb7b, State: Initialized, Role: FOLLOWER
I20250625 13:58:02.166663 19758 consensus_queue.cc:260] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 41563 } } peers { permanent_uuid: "65707482989741b7ba8973d9cf8c7d7c" member_type: VOTER last_known_addr { host: "127.18.36.67" port: 42459 } } peers { permanent_uuid: "935a53707abf4076933a524d32ff50f5" member_type: VOTER last_known_addr { host: "127.18.36.66" port: 32807 } }
I20250625 13:58:02.167591 19760 consensus_queue.cc:260] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 41563 } } peers { permanent_uuid: "65707482989741b7ba8973d9cf8c7d7c" member_type: VOTER last_known_addr { host: "127.18.36.67" port: 42459 } } peers { permanent_uuid: "935a53707abf4076933a524d32ff50f5" member_type: VOTER last_known_addr { host: "127.18.36.66" port: 32807 } }
I20250625 13:58:02.170420 19734 heartbeater.cc:499] Master 127.18.36.126:46393 was elected leader, sending a full tablet report...
I20250625 13:58:02.171536 19758 ts_tablet_manager.cc:1428] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c: Time spent starting tablet: real 0.040s user 0.033s sys 0.004s
I20250625 13:58:02.169385 19759 raft_consensus.cc:357] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 41563 } } peers { permanent_uuid: "65707482989741b7ba8973d9cf8c7d7c" member_type: VOTER last_known_addr { host: "127.18.36.67" port: 42459 } } peers { permanent_uuid: "935a53707abf4076933a524d32ff50f5" member_type: VOTER last_known_addr { host: "127.18.36.66" port: 32807 } }
I20250625 13:58:02.172282 19759 raft_consensus.cc:383] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 13:58:02.172696 19759 raft_consensus.cc:738] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935a53707abf4076933a524d32ff50f5, State: Initialized, Role: FOLLOWER
I20250625 13:58:02.174798 19760 ts_tablet_manager.cc:1428] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b: Time spent starting tablet: real 0.032s user 0.018s sys 0.012s
I20250625 13:58:02.174357 19759 consensus_queue.cc:260] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 41563 } } peers { permanent_uuid: "65707482989741b7ba8973d9cf8c7d7c" member_type: VOTER last_known_addr { host: "127.18.36.67" port: 42459 } } peers { permanent_uuid: "935a53707abf4076933a524d32ff50f5" member_type: VOTER last_known_addr { host: "127.18.36.66" port: 32807 } }
I20250625 13:58:02.179626 19759 ts_tablet_manager.cc:1428] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5: Time spent starting tablet: real 0.041s user 0.026s sys 0.011s
W20250625 13:58:02.377394 19735 tablet.cc:2378] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 13:58:02.378041 19735 tablet_replica_mm_ops.cc:240] Deltamemstore flush is disabled (check --enable_flush_deltamemstores)
W20250625 13:58:02.378300 19735 tablet_replica_mm_ops.cc:163] Memrowset flush is disabled (check --enable_flush_memrowset)
W20250625 13:58:02.392163 19602 tablet.cc:2378] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 13:58:02.392971 19602 tablet_replica_mm_ops.cc:240] Deltamemstore flush is disabled (check --enable_flush_deltamemstores)
W20250625 13:58:02.393169 19602 tablet_replica_mm_ops.cc:163] Memrowset flush is disabled (check --enable_flush_memrowset)
W20250625 13:58:02.421090 19468 tablet.cc:2378] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 13:58:02.421725 19468 tablet_replica_mm_ops.cc:240] Deltamemstore flush is disabled (check --enable_flush_deltamemstores)
W20250625 13:58:02.421931 19468 tablet_replica_mm_ops.cc:163] Memrowset flush is disabled (check --enable_flush_memrowset)
I20250625 13:58:02.511981 19765 raft_consensus.cc:491] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:58:02.512602 19765 raft_consensus.cc:513] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 41563 } } peers { permanent_uuid: "65707482989741b7ba8973d9cf8c7d7c" member_type: VOTER last_known_addr { host: "127.18.36.67" port: 42459 } } peers { permanent_uuid: "935a53707abf4076933a524d32ff50f5" member_type: VOTER last_known_addr { host: "127.18.36.66" port: 32807 } }
I20250625 13:58:02.515563 19765 leader_election.cc:290] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65707482989741b7ba8973d9cf8c7d7c (127.18.36.67:42459), 935a53707abf4076933a524d32ff50f5 (127.18.36.66:32807)
I20250625 13:58:02.528075 19766 raft_consensus.cc:491] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:58:02.528834 19766 raft_consensus.cc:513] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 41563 } } peers { permanent_uuid: "65707482989741b7ba8973d9cf8c7d7c" member_type: VOTER last_known_addr { host: "127.18.36.67" port: 42459 } } peers { permanent_uuid: "935a53707abf4076933a524d32ff50f5" member_type: VOTER last_known_addr { host: "127.18.36.66" port: 32807 } }
I20250625 13:58:02.532913 19764 raft_consensus.cc:491] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 13:58:02.533179 19689 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17c5459006ae456ca6496d88d53e799e" candidate_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65707482989741b7ba8973d9cf8c7d7c" is_pre_election: true
I20250625 13:58:02.533614 19764 raft_consensus.cc:513] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 41563 } } peers { permanent_uuid: "65707482989741b7ba8973d9cf8c7d7c" member_type: VOTER last_known_addr { host: "127.18.36.67" port: 42459 } } peers { permanent_uuid: "935a53707abf4076933a524d32ff50f5" member_type: VOTER last_known_addr { host: "127.18.36.66" port: 32807 } }
I20250625 13:58:02.535048 19689 raft_consensus.cc:2466] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bfb8feea378419aafa60abbf8ddcb7b in term 0.
I20250625 13:58:02.532769 19766 leader_election.cc:290] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bfb8feea378419aafa60abbf8ddcb7b (127.18.36.65:41563), 65707482989741b7ba8973d9cf8c7d7c (127.18.36.67:42459)
I20250625 13:58:02.543604 19764 leader_election.cc:290] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7bfb8feea378419aafa60abbf8ddcb7b (127.18.36.65:41563), 935a53707abf4076933a524d32ff50f5 (127.18.36.66:32807)
I20250625 13:58:02.554705 19556 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17c5459006ae456ca6496d88d53e799e" candidate_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935a53707abf4076933a524d32ff50f5" is_pre_election: true
I20250625 13:58:02.555787 19556 raft_consensus.cc:2466] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bfb8feea378419aafa60abbf8ddcb7b in term 0.
I20250625 13:58:02.557756 19356 leader_election.cc:304] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bfb8feea378419aafa60abbf8ddcb7b, 935a53707abf4076933a524d32ff50f5; no voters:
I20250625 13:58:02.559128 19765 raft_consensus.cc:2802] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 13:58:02.559608 19765 raft_consensus.cc:491] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 13:58:02.559953 19765 raft_consensus.cc:3058] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:58:02.569051 19765 raft_consensus.cc:513] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 41563 } } peers { permanent_uuid: "65707482989741b7ba8973d9cf8c7d7c" member_type: VOTER last_known_addr { host: "127.18.36.67" port: 42459 } } peers { permanent_uuid: "935a53707abf4076933a524d32ff50f5" member_type: VOTER last_known_addr { host: "127.18.36.66" port: 32807 } }
I20250625 13:58:02.574776 19422 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17c5459006ae456ca6496d88d53e799e" candidate_uuid: "935a53707abf4076933a524d32ff50f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" is_pre_election: true
I20250625 13:58:02.575634 19556 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17c5459006ae456ca6496d88d53e799e" candidate_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935a53707abf4076933a524d32ff50f5"
I20250625 13:58:02.576113 19422 raft_consensus.cc:2391] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 935a53707abf4076933a524d32ff50f5 in current term 1: Already voted for candidate 7bfb8feea378419aafa60abbf8ddcb7b in this term.
I20250625 13:58:02.576328 19556 raft_consensus.cc:3058] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5 [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:58:02.577795 19421 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17c5459006ae456ca6496d88d53e799e" candidate_uuid: "65707482989741b7ba8973d9cf8c7d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" is_pre_election: true
I20250625 13:58:02.578469 19689 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17c5459006ae456ca6496d88d53e799e" candidate_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65707482989741b7ba8973d9cf8c7d7c"
I20250625 13:58:02.578871 19421 raft_consensus.cc:2391] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65707482989741b7ba8973d9cf8c7d7c in current term 1: Already voted for candidate 7bfb8feea378419aafa60abbf8ddcb7b in this term.
I20250625 13:58:02.578847 19765 leader_election.cc:290] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [CANDIDATE]: Term 1 election: Requested vote from peers 65707482989741b7ba8973d9cf8c7d7c (127.18.36.67:42459), 935a53707abf4076933a524d32ff50f5 (127.18.36.66:32807)
I20250625 13:58:02.579365 19689 raft_consensus.cc:3058] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c [term 0 FOLLOWER]: Advancing to term 1
I20250625 13:58:02.585525 19555 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17c5459006ae456ca6496d88d53e799e" candidate_uuid: "65707482989741b7ba8973d9cf8c7d7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935a53707abf4076933a524d32ff50f5" is_pre_election: true
I20250625 13:58:02.586807 19556 raft_consensus.cc:2466] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bfb8feea378419aafa60abbf8ddcb7b in term 1.
I20250625 13:58:02.587908 19689 raft_consensus.cc:2466] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bfb8feea378419aafa60abbf8ddcb7b in term 1.
I20250625 13:58:02.588135 19623 leader_election.cc:304] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65707482989741b7ba8973d9cf8c7d7c; no voters: 7bfb8feea378419aafa60abbf8ddcb7b, 935a53707abf4076933a524d32ff50f5
I20250625 13:58:02.588491 19356 leader_election.cc:304] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bfb8feea378419aafa60abbf8ddcb7b, 935a53707abf4076933a524d32ff50f5; no voters:
I20250625 13:58:02.588634 19688 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17c5459006ae456ca6496d88d53e799e" candidate_uuid: "935a53707abf4076933a524d32ff50f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65707482989741b7ba8973d9cf8c7d7c" is_pre_election: true
I20250625 13:58:02.589569 19765 raft_consensus.cc:2802] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [term 1 FOLLOWER]: Leader election won for term 1
I20250625 13:58:02.589712 19688 raft_consensus.cc:2391] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 935a53707abf4076933a524d32ff50f5 in current term 1: Already voted for candidate 7bfb8feea378419aafa60abbf8ddcb7b in this term.
I20250625 13:58:02.590215 19764 raft_consensus.cc:2747] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:58:02.591054 19491 leader_election.cc:304] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 935a53707abf4076933a524d32ff50f5; no voters: 65707482989741b7ba8973d9cf8c7d7c, 7bfb8feea378419aafa60abbf8ddcb7b
I20250625 13:58:02.591902 19766 raft_consensus.cc:2747] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 13:58:02.592383 19765 raft_consensus.cc:695] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [term 1 LEADER]: Becoming Leader. State: Replica: 7bfb8feea378419aafa60abbf8ddcb7b, State: Running, Role: LEADER
I20250625 13:58:02.593472 19765 consensus_queue.cc:237] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 41563 } } peers { permanent_uuid: "65707482989741b7ba8973d9cf8c7d7c" member_type: VOTER last_known_addr { host: "127.18.36.67" port: 42459 } } peers { permanent_uuid: "935a53707abf4076933a524d32ff50f5" member_type: VOTER last_known_addr { host: "127.18.36.66" port: 32807 } }
I20250625 13:58:02.606411 19276 catalog_manager.cc:5582] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bfb8feea378419aafa60abbf8ddcb7b (127.18.36.65). New cstate: current_term: 1 leader_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bfb8feea378419aafa60abbf8ddcb7b" member_type: VOTER last_known_addr { host: "127.18.36.65" port: 41563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65707482989741b7ba8973d9cf8c7d7c" member_type: VOTER last_known_addr { host: "127.18.36.67" port: 42459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "935a53707abf4076933a524d32ff50f5" member_type: VOTER last_known_addr { host: "127.18.36.66" port: 32807 } health_report { overall_health: UNKNOWN } } }
I20250625 13:58:02.699903 19556 raft_consensus.cc:1273] T 17c5459006ae456ca6496d88d53e799e P 935a53707abf4076933a524d32ff50f5 [term 1 FOLLOWER]: Refusing update from remote peer 7bfb8feea378419aafa60abbf8ddcb7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:58:02.699903 19688 raft_consensus.cc:1273] T 17c5459006ae456ca6496d88d53e799e P 65707482989741b7ba8973d9cf8c7d7c [term 1 FOLLOWER]: Refusing update from remote peer 7bfb8feea378419aafa60abbf8ddcb7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 13:58:02.701889 19765 consensus_queue.cc:1035] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "65707482989741b7ba8973d9cf8c7d7c" member_type: VOTER last_known_addr { host: "127.18.36.67" port: 42459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:58:02.702715 19774 consensus_queue.cc:1035] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "935a53707abf4076933a524d32ff50f5" member_type: VOTER last_known_addr { host: "127.18.36.66" port: 32807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 13:58:02.726253 19781 mvcc.cc:204] Tried to move back new op lower bound from 7171522079519293440 to 7171522079111155712. Current Snapshot: MvccSnapshot[applied={T|T < 7171522079519293440}]
I20250625 13:58:02.795243 19421 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "17c5459006ae456ca6496d88d53e799e"
dest_uuid: "7bfb8feea378419aafa60abbf8ddcb7b"
from {username='slave'} at 127.0.0.1:44532
I20250625 13:58:02.795853 19421 raft_consensus.cc:474] T 17c5459006ae456ca6496d88d53e799e P 7bfb8feea378419aafa60abbf8ddcb7b [term 1 LEADER]: Not starting forced leader election -- already a leader
W20250625 13:58:18.062855 19793 meta_cache.cc:1261] Time spent looking up entry by key: real 0.071s user 0.002s sys 0.000s
W20250625 13:58:49.967778 19792 meta_cache.cc:1261] Time spent looking up entry by key: real 0.050s user 0.000s sys 0.000s
W20250625 13:58:58.177498 19310 debug-util.cc:398] Leaking SignalData structure 0x7b08000972c0 after lost signal to thread 19247
W20250625 13:58:58.178366 19310 debug-util.cc:398] Leaking SignalData structure 0x7b08000a13a0 after lost signal to thread 19313
W20250625 13:58:58.475787 19797 meta_cache.cc:1261] Time spent looking up entry by key: real 0.062s user 0.003s sys 0.000s
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:2151: Failure
Failed
Bad status: Timed out: Timed out waiting for number of WAL segments on tablet 17c5459006ae456ca6496d88d53e799e on TS 0 to be 6. Found 5
I20250625 13:59:03.003201 18577 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20250625 13:59:03.007987 18577 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 7bfb8feea378419aafa60abbf8ddcb7b and pid 19338
************************ BEGIN STACKS **************************
[New LWP 19339]
[New LWP 19340]
[New LWP 19341]
[New LWP 19342]
[New LWP 19343]
[New LWP 19350]
[New LWP 19351]
[New LWP 19352]
[New LWP 19355]
[New LWP 19356]
[New LWP 19357]
[New LWP 19358]
[New LWP 19359]
[New LWP 19360]
[New LWP 19361]
[New LWP 19362]
[New LWP 19363]
[New LWP 19364]
[New LWP 19365]
[New LWP 19366]
[New LWP 19367]
[New LWP 19368]
[New LWP 19369]
[New LWP 19370]
[New LWP 19371]
[New LWP 19372]
[New LWP 19373]
[New LWP 19374]
[New LWP 19375]
[New LWP 19376]
[New LWP 19377]
[New LWP 19378]
[New LWP 19379]
[New LWP 19380]
[New LWP 19381]
[New LWP 19382]
[New LWP 19383]
[New LWP 19384]
[New LWP 19385]
[New LWP 19386]
[New LWP 19387]
[New LWP 19388]
[New LWP 19389]
[New LWP 19390]
[New LWP 19391]
[New LWP 19392]
[New LWP 19393]
[New LWP 19394]
[New LWP 19395]
[New LWP 19396]
[New LWP 19397]
[New LWP 19398]
[New LWP 19399]
[New LWP 19400]
[New LWP 19401]
[New LWP 19402]
[New LWP 19403]
[New LWP 19404]
[New LWP 19405]
[New LWP 19406]
[New LWP 19407]
[New LWP 19408]
[New LWP 19409]
[New LWP 19410]
[New LWP 19411]
[New LWP 19412]
[New LWP 19413]
[New LWP 19414]
[New LWP 19415]
[New LWP 19416]
[New LWP 19417]
[New LWP 19418]
[New LWP 19419]
[New LWP 19420]
[New LWP 19421]
[New LWP 19422]
[New LWP 19423]
[New LWP 19424]
[New LWP 19425]
[New LWP 19426]
[New LWP 19427]
[New LWP 19428]
[New LWP 19429]
[New LWP 19430]
[New LWP 19431]
[New LWP 19432]
[New LWP 19433]
[New LWP 19434]
[New LWP 19435]
[New LWP 19436]
[New LWP 19437]
[New LWP 19438]
[New LWP 19439]
[New LWP 19440]
[New LWP 19441]
[New LWP 19442]
[New LWP 19443]
[New LWP 19444]
[New LWP 19445]
[New LWP 19446]
[New LWP 19447]
[New LWP 19448]
[New LWP 19449]
[New LWP 19450]
[New LWP 19451]
[New LWP 19452]
[New LWP 19453]
[New LWP 19454]
[New LWP 19455]
[New LWP 19456]
[New LWP 19457]
[New LWP 19458]
[New LWP 19459]
[New LWP 19460]
[New LWP 19461]
[New LWP 19462]
[New LWP 19463]
[New LWP 19464]
[New LWP 19465]
[New LWP 19466]
[New LWP 19467]
[New LWP 19468]
[New LWP 19951]
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
0x00007fa5c9cd6d50 in ?? ()
Id Target Id Frame
* 1 LWP 19338 "kudu" 0x00007fa5c9cd6d50 in ?? ()
2 LWP 19339 "kudu" 0x00007fa5c509a7a0 in ?? ()
3 LWP 19340 "kudu" 0x00007fa5c9cd2fb9 in ?? ()
4 LWP 19341 "kudu" 0x00007fa5c9cd2fb9 in ?? ()
5 LWP 19342 "kudu" 0x00007fa5c9cd2fb9 in ?? ()
6 LWP 19343 "kernel-watcher-" 0x00007fa5c9cd2fb9 in ?? ()
7 LWP 19350 "ntp client-1935" 0x00007fa5c9cd69e2 in ?? ()
8 LWP 19351 "file cache-evic" 0x00007fa5c9cd2fb9 in ?? ()
9 LWP 19352 "sq_acceptor" 0x00007fa5c50cacb9 in ?? ()
10 LWP 19355 "rpc reactor-193" 0x00007fa5c50d7a47 in ?? ()
11 LWP 19356 "rpc reactor-193" 0x00007fa5c50d7a47 in ?? ()
12 LWP 19357 "rpc reactor-193" 0x00007fa5c50d7a47 in ?? ()
13 LWP 19358 "rpc reactor-193" 0x00007fa5c50d7a47 in ?? ()
14 LWP 19359 "MaintenanceMgr " 0x00007fa5c9cd2ad3 in ?? ()
15 LWP 19360 "txn-status-mana" 0x00007fa5c9cd2fb9 in ?? ()
16 LWP 19361 "collect_and_rem" 0x00007fa5c9cd2fb9 in ?? ()
17 LWP 19362 "tc-session-exp-" 0x00007fa5c9cd2fb9 in ?? ()
18 LWP 19363 "rpc worker-1936" 0x00007fa5c9cd2ad3 in ?? ()
19 LWP 19364 "rpc worker-1936" 0x00007fa5c9cd2ad3 in ?? ()
20 LWP 19365 "rpc worker-1936" 0x00007fa5c9cd2ad3 in ?? ()
21 LWP 19366 "rpc worker-1936" 0x00007fa5c9cd2ad3 in ?? ()
22 LWP 19367 "rpc worker-1936" 0x00007fa5c9cd2ad3 in ?? ()
23 LWP 19368 "rpc worker-1936" 0x00007fa5c9cd2ad3 in ?? ()
24 LWP 19369 "rpc worker-1936" 0x00007fa5c9cd2ad3 in ?? ()
25 LWP 19370 "rpc worker-1937" 0x00007fa5c9cd2ad3 in ?? ()
26 LWP 19371 "rpc worker-1937" 0x00007fa5c9cd2ad3 in ?? ()
27 LWP 19372 "rpc worker-1937" 0x00007fa5c9cd2ad3 in ?? ()
28 LWP 19373 "rpc worker-1937" 0x00007fa5c9cd2ad3 in ?? ()
29 LWP 19374 "rpc worker-1937" 0x00007fa5c9cd2ad3 in ?? ()
30 LWP 19375 "rpc worker-1937" 0x00007fa5c9cd2ad3 in ?? ()
31 LWP 19376 "rpc worker-1937" 0x00007fa5c9cd2ad3 in ?? ()
32 LWP 19377 "rpc worker-1937" 0x00007fa5c9cd2ad3 in ?? ()
33 LWP 19378 "rpc worker-1937" 0x00007fa5c9cd2ad3 in ?? ()
34 LWP 19379 "rpc worker-1937" 0x00007fa5c9cd2ad3 in ?? ()
35 LWP 19380 "rpc worker-1938" 0x00007fa5c9cd2ad3 in ?? ()
36 LWP 19381 "rpc worker-1938" 0x00007fa5c9cd2ad3 in ?? ()
37 LWP 19382 "rpc worker-1938" 0x00007fa5c9cd2ad3 in ?? ()
38 LWP 19383 "rpc worker-1938" 0x00007fa5c9cd2ad3 in ?? ()
39 LWP 19384 "rpc worker-1938" 0x00007fa5c9cd2ad3 in ?? ()
40 LWP 19385 "rpc worker-1938" 0x00007fa5c9cd2ad3 in ?? ()
41 LWP 19386 "rpc worker-1938" 0x00007fa5c9cd2ad3 in ?? ()
42 LWP 19387 "rpc worker-1938" 0x00007fa5c9cd2ad3 in ?? ()
43 LWP 19388 "rpc worker-1938" 0x00007fa5c9cd2ad3 in ?? ()
44 LWP 19389 "rpc worker-1938" 0x00007fa5c9cd2ad3 in ?? ()
45 LWP 19390 "rpc worker-1939" 0x00007fa5c9cd2ad3 in ?? ()
46 LWP 19391 "rpc worker-1939" 0x00007fa5c9cd2ad3 in ?? ()
47 LWP 19392 "rpc worker-1939" 0x00007fa5c9cd2ad3 in ?? ()
48 LWP 19393 "rpc worker-1939" 0x00007fa5c9cd2ad3 in ?? ()
49 LWP 19394 "rpc worker-1939" 0x00007fa5c9cd2ad3 in ?? ()
50 LWP 19395 "rpc worker-1939" 0x00007fa5c9cd2ad3 in ?? ()
51 LWP 19396 "rpc worker-1939" 0x00007fa5c9cd2ad3 in ?? ()
52 LWP 19397 "rpc worker-1939" 0x00007fa5c9cd2ad3 in ?? ()
53 LWP 19398 "rpc worker-1939" 0x00007fa5c9cd2ad3 in ?? ()
54 LWP 19399 "rpc worker-1939" 0x00007fa5c9cd2ad3 in ?? ()
55 LWP 19400 "rpc worker-1940" 0x00007fa5c9cd2ad3 in ?? ()
56 LWP 19401 "rpc worker-1940" 0x00007fa5c9cd2ad3 in ?? ()
57 LWP 19402 "rpc worker-1940" 0x00007fa5c9cd2ad3 in ?? ()
58 LWP 19403 "rpc worker-1940" 0x00007fa5c9cd2ad3 in ?? ()
59 LWP 19404 "rpc worker-1940" 0x00007fa5c9cd2ad3 in ?? ()
60 LWP 19405 "rpc worker-1940" 0x00007fa5c9cd2ad3 in ?? ()
61 LWP 19406 "rpc worker-1940" 0x00007fa5c9cd2ad3 in ?? ()
62 LWP 19407 "rpc worker-1940" 0x00007fa5c9cd2ad3 in ?? ()
63 LWP 19408 "rpc worker-1940" 0x00007fa5c9cd2ad3 in ?? ()
64 LWP 19409 "rpc worker-1940" 0x00007fa5c9cd2ad3 in ?? ()
65 LWP 19410 "rpc worker-1941" 0x00007fa5c9cd2ad3 in ?? ()
66 LWP 19411 "rpc worker-1941" 0x00007fa5c9cd2ad3 in ?? ()
67 LWP 19412 "rpc worker-1941" 0x00007fa5c9cd2ad3 in ?? ()
68 LWP 19413 "rpc worker-1941" 0x00007fa5c9cd2ad3 in ?? ()
69 LWP 19414 "rpc worker-1941" 0x00007fa5c9cd2ad3 in ?? ()
70 LWP 19415 "rpc worker-1941" 0x00007fa5c9cd2ad3 in ?? ()
71 LWP 19416 "rpc worker-1941" 0x00007fa5c9cd2ad3 in ?? ()
72 LWP 19417 "rpc worker-1941" 0x00007fa5c9cd2ad3 in ?? ()
73 LWP 19418 "rpc worker-1941" 0x00007fa5c9cd2ad3 in ?? ()
74 LWP 19419 "rpc worker-1941" 0x00007fa5c9cd2ad3 in ?? ()
75 LWP 19420 "rpc worker-1942" 0x00007fa5c9cd2ad3 in ?? ()
76 LWP 19421 "rpc worker-1942" 0x00007fa5c9cd2ad3 in ?? ()
77 LWP 19422 "rpc worker-1942" 0x00007fa5c9cd2ad3 in ?? ()
78 LWP 19423 "rpc worker-1942" 0x00007fa5c9cd2ad3 in ?? ()
79 LWP 19424 "rpc worker-1942" 0x00007fa5c9cd2ad3 in ?? ()
80 LWP 19425 "rpc worker-1942" 0x00007fa5c9cd2ad3 in ?? ()
81 LWP 19426 "rpc worker-1942" 0x00007fa5c9cd2ad3 in ?? ()
82 LWP 19427 "rpc worker-1942" 0x00007fa5c9cd2ad3 in ?? ()
83 LWP 19428 "rpc worker-1942" 0x00007fa5c9cd2ad3 in ?? ()
84 LWP 19429 "rpc worker-1942" 0x00007fa5c9cd2ad3 in ?? ()
85 LWP 19430 "rpc worker-1943" 0x00007fa5c9cd2ad3 in ?? ()
86 LWP 19431 "rpc worker-1943" 0x00007fa5c9cd2ad3 in ?? ()
87 LWP 19432 "rpc worker-1943" 0x00007fa5c9cd2ad3 in ?? ()
88 LWP 19433 "rpc worker-1943" 0x00007fa5c9cd2ad3 in ?? ()
89 LWP 19434 "rpc worker-1943" 0x00007fa5c9cd2ad3 in ?? ()
90 LWP 19435 "rpc worker-1943" 0x00007fa5c9cd2ad3 in ?? ()
91 LWP 19436 "rpc worker-1943" 0x00007fa5c9cd2ad3 in ?? ()
92 LWP 19437 "rpc worker-1943" 0x00007fa5c9cd2ad3 in ?? ()
93 LWP 19438 "rpc worker-1943" 0x00007fa5c9cd2ad3 in ?? ()
94 LWP 19439 "rpc worker-1943" 0x00007fa5c9cd2ad3 in ?? ()
95 LWP 19440 "rpc worker-1944" 0x00007fa5c9cd2ad3 in ?? ()
96 LWP 19441 "rpc worker-1944" 0x00007fa5c9cd2ad3 in ?? ()
97 LWP 19442 "rpc worker-1944" 0x00007fa5c9cd2ad3 in ?? ()
98 LWP 19443 "rpc worker-1944" 0x00007fa5c9cd2ad3 in ?? ()
99 LWP 19444 "rpc worker-1944" 0x00007fa5c9cd2ad3 in ?? ()
100 LWP 19445 "rpc worker-1944" 0x00007fa5c9cd2ad3 in ?? ()
101 LWP 19446 "rpc worker-1944" 0x00007fa5c9cd2ad3 in ?? ()
102 LWP 19447 "rpc worker-1944" 0x00007fa5c9cd2ad3 in ?? ()
103 LWP 19448 "rpc worker-1944" 0x00007fa5c9cd2ad3 in ?? ()
104 LWP 19449 "rpc worker-1944" 0x00007fa5c9cd2ad3 in ?? ()
105 LWP 19450 "rpc worker-1945" 0x00007fa5c9cd2ad3 in ?? ()
106 LWP 19451 "rpc worker-1945" 0x00007fa5c9cd2ad3 in ?? ()
107 LWP 19452 "rpc worker-1945" 0x00007fa5c9cd2ad3 in ?? ()
108 LWP 19453 "rpc worker-1945" 0x00007fa5c9cd2ad3 in ?? ()
109 LWP 19454 "rpc worker-1945" 0x00007fa5c9cd2ad3 in ?? ()
110 LWP 19455 "rpc worker-1945" 0x00007fa5c9cd2ad3 in ?? ()
111 LWP 19456 "rpc worker-1945" 0x00007fa5c9cd2ad3 in ?? ()
112 LWP 19457 "rpc worker-1945" 0x00007fa5c9cd2ad3 in ?? ()
113 LWP 19458 "rpc worker-1945" 0x00007fa5c9cd2ad3 in ?? ()
114 LWP 19459 "rpc worker-1945" 0x00007fa5c9cd2ad3 in ?? ()
115 LWP 19460 "rpc worker-1946" 0x00007fa5c9cd2ad3 in ?? ()
116 LWP 19461 "rpc worker-1946" 0x00007fa5c9cd2ad3 in ?? ()
117 LWP 19462 "rpc worker-1946" 0x00007fa5c9cd2ad3 in ?? ()
118 LWP 19463 "diag-logger-194" 0x00007fa5c9cd2fb9 in ?? ()
119 LWP 19464 "result-tracker-" 0x00007fa5c9cd2fb9 in ?? ()
120 LWP 19465 "excess-log-dele" 0x00007fa5c9cd2fb9 in ?? ()
121 LWP 19466 "acceptor-19466" 0x00007fa5c50d90c7 in ?? ()
122 LWP 19467 "heartbeat-19467" 0x00007fa5c9cd2fb9 in ?? ()
123 LWP 19468 "maintenance_sch" 0x00007fa5c9cd2fb9 in ?? ()
124 LWP 19951 "raft [worker]-1" 0x00007fa5c9cd2fb9 in ?? ()
Thread 124 (LWP 19951):
#0 0x00007fa5c9cd2fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 123 (LWP 19468):
#0 0x00007fa5c9cd2fb9 in ?? ()
#1 0x00007b0100000000 in ?? ()
#2 0x0000000000000107 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007fa57deb96c0 in ?? ()
#6 0x000000000000020e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 19467):
#0 0x00007fa5c9cd2fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 19466):
#0 0x00007fa5c50d90c7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 19465):
#0 0x00007fa5c9cd2fb9 in ?? ()
#1 0x00007fa57f6bc940 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffededb1e60 in ?? ()
#5 0x00007fa57f6bc7b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 19464):
#0 0x00007fa5c9cd2fb9 in ?? ()
#1 0x0000000085352fb8 in ?? ()
#2 0x0000000000000042 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001008 in ?? ()
#5 0x00007fa57febd800 in ?? ()
#6 0x0000000000000084 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 19463):
#0 0x00007fa5c9cd2fb9 in ?? ()
#1 0x00007fa5c3136008 in ?? ()
#2 0x0000000000000042 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000c90 in ?? ()
#5 0x00007fa5806be750 in ?? ()
#6 0x0000000000000084 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 19462):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 19461):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 19460):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 19459):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 19458):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 19457):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 19456):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 19455):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 19454):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 19453):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 19452):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 19451):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 19450):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 19449):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 19448):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 19447):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 19446):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 19445):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 19444):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 19443):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 19442):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 19441):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 19440):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 19439):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 19438):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 19437):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 19436):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 19435):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 19434):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 19433):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 19432):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 19431):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 19430):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 19429):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 19428):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 19427):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 19426):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 19425):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 19424):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 19423):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 19422):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24001147cc in ?? ()
#4 0x00007fa5958ba710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5958ba730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007fa5c9cd2770 in ?? ()
#10 0x00007fa5958ba730 in ?? ()
#11 0x00007fa5bfa86a48 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 76 (LWP 19421):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240010ffc8 in ?? ()
#4 0x00007fa5960bb710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5960bb730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 19420):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 19419):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 19418):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 19417):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 19416):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 19415):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 19414):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 19413):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 19412):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 19411):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 19410):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 19409):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 19408):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 19407):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 19406):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 19405):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 19404):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 19403):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 19402):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000b902c in ?? ()
#4 0x00007fa59fcbc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa59fcbc730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x007f0400000026c8 in ?? ()
#9 0x00007fa5c9cd2770 in ?? ()
#10 0x00007fa59fcbc730 in ?? ()
#11 0x0002008300000dfe in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 19401):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 19400):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 19399):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 19398):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 19397):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 19396):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 19395):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 19394):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 19393):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 19392):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 19391):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 19390):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 19389):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 19388):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 19387):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 19386):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 19385):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 19384):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 19383):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 19382):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000065 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240005ffec in ?? ()
#4 0x00007fa5aa0be710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5aa0be730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007fa5c9cd2770 in ?? ()
#10 0x00007fa5aa0be730 in ?? ()
#11 0x00007fa5781e8ea0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 36 (LWP 19381):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x00000000000003ea in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240005d7f8 in ?? ()
#4 0x00007fa5aaab6710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5aaab6730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 19380):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x00000000000003aa in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b2400058ff8 in ?? ()
#4 0x00007fa5ab2b7710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5ab2b7730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 34 (LWP 19379):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x000000000000000d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000547fc in ?? ()
#4 0x00007fa5abab8710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5abab8730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007fa5c9cd2770 in ?? ()
#10 0x00007fa5abab8730 in ?? ()
#11 0x00007fa5781b88a0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 33 (LWP 19378):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 19377):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 19376):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 19375):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 19374):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 19373):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 19372):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 19371):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 19370):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 19369):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 19368):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 19367):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 19366):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 19365):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 19364):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 19363):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 19362):
#0 0x00007fa5c9cd2fb9 in ?? ()
#1 0x0000000017a335f0 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800003a00 in ?? ()
#5 0x00007fa5b4692700 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 19361):
#0 0x00007fa5c9cd2fb9 in ?? ()
#1 0x00007fa5b4e939a8 in ?? ()
#2 0x000000000000000d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000372d8 in ?? ()
#5 0x00007fa5b4e93840 in ?? ()
#6 0x000000000000001a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 19360):
#0 0x00007fa5c9cd2fb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007fa5b5694410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 19359):
#0 0x00007fa5c9cd2ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 19358):
#0 0x00007fa5c50d7a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 19357):
#0 0x00007fa5c50d7a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 11 (LWP 19356):
#0 0x00007fa5c50d7a47 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0040e00001d691c8 in ?? ()
#3 0x00007fa5b7698500 in ?? ()
#4 0x00007fa5b7699b80 in ?? ()
#5 0x00007fa5b7698500 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x00007b5800001200 in ?? ()
#8 0x0000000000488695 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007fa5c2960000 in ?? ()
#10 0x0000000000488599 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007fa5b7699b80 in ?? ()
#12 0x00007fa5c7b30069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007fa5cd2b11a0 in ?? ()
#15 0x00007b4c000026d0 in ?? ()
#16 0x00007b4c000026d8 in ?? ()
#17 0x00007fa5b76987a0 in ?? ()
#18 0x00007b4400036b40 in ?? ()
#19 0x00007fa5b7698cd0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 10 (LWP 19355):
#0 0x00007fa5c50d7a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 9 (LWP 19352):
#0 0x00007fa5c50cacb9 in ?? ()
#1 0x00007fa5bdebcc10 in ?? ()
#2 0x00007b0400009010 in ?? ()
#3 0x00007fa5bdebdb80 in ?? ()
#4 0x00007fa5bdebcc10 in ?? ()
#5 0x00007b0400009010 in ?? ()
#6 0x00000000004888a3 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007fa5c2b04000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007fa5bdebdb80 in ?? ()
#10 0x00007fa5ceaaf908 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 8 (LWP 19351):
#0 0x00007fa5c9cd2fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 7 (LWP 19350):
#0 0x00007fa5c9cd69e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 19343):
#0 0x00007fa5c9cd2fb9 in ?? ()
#1 0x00007fa5beebea40 in ?? ()
#2 0x0000000000000151 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400035b98 in ?? ()
#5 0x00007fa5beebe5d0 in ?? ()
#6 0x00000000000002a2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 19342):
#0 0x00007fa5c9cd2fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 19341):
#0 0x00007fa5c9cd2fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 19340):
#0 0x00007fa5c9cd2fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 19339):
#0 0x00007fa5c509a7a0 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 19338):
#0 0x00007fa5c9cd6d50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x0000000000467b2b in __sanitizer::internal_alloc_placeholder ()
#3 0x00007fa5c42f8cc0 in ?? ()
#4 0x00007fa5c42f8cc0 in ?? ()
#5 0x00007ffededb1c70 in ?? ()
#6 0x000000000048aef4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000a94cd2 in ?? ()
#9 0x00007fa5c42f8cc0 in ?? ()
#10 0x00007fa5c81fef0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250625 13:59:04.182440 18577 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID 935a53707abf4076933a524d32ff50f5 and pid 19471
************************ BEGIN STACKS **************************
[New LWP 19472]
[New LWP 19473]
[New LWP 19474]
[New LWP 19475]
[New LWP 19476]
[New LWP 19484]
[New LWP 19485]
[New LWP 19486]
[New LWP 19489]
[New LWP 19490]
[New LWP 19491]
[New LWP 19492]
[New LWP 19493]
[New LWP 19494]
[New LWP 19495]
[New LWP 19496]
[New LWP 19497]
[New LWP 19498]
[New LWP 19499]
[New LWP 19500]
[New LWP 19501]
[New LWP 19502]
[New LWP 19503]
[New LWP 19504]
[New LWP 19505]
[New LWP 19506]
[New LWP 19507]
[New LWP 19508]
[New LWP 19509]
[New LWP 19510]
[New LWP 19511]
[New LWP 19512]
[New LWP 19513]
[New LWP 19514]
[New LWP 19515]
[New LWP 19516]
[New LWP 19517]
[New LWP 19518]
[New LWP 19519]
[New LWP 19520]
[New LWP 19521]
[New LWP 19522]
[New LWP 19523]
[New LWP 19524]
[New LWP 19525]
[New LWP 19526]
[New LWP 19527]
[New LWP 19528]
[New LWP 19529]
[New LWP 19530]
[New LWP 19531]
[New LWP 19532]
[New LWP 19533]
[New LWP 19534]
[New LWP 19535]
[New LWP 19536]
[New LWP 19537]
[New LWP 19538]
[New LWP 19539]
[New LWP 19540]
[New LWP 19541]
[New LWP 19542]
[New LWP 19543]
[New LWP 19544]
[New LWP 19545]
[New LWP 19546]
[New LWP 19547]
[New LWP 19548]
[New LWP 19549]
[New LWP 19550]
[New LWP 19551]
[New LWP 19552]
[New LWP 19553]
[New LWP 19554]
[New LWP 19555]
[New LWP 19556]
[New LWP 19557]
[New LWP 19558]
[New LWP 19559]
[New LWP 19560]
[New LWP 19561]
[New LWP 19562]
[New LWP 19563]
[New LWP 19564]
[New LWP 19565]
[New LWP 19566]
[New LWP 19567]
[New LWP 19568]
[New LWP 19569]
[New LWP 19570]
[New LWP 19571]
[New LWP 19572]
[New LWP 19573]
[New LWP 19574]
[New LWP 19575]
[New LWP 19576]
[New LWP 19577]
[New LWP 19578]
[New LWP 19579]
[New LWP 19580]
[New LWP 19581]
[New LWP 19582]
[New LWP 19583]
[New LWP 19584]
[New LWP 19585]
[New LWP 19586]
[New LWP 19587]
[New LWP 19588]
[New LWP 19589]
[New LWP 19590]
[New LWP 19591]
[New LWP 19592]
[New LWP 19593]
[New LWP 19594]
[New LWP 19595]
[New LWP 19596]
[New LWP 19597]
[New LWP 19598]
[New LWP 19599]
[New LWP 19600]
[New LWP 19601]
[New LWP 19602]
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
0x00007fe4b9c88d50 in ?? ()
Id Target Id Frame
* 1 LWP 19471 "kudu" 0x00007fe4b9c88d50 in ?? ()
2 LWP 19472 "kudu" 0x00007fe4b504c7a0 in ?? ()
3 LWP 19473 "kudu" 0x00007fe4b9c84fb9 in ?? ()
4 LWP 19474 "kudu" 0x00007fe4b9c84fb9 in ?? ()
5 LWP 19475 "kudu" 0x00007fe4b9c84fb9 in ?? ()
6 LWP 19476 "kernel-watcher-" 0x00007fe4b9c84fb9 in ?? ()
7 LWP 19484 "ntp client-1948" 0x00007fe4b9c889e2 in ?? ()
8 LWP 19485 "file cache-evic" 0x00007fe4b9c84fb9 in ?? ()
9 LWP 19486 "sq_acceptor" 0x00007fe4b507ccb9 in ?? ()
10 LWP 19489 "rpc reactor-194" 0x00007fe4b5089a47 in ?? ()
11 LWP 19490 "rpc reactor-194" 0x00007fe4b5089a47 in ?? ()
12 LWP 19491 "rpc reactor-194" 0x00007fe4b5089a47 in ?? ()
13 LWP 19492 "rpc reactor-194" 0x00007fe4b5089a47 in ?? ()
14 LWP 19493 "MaintenanceMgr " 0x00007fe4b9c84ad3 in ?? ()
15 LWP 19494 "txn-status-mana" 0x00007fe4b9c84fb9 in ?? ()
16 LWP 19495 "collect_and_rem" 0x00007fe4b9c84fb9 in ?? ()
17 LWP 19496 "tc-session-exp-" 0x00007fe4b9c84fb9 in ?? ()
18 LWP 19497 "rpc worker-1949" 0x00007fe4b9c84ad3 in ?? ()
19 LWP 19498 "rpc worker-1949" 0x00007fe4b9c84ad3 in ?? ()
20 LWP 19499 "rpc worker-1949" 0x00007fe4b9c84ad3 in ?? ()
21 LWP 19500 "rpc worker-1950" 0x00007fe4b9c84ad3 in ?? ()
22 LWP 19501 "rpc worker-1950" 0x00007fe4b9c84ad3 in ?? ()
23 LWP 19502 "rpc worker-1950" 0x00007fe4b9c84ad3 in ?? ()
24 LWP 19503 "rpc worker-1950" 0x00007fe4b9c84ad3 in ?? ()
25 LWP 19504 "rpc worker-1950" 0x00007fe4b9c84ad3 in ?? ()
26 LWP 19505 "rpc worker-1950" 0x00007fe4b9c84ad3 in ?? ()
27 LWP 19506 "rpc worker-1950" 0x00007fe4b9c84ad3 in ?? ()
28 LWP 19507 "rpc worker-1950" 0x00007fe4b9c84ad3 in ?? ()
29 LWP 19508 "rpc worker-1950" 0x00007fe4b9c84ad3 in ?? ()
30 LWP 19509 "rpc worker-1950" 0x00007fe4b9c84ad3 in ?? ()
31 LWP 19510 "rpc worker-1951" 0x00007fe4b9c84ad3 in ?? ()
32 LWP 19511 "rpc worker-1951" 0x00007fe4b9c84ad3 in ?? ()
33 LWP 19512 "rpc worker-1951" 0x00007fe4b9c84ad3 in ?? ()
34 LWP 19513 "rpc worker-1951" 0x00007fe4b9c84ad3 in ?? ()
35 LWP 19514 "rpc worker-1951" 0x00007fe4b9c84ad3 in ?? ()
36 LWP 19515 "rpc worker-1951" 0x00007fe4b9c84ad3 in ?? ()
37 LWP 19516 "rpc worker-1951" 0x00007fe4b9c84ad3 in ?? ()
38 LWP 19517 "rpc worker-1951" 0x00007fe4b9c84ad3 in ?? ()
39 LWP 19518 "rpc worker-1951" 0x00007fe4b9c84ad3 in ?? ()
40 LWP 19519 "rpc worker-1951" 0x00007fe4b9c84ad3 in ?? ()
41 LWP 19520 "rpc worker-1952" 0x00007fe4b9c84ad3 in ?? ()
42 LWP 19521 "rpc worker-1952" 0x00007fe4b9c84ad3 in ?? ()
43 LWP 19522 "rpc worker-1952" 0x00007fe4b9c84ad3 in ?? ()
44 LWP 19523 "rpc worker-1952" 0x00007fe4b9c84ad3 in ?? ()
45 LWP 19524 "rpc worker-1952" 0x00007fe4b9c84ad3 in ?? ()
46 LWP 19525 "rpc worker-1952" 0x00007fe4b9c84ad3 in ?? ()
47 LWP 19526 "rpc worker-1952" 0x00007fe4b9c84ad3 in ?? ()
48 LWP 19527 "rpc worker-1952" 0x00007fe4b9c84ad3 in ?? ()
49 LWP 19528 "rpc worker-1952" 0x00007fe4b9c84ad3 in ?? ()
50 LWP 19529 "rpc worker-1952" 0x00007fe4b9c84ad3 in ?? ()
51 LWP 19530 "rpc worker-1953" 0x00007fe4b9c84ad3 in ?? ()
52 LWP 19531 "rpc worker-1953" 0x00007fe4b9c84ad3 in ?? ()
53 LWP 19532 "rpc worker-1953" 0x00007fe4b9c84ad3 in ?? ()
54 LWP 19533 "rpc worker-1953" 0x00007fe4b9c84ad3 in ?? ()
55 LWP 19534 "rpc worker-1953" 0x00007fe4b9c84ad3 in ?? ()
56 LWP 19535 "rpc worker-1953" 0x00007fe4b9c84ad3 in ?? ()
57 LWP 19536 "rpc worker-1953" 0x00007fe4b9c84ad3 in ?? ()
58 LWP 19537 "rpc worker-1953" 0x00007fe4b9c84ad3 in ?? ()
59 LWP 19538 "rpc worker-1953" 0x00007fe4b9c84ad3 in ?? ()
60 LWP 19539 "rpc worker-1953" 0x00007fe4b9c84ad3 in ?? ()
61 LWP 19540 "rpc worker-1954" 0x00007fe4b9c84ad3 in ?? ()
62 LWP 19541 "rpc worker-1954" 0x00007fe4b9c84ad3 in ?? ()
63 LWP 19542 "rpc worker-1954" 0x00007fe4b9c84ad3 in ?? ()
64 LWP 19543 "rpc worker-1954" 0x00007fe4b9c84ad3 in ?? ()
65 LWP 19544 "rpc worker-1954" 0x00007fe4b9c84ad3 in ?? ()
66 LWP 19545 "rpc worker-1954" 0x00007fe4b9c84ad3 in ?? ()
67 LWP 19546 "rpc worker-1954" 0x00007fe4b9c84ad3 in ?? ()
68 LWP 19547 "rpc worker-1954" 0x00007fe4b9c84ad3 in ?? ()
69 LWP 19548 "rpc worker-1954" 0x00007fe4b9c84ad3 in ?? ()
70 LWP 19549 "rpc worker-1954" 0x00007fe4b9c84ad3 in ?? ()
71 LWP 19550 "rpc worker-1955" 0x00007fe4b9c84ad3 in ?? ()
72 LWP 19551 "rpc worker-1955" 0x00007fe4b9c84ad3 in ?? ()
73 LWP 19552 "rpc worker-1955" 0x00007fe4b9c84ad3 in ?? ()
74 LWP 19553 "rpc worker-1955" 0x00007fe4b9c84ad3 in ?? ()
75 LWP 19554 "rpc worker-1955" 0x00007fe4b9c84ad3 in ?? ()
76 LWP 19555 "rpc worker-1955" 0x00007fe4b9c84ad3 in ?? ()
77 LWP 19556 "rpc worker-1955" 0x00007fe4b9c84ad3 in ?? ()
78 LWP 19557 "rpc worker-1955" 0x00007fe4b9c84ad3 in ?? ()
79 LWP 19558 "rpc worker-1955" 0x00007fe4b9c84ad3 in ?? ()
80 LWP 19559 "rpc worker-1955" 0x00007fe4b9c84ad3 in ?? ()
81 LWP 19560 "rpc worker-1956" 0x00007fe4b9c84ad3 in ?? ()
82 LWP 19561 "rpc worker-1956" 0x00007fe4b9c84ad3 in ?? ()
83 LWP 19562 "rpc worker-1956" 0x00007fe4b9c84ad3 in ?? ()
84 LWP 19563 "rpc worker-1956" 0x00007fe4b9c84ad3 in ?? ()
85 LWP 19564 "rpc worker-1956" 0x00007fe4b9c84ad3 in ?? ()
86 LWP 19565 "rpc worker-1956" 0x00007fe4b9c84ad3 in ?? ()
87 LWP 19566 "rpc worker-1956" 0x00007fe4b9c84ad3 in ?? ()
88 LWP 19567 "rpc worker-1956" 0x00007fe4b9c84ad3 in ?? ()
89 LWP 19568 "rpc worker-1956" 0x00007fe4b9c84ad3 in ?? ()
90 LWP 19569 "rpc worker-1956" 0x00007fe4b9c84ad3 in ?? ()
91 LWP 19570 "rpc worker-1957" 0x00007fe4b9c84ad3 in ?? ()
92 LWP 19571 "rpc worker-1957" 0x00007fe4b9c84ad3 in ?? ()
93 LWP 19572 "rpc worker-1957" 0x00007fe4b9c84ad3 in ?? ()
94 LWP 19573 "rpc worker-1957" 0x00007fe4b9c84ad3 in ?? ()
95 LWP 19574 "rpc worker-1957" 0x00007fe4b9c84ad3 in ?? ()
96 LWP 19575 "rpc worker-1957" 0x00007fe4b9c84ad3 in ?? ()
97 LWP 19576 "rpc worker-1957" 0x00007fe4b9c84ad3 in ?? ()
98 LWP 19577 "rpc worker-1957" 0x00007fe4b9c84ad3 in ?? ()
99 LWP 19578 "rpc worker-1957" 0x00007fe4b9c84ad3 in ?? ()
100 LWP 19579 "rpc worker-1957" 0x00007fe4b9c84ad3 in ?? ()
101 LWP 19580 "rpc worker-1958" 0x00007fe4b9c84ad3 in ?? ()
102 LWP 19581 "rpc worker-1958" 0x00007fe4b9c84ad3 in ?? ()
103 LWP 19582 "rpc worker-1958" 0x00007fe4b9c84ad3 in ?? ()
104 LWP 19583 "rpc worker-1958" 0x00007fe4b9c84ad3 in ?? ()
105 LWP 19584 "rpc worker-1958" 0x00007fe4b9c84ad3 in ?? ()
106 LWP 19585 "rpc worker-1958" 0x00007fe4b9c84ad3 in ?? ()
107 LWP 19586 "rpc worker-1958" 0x00007fe4b9c84ad3 in ?? ()
108 LWP 19587 "rpc worker-1958" 0x00007fe4b9c84ad3 in ?? ()
109 LWP 19588 "rpc worker-1958" 0x00007fe4b9c84ad3 in ?? ()
110 LWP 19589 "rpc worker-1958" 0x00007fe4b9c84ad3 in ?? ()
111 LWP 19590 "rpc worker-1959" 0x00007fe4b9c84ad3 in ?? ()
112 LWP 19591 "rpc worker-1959" 0x00007fe4b9c84ad3 in ?? ()
113 LWP 19592 "rpc worker-1959" 0x00007fe4b9c84ad3 in ?? ()
114 LWP 19593 "rpc worker-1959" 0x00007fe4b9c84ad3 in ?? ()
115 LWP 19594 "rpc worker-1959" 0x00007fe4b9c84ad3 in ?? ()
116 LWP 19595 "rpc worker-1959" 0x00007fe4b9c84ad3 in ?? ()
117 LWP 19596 "rpc worker-1959" 0x00007fe4b9c84ad3 in ?? ()
118 LWP 19597 "diag-logger-195" 0x00007fe4b9c84fb9 in ?? ()
119 LWP 19598 "result-tracker-" 0x00007fe4b9c84fb9 in ?? ()
120 LWP 19599 "excess-log-dele" 0x00007fe4b9c84fb9 in ?? ()
121 LWP 19600 "acceptor-19600" 0x00007fe4b508b0c7 in ?? ()
122 LWP 19601 "heartbeat-19601" 0x00007fe4b9c84fb9 in ?? ()
123 LWP 19602 "maintenance_sch" 0x00007fe4b9c84fb9 in ?? ()
Thread 123 (LWP 19602):
#0 0x00007fe4b9c84fb9 in ?? ()
#1 0x00007b0100000000 in ?? ()
#2 0x0000000000000103 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007fe46deb96c0 in ?? ()
#6 0x0000000000000206 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 19601):
#0 0x00007fe4b9c84fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 19600):
#0 0x00007fe4b508b0c7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 19599):
#0 0x00007fe4b9c84fb9 in ?? ()
#1 0x00007fe46f6bc940 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffdc9b6e400 in ?? ()
#5 0x00007fe46f6bc7b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 19598):
#0 0x00007fe4b9c84fb9 in ?? ()
#1 0x0000000085352fb8 in ?? ()
#2 0x0000000000000041 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001008 in ?? ()
#5 0x00007fe46febd800 in ?? ()
#6 0x0000000000000082 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 19597):
#0 0x00007fe4b9c84fb9 in ?? ()
#1 0x00007fe4b310e008 in ?? ()
#2 0x0000000000000041 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000c90 in ?? ()
#5 0x00007fe4706be750 in ?? ()
#6 0x0000000000000082 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 19596):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 19595):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 19594):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 19593):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 19592):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 19591):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 19590):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 19589):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 19588):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 19587):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 19586):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 19585):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 19584):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 19583):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 19582):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 19581):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 19580):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 19579):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 19578):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 19577):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 19576):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 19575):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 19574):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 19573):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 19572):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 19571):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 19570):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 19569):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 19568):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 19567):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 19566):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 19565):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 19564):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 19563):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 19562):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 19561):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 19560):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 19559):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 19558):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 19557):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 19556):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000712 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b24001147c8 in ?? ()
#4 0x00007fe4858ba710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fe4858ba730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 76 (LWP 19555):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x00000000000007d0 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240010ffc8 in ?? ()
#4 0x00007fe4860bb710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fe4860bb730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 19554):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x00000000000000f0 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240010d7d8 in ?? ()
#4 0x00007fe4868bc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fe4868bc730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 19553):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 19552):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 19551):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 19550):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 19549):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 19548):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 19547):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 19546):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 19545):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 19544):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 19543):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 19542):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 19541):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 19540):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 19539):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 19538):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 19537):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 19536):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000b902c in ?? ()
#4 0x00007fe48fcbc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fe48fcbc730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x007f0400000026c8 in ?? ()
#9 0x00007fe4b9c84770 in ?? ()
#10 0x00007fe48fcbc730 in ?? ()
#11 0x0002008300000dfe in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 19535):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 19534):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 19533):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 19532):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 19531):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 19530):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 19529):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 19528):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 19527):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 19526):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 19525):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 19524):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 19523):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 19522):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 19521):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 19520):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 19519):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 19518):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 19517):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 19516):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240005ffec in ?? ()
#4 0x00007fe49a0be710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fe49a0be730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007fe4b9c84770 in ?? ()
#10 0x00007fe49a0be730 in ?? ()
#11 0x00007fe4b209fc58 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 36 (LWP 19515):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 19514):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 19513):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 19512):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 19511):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 19510):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 19509):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 19508):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 19507):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 19506):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 19505):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 19504):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 19503):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 19502):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 19501):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 19500):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 19499):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 19498):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 19497):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 19496):
#0 0x00007fe4b9c84fb9 in ?? ()
#1 0x0000000017a335f0 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800003a00 in ?? ()
#5 0x00007fe4a4692700 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 19495):
#0 0x00007fe4b9c84fb9 in ?? ()
#1 0x00007fe4a4e939a8 in ?? ()
#2 0x000000000000000d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000372d8 in ?? ()
#5 0x00007fe4a4e93840 in ?? ()
#6 0x000000000000001a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 19494):
#0 0x00007fe4b9c84fb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007fe4a5694410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 19493):
#0 0x00007fe4b9c84ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 19492):
#0 0x00007fe4b5089a47 in ?? ()
#1 0x00007b0c000d5cc0 in ?? ()
#2 0x0048e000019f5a30 in ?? ()
#3 0x00007fe4a6696500 in ?? ()
#4 0x00007fe4a6697b80 in ?? ()
#5 0x00007fe4a6696500 in ?? ()
#6 0x0000000000000012 in ?? ()
#7 0x00007b5800001b00 in ?? ()
#8 0x0000000000488695 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007fe4b27ee000 in ?? ()
#10 0x0000000000488599 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007fe4a6697b80 in ?? ()
#12 0x00007fe4b7ae2069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007fe4bd2631a0 in ?? ()
#15 0x00007b4c00003310 in ?? ()
#16 0x00007b4c00003318 in ?? ()
#17 0x00007fe4a66967a0 in ?? ()
#18 0x00007b4400033d00 in ?? ()
#19 0x00007fe4a6696cd0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 12 (LWP 19491):
#0 0x00007fe4b5089a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 11 (LWP 19490):
#0 0x00007fe4b5089a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 10 (LWP 19489):
#0 0x00007fe4b5089a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 9 (LWP 19486):
#0 0x00007fe4b507ccb9 in ?? ()
#1 0x00007fe4adebcc10 in ?? ()
#2 0x00007b040000a050 in ?? ()
#3 0x00007fe4adebdb80 in ?? ()
#4 0x00007fe4adebcc10 in ?? ()
#5 0x00007b040000a050 in ?? ()
#6 0x00000000004888a3 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007fe4b29ee000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007fe4adebdb80 in ?? ()
#10 0x00007fe4bea61908 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 8 (LWP 19485):
#0 0x00007fe4b9c84fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 7 (LWP 19484):
#0 0x00007fe4b9c889e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 19476):
#0 0x00007fe4b9c84fb9 in ?? ()
#1 0x00007fe4aeebea40 in ?? ()
#2 0x000000000000014b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400035b98 in ?? ()
#5 0x00007fe4aeebe5d0 in ?? ()
#6 0x0000000000000296 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 19475):
#0 0x00007fe4b9c84fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 19474):
#0 0x00007fe4b9c84fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 19473):
#0 0x00007fe4b9c84fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 19472):
#0 0x00007fe4b504c7a0 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 19471):
#0 0x00007fe4b9c88d50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x0000000000467b2b in __sanitizer::internal_alloc_placeholder ()
#3 0x00007fe4b42aacc0 in ?? ()
#4 0x00007fe4b42aacc0 in ?? ()
#5 0x00007ffdc9b6e210 in ?? ()
#6 0x000000000048aef4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000a95f82 in ?? ()
#9 0x00007fe4b42aacc0 in ?? ()
#10 0x00007fe4b81b0f0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250625 13:59:05.268559 18577 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID 65707482989741b7ba8973d9cf8c7d7c and pid 19605
************************ BEGIN STACKS **************************
[New LWP 19606]
[New LWP 19607]
[New LWP 19608]
[New LWP 19609]
[New LWP 19610]
[New LWP 19617]
[New LWP 19618]
[New LWP 19619]
[New LWP 19622]
[New LWP 19623]
[New LWP 19624]
[New LWP 19625]
[New LWP 19626]
[New LWP 19627]
[New LWP 19628]
[New LWP 19629]
[New LWP 19630]
[New LWP 19631]
[New LWP 19632]
[New LWP 19633]
[New LWP 19634]
[New LWP 19635]
[New LWP 19636]
[New LWP 19637]
[New LWP 19638]
[New LWP 19639]
[New LWP 19640]
[New LWP 19641]
[New LWP 19642]
[New LWP 19643]
[New LWP 19644]
[New LWP 19645]
[New LWP 19646]
[New LWP 19647]
[New LWP 19648]
[New LWP 19649]
[New LWP 19650]
[New LWP 19651]
[New LWP 19652]
[New LWP 19653]
[New LWP 19654]
[New LWP 19655]
[New LWP 19656]
[New LWP 19657]
[New LWP 19658]
[New LWP 19659]
[New LWP 19660]
[New LWP 19661]
[New LWP 19662]
[New LWP 19663]
[New LWP 19664]
[New LWP 19665]
[New LWP 19666]
[New LWP 19667]
[New LWP 19668]
[New LWP 19669]
[New LWP 19670]
[New LWP 19671]
[New LWP 19672]
[New LWP 19673]
[New LWP 19674]
[New LWP 19675]
[New LWP 19676]
[New LWP 19677]
[New LWP 19678]
[New LWP 19679]
[New LWP 19680]
[New LWP 19681]
[New LWP 19682]
[New LWP 19683]
[New LWP 19684]
[New LWP 19685]
[New LWP 19686]
[New LWP 19687]
[New LWP 19688]
[New LWP 19689]
[New LWP 19690]
[New LWP 19691]
[New LWP 19692]
[New LWP 19693]
[New LWP 19694]
[New LWP 19695]
[New LWP 19696]
[New LWP 19697]
[New LWP 19698]
[New LWP 19699]
[New LWP 19700]
[New LWP 19701]
[New LWP 19702]
[New LWP 19703]
[New LWP 19704]
[New LWP 19705]
[New LWP 19706]
[New LWP 19707]
[New LWP 19708]
[New LWP 19709]
[New LWP 19710]
[New LWP 19711]
[New LWP 19712]
[New LWP 19713]
[New LWP 19714]
[New LWP 19715]
[New LWP 19716]
[New LWP 19717]
[New LWP 19718]
[New LWP 19719]
[New LWP 19720]
[New LWP 19721]
[New LWP 19722]
[New LWP 19723]
[New LWP 19724]
[New LWP 19725]
[New LWP 19726]
[New LWP 19727]
[New LWP 19728]
[New LWP 19729]
[New LWP 19730]
[New LWP 19731]
[New LWP 19732]
[New LWP 19733]
[New LWP 19734]
[New LWP 19735]
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
0x00007f14977f0d50 in ?? ()
Id Target Id Frame
* 1 LWP 19605 "kudu" 0x00007f14977f0d50 in ?? ()
2 LWP 19606 "kudu" 0x00007f1492bb47a0 in ?? ()
3 LWP 19607 "kudu" 0x00007f14977ecfb9 in ?? ()
4 LWP 19608 "kudu" 0x00007f14977ecfb9 in ?? ()
5 LWP 19609 "kudu" 0x00007f14977ecfb9 in ?? ()
6 LWP 19610 "kernel-watcher-" 0x00007f14977ecfb9 in ?? ()
7 LWP 19617 "ntp client-1961" 0x00007f14977f09e2 in ?? ()
8 LWP 19618 "file cache-evic" 0x00007f14977ecfb9 in ?? ()
9 LWP 19619 "sq_acceptor" 0x00007f1492be4cb9 in ?? ()
10 LWP 19622 "rpc reactor-196" 0x00007f1492bf1a47 in ?? ()
11 LWP 19623 "rpc reactor-196" 0x00007f1492bf1a47 in ?? ()
12 LWP 19624 "rpc reactor-196" 0x00007f1492bf1a47 in ?? ()
13 LWP 19625 "rpc reactor-196" 0x00007f1492bf1a47 in ?? ()
14 LWP 19626 "MaintenanceMgr " 0x00007f14977ecad3 in ?? ()
15 LWP 19627 "txn-status-mana" 0x00007f14977ecfb9 in ?? ()
16 LWP 19628 "collect_and_rem" 0x00007f14977ecfb9 in ?? ()
17 LWP 19629 "tc-session-exp-" 0x00007f14977ecfb9 in ?? ()
18 LWP 19630 "rpc worker-1963" 0x00007f14977ecad3 in ?? ()
19 LWP 19631 "rpc worker-1963" 0x00007f14977ecad3 in ?? ()
20 LWP 19632 "rpc worker-1963" 0x00007f14977ecad3 in ?? ()
21 LWP 19633 "rpc worker-1963" 0x00007f14977ecad3 in ?? ()
22 LWP 19634 "rpc worker-1963" 0x00007f14977ecad3 in ?? ()
23 LWP 19635 "rpc worker-1963" 0x00007f14977ecad3 in ?? ()
24 LWP 19636 "rpc worker-1963" 0x00007f14977ecad3 in ?? ()
25 LWP 19637 "rpc worker-1963" 0x00007f14977ecad3 in ?? ()
26 LWP 19638 "rpc worker-1963" 0x00007f14977ecad3 in ?? ()
27 LWP 19639 "rpc worker-1963" 0x00007f14977ecad3 in ?? ()
28 LWP 19640 "rpc worker-1964" 0x00007f14977ecad3 in ?? ()
29 LWP 19641 "rpc worker-1964" 0x00007f14977ecad3 in ?? ()
30 LWP 19642 "rpc worker-1964" 0x00007f14977ecad3 in ?? ()
31 LWP 19643 "rpc worker-1964" 0x00007f14977ecad3 in ?? ()
32 LWP 19644 "rpc worker-1964" 0x00007f14977ecad3 in ?? ()
33 LWP 19645 "rpc worker-1964" 0x00007f14977ecad3 in ?? ()
34 LWP 19646 "rpc worker-1964" 0x00007f14977ecad3 in ?? ()
35 LWP 19647 "rpc worker-1964" 0x00007f14977ecad3 in ?? ()
36 LWP 19648 "rpc worker-1964" 0x00007f14977ecad3 in ?? ()
37 LWP 19649 "rpc worker-1964" 0x00007f14977ecad3 in ?? ()
38 LWP 19650 "rpc worker-1965" 0x00007f14977ecad3 in ?? ()
39 LWP 19651 "rpc worker-1965" 0x00007f14977ecad3 in ?? ()
40 LWP 19652 "rpc worker-1965" 0x00007f14977ecad3 in ?? ()
41 LWP 19653 "rpc worker-1965" 0x00007f14977ecad3 in ?? ()
42 LWP 19654 "rpc worker-1965" 0x00007f14977ecad3 in ?? ()
43 LWP 19655 "rpc worker-1965" 0x00007f14977ecad3 in ?? ()
44 LWP 19656 "rpc worker-1965" 0x00007f14977ecad3 in ?? ()
45 LWP 19657 "rpc worker-1965" 0x00007f14977ecad3 in ?? ()
46 LWP 19658 "rpc worker-1965" 0x00007f14977ecad3 in ?? ()
47 LWP 19659 "rpc worker-1965" 0x00007f14977ecad3 in ?? ()
48 LWP 19660 "rpc worker-1966" 0x00007f14977ecad3 in ?? ()
49 LWP 19661 "rpc worker-1966" 0x00007f14977ecad3 in ?? ()
50 LWP 19662 "rpc worker-1966" 0x00007f14977ecad3 in ?? ()
51 LWP 19663 "rpc worker-1966" 0x00007f14977ecad3 in ?? ()
52 LWP 19664 "rpc worker-1966" 0x00007f14977ecad3 in ?? ()
53 LWP 19665 "rpc worker-1966" 0x00007f14977ecad3 in ?? ()
54 LWP 19666 "rpc worker-1966" 0x00007f14977ecad3 in ?? ()
55 LWP 19667 "rpc worker-1966" 0x00007f14977ecad3 in ?? ()
56 LWP 19668 "rpc worker-1966" 0x00007f14977ecad3 in ?? ()
57 LWP 19669 "rpc worker-1966" 0x00007f14977ecad3 in ?? ()
58 LWP 19670 "rpc worker-1967" 0x00007f14977ecad3 in ?? ()
59 LWP 19671 "rpc worker-1967" 0x00007f14977ecad3 in ?? ()
60 LWP 19672 "rpc worker-1967" 0x00007f14977ecad3 in ?? ()
61 LWP 19673 "rpc worker-1967" 0x00007f14977ecad3 in ?? ()
62 LWP 19674 "rpc worker-1967" 0x00007f14977ecad3 in ?? ()
63 LWP 19675 "rpc worker-1967" 0x00007f14977ecad3 in ?? ()
64 LWP 19676 "rpc worker-1967" 0x00007f14977ecad3 in ?? ()
65 LWP 19677 "rpc worker-1967" 0x00007f14977ecad3 in ?? ()
66 LWP 19678 "rpc worker-1967" 0x00007f14977ecad3 in ?? ()
67 LWP 19679 "rpc worker-1967" 0x00007f14977ecad3 in ?? ()
68 LWP 19680 "rpc worker-1968" 0x00007f14977ecad3 in ?? ()
69 LWP 19681 "rpc worker-1968" 0x00007f14977ecad3 in ?? ()
70 LWP 19682 "rpc worker-1968" 0x00007f14977ecad3 in ?? ()
71 LWP 19683 "rpc worker-1968" 0x00007f14977ecad3 in ?? ()
72 LWP 19684 "rpc worker-1968" 0x00007f14977ecad3 in ?? ()
73 LWP 19685 "rpc worker-1968" 0x00007f14977ecad3 in ?? ()
74 LWP 19686 "rpc worker-1968" 0x00007f14977ecad3 in ?? ()
75 LWP 19687 "rpc worker-1968" 0x00007f14977ecad3 in ?? ()
76 LWP 19688 "rpc worker-1968" 0x00007f14977ecad3 in ?? ()
77 LWP 19689 "rpc worker-1968" 0x00007f14977ecad3 in ?? ()
78 LWP 19690 "rpc worker-1969" 0x00007f14977ecad3 in ?? ()
79 LWP 19691 "rpc worker-1969" 0x00007f14977ecad3 in ?? ()
80 LWP 19692 "rpc worker-1969" 0x00007f14977ecad3 in ?? ()
81 LWP 19693 "rpc worker-1969" 0x00007f14977ecad3 in ?? ()
82 LWP 19694 "rpc worker-1969" 0x00007f14977ecad3 in ?? ()
83 LWP 19695 "rpc worker-1969" 0x00007f14977ecad3 in ?? ()
84 LWP 19696 "rpc worker-1969" 0x00007f14977ecad3 in ?? ()
85 LWP 19697 "rpc worker-1969" 0x00007f14977ecad3 in ?? ()
86 LWP 19698 "rpc worker-1969" 0x00007f14977ecad3 in ?? ()
87 LWP 19699 "rpc worker-1969" 0x00007f14977ecad3 in ?? ()
88 LWP 19700 "rpc worker-1970" 0x00007f14977ecad3 in ?? ()
89 LWP 19701 "rpc worker-1970" 0x00007f14977ecad3 in ?? ()
90 LWP 19702 "rpc worker-1970" 0x00007f14977ecad3 in ?? ()
91 LWP 19703 "rpc worker-1970" 0x00007f14977ecad3 in ?? ()
92 LWP 19704 "rpc worker-1970" 0x00007f14977ecad3 in ?? ()
93 LWP 19705 "rpc worker-1970" 0x00007f14977ecad3 in ?? ()
94 LWP 19706 "rpc worker-1970" 0x00007f14977ecad3 in ?? ()
95 LWP 19707 "rpc worker-1970" 0x00007f14977ecad3 in ?? ()
96 LWP 19708 "rpc worker-1970" 0x00007f14977ecad3 in ?? ()
97 LWP 19709 "rpc worker-1970" 0x00007f14977ecad3 in ?? ()
98 LWP 19710 "rpc worker-1971" 0x00007f14977ecad3 in ?? ()
99 LWP 19711 "rpc worker-1971" 0x00007f14977ecad3 in ?? ()
100 LWP 19712 "rpc worker-1971" 0x00007f14977ecad3 in ?? ()
101 LWP 19713 "rpc worker-1971" 0x00007f14977ecad3 in ?? ()
102 LWP 19714 "rpc worker-1971" 0x00007f14977ecad3 in ?? ()
103 LWP 19715 "rpc worker-1971" 0x00007f14977ecad3 in ?? ()
104 LWP 19716 "rpc worker-1971" 0x00007f14977ecad3 in ?? ()
105 LWP 19717 "rpc worker-1971" 0x00007f14977ecad3 in ?? ()
106 LWP 19718 "rpc worker-1971" 0x00007f14977ecad3 in ?? ()
107 LWP 19719 "rpc worker-1971" 0x00007f14977ecad3 in ?? ()
108 LWP 19720 "rpc worker-1972" 0x00007f14977ecad3 in ?? ()
109 LWP 19721 "rpc worker-1972" 0x00007f14977ecad3 in ?? ()
110 LWP 19722 "rpc worker-1972" 0x00007f14977ecad3 in ?? ()
111 LWP 19723 "rpc worker-1972" 0x00007f14977ecad3 in ?? ()
112 LWP 19724 "rpc worker-1972" 0x00007f14977ecad3 in ?? ()
113 LWP 19725 "rpc worker-1972" 0x00007f14977ecad3 in ?? ()
114 LWP 19726 "rpc worker-1972" 0x00007f14977ecad3 in ?? ()
115 LWP 19727 "rpc worker-1972" 0x00007f14977ecad3 in ?? ()
116 LWP 19728 "rpc worker-1972" 0x00007f14977ecad3 in ?? ()
117 LWP 19729 "rpc worker-1972" 0x00007f14977ecad3 in ?? ()
118 LWP 19730 "diag-logger-197" 0x00007f14977ecfb9 in ?? ()
119 LWP 19731 "result-tracker-" 0x00007f14977ecfb9 in ?? ()
120 LWP 19732 "excess-log-dele" 0x00007f14977ecfb9 in ?? ()
121 LWP 19733 "acceptor-19733" 0x00007f1492bf30c7 in ?? ()
122 LWP 19734 "heartbeat-19734" 0x00007f14977ecfb9 in ?? ()
123 LWP 19735 "maintenance_sch" 0x00007f14977ecfb9 in ?? ()
Thread 123 (LWP 19735):
#0 0x00007f14977ecfb9 in ?? ()
#1 0x00007f0100000000 in ?? ()
#2 0x00000000000000ff in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007f144b9b96c0 in ?? ()
#6 0x00000000000001fe in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 19734):
#0 0x00007f14977ecfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 19733):
#0 0x00007f1492bf30c7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 19732):
#0 0x00007f14977ecfb9 in ?? ()
#1 0x00007f144d1bc940 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffe2e089c80 in ?? ()
#5 0x00007f144d1bc7b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 19731):
#0 0x00007f14977ecfb9 in ?? ()
#1 0x0000000085352fb8 in ?? ()
#2 0x0000000000000040 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001008 in ?? ()
#5 0x00007f144d9bd800 in ?? ()
#6 0x0000000000000080 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 19730):
#0 0x00007f14977ecfb9 in ?? ()
#1 0x00007f1490c36008 in ?? ()
#2 0x0000000000000040 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000c90 in ?? ()
#5 0x00007f144e1be750 in ?? ()
#6 0x0000000000000080 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 19729):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 19728):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 19727):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 19726):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 19725):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 19724):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 19723):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 19722):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 19721):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 19720):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 19719):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 19718):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 19717):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 19716):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 19715):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 19714):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 19713):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 19712):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 19711):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 19710):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 19709):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 19708):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 19707):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 19706):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 19705):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 19704):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 19703):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 19702):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 19701):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 19700):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 19699):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 19698):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 19697):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 19696):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 19695):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 19694):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 19693):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 19692):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 19691):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 19690):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 19689):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000630 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b24001147c8 in ?? ()
#4 0x00007f14633ba710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f14633ba730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 76 (LWP 19688):
#0 0x00007f14977ecad3 in ?? ()
#1 0x00000000000007b2 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240010ffc8 in ?? ()
#4 0x00007f1463bbb710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1463bbb730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 19687):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000180 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240010d7d8 in ?? ()
#4 0x00007f14643bc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f14643bc730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 19686):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 19685):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 19684):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 19683):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 19682):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 19681):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 19680):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 19679):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 19678):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 19677):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 19676):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 19675):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 19674):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 19673):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 19672):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 19671):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 19670):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 19669):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000b902c in ?? ()
#4 0x00007f146d7bc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f146d7bc730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x007f0400000026c8 in ?? ()
#9 0x00007f14977ec770 in ?? ()
#10 0x00007f146d7bc730 in ?? ()
#11 0x0002008300000dfe in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 19668):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 19667):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 19666):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 19665):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 19664):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 19663):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 19662):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 19661):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 19660):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 19659):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 19658):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 19657):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 19656):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 19655):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 19654):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 19653):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 19652):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 19651):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 19650):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 19649):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240005ffec in ?? ()
#4 0x00007f1477bbe710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1477bbe730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f14977ec770 in ?? ()
#10 0x00007f1477bbe730 in ?? ()
#11 0x00007f148fd5bc58 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 36 (LWP 19648):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 19647):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 19646):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 19645):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 19644):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 19643):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 19642):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 19641):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 19640):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 19639):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 19638):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 19637):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 19636):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 19635):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 19634):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 19633):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 19632):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 19631):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 19630):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 19629):
#0 0x00007f14977ecfb9 in ?? ()
#1 0x0000000017a335f0 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800003a00 in ?? ()
#5 0x00007f1482192700 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 19628):
#0 0x00007f14977ecfb9 in ?? ()
#1 0x00007f14829939a8 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000372d8 in ?? ()
#5 0x00007f1482993840 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 19627):
#0 0x00007f14977ecfb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007f1483194410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 19626):
#0 0x00007f14977ecad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 19625):
#0 0x00007f1492bf1a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 19624):
#0 0x00007f1492bf1a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 11 (LWP 19623):
#0 0x00007f1492bf1a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 10 (LWP 19622):
#0 0x00007f1492bf1a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 9 (LWP 19619):
#0 0x00007f1492be4cb9 in ?? ()
#1 0x00007f148b9bcc10 in ?? ()
#2 0x00007b040000a850 in ?? ()
#3 0x00007f148b9bdb80 in ?? ()
#4 0x00007f148b9bcc10 in ?? ()
#5 0x00007b040000a850 in ?? ()
#6 0x00000000004888a3 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007f1490666000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007f148b9bdb80 in ?? ()
#10 0x00007f149c5c9908 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 8 (LWP 19618):
#0 0x00007f14977ecfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 7 (LWP 19617):
#0 0x00007f14977f09e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 19610):
#0 0x00007f14977ecfb9 in ?? ()
#1 0x00007f148c9bea40 in ?? ()
#2 0x0000000000000147 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400035b98 in ?? ()
#5 0x00007f148c9be5d0 in ?? ()
#6 0x000000000000028e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 19609):
#0 0x00007f14977ecfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 19608):
#0 0x00007f14977ecfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 19607):
#0 0x00007f14977ecfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 19606):
#0 0x00007f1492bb47a0 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 19605):
#0 0x00007f14977f0d50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x0000000000467b2b in __sanitizer::internal_alloc_placeholder ()
#3 0x00007f1491e12cc0 in ?? ()
#4 0x00007f1491e12cc0 in ?? ()
#5 0x00007ffe2e089a90 in ?? ()
#6 0x000000000048aef4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000a9501f in ?? ()
#9 0x00007f1491e12cc0 in ?? ()
#10 0x00007f1495d18f0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250625 13:59:06.336294 18577 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu with pid 19338
I20250625 13:59:06.389073 18577 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu with pid 19471
I20250625 13:59:06.451881 18577 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu with pid 19605
I20250625 13:59:06.502106 18577 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBL_i2M/build/tsan/bin/kudu with pid 19246
2025-06-25T13:59:06Z chronyd exiting
I20250625 13:59:06.561419 18577 test_util.cc:183] -----------------------------------------------
I20250625 13:59:06.561642 18577 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskBL_i2M/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750859848574329-18577-0
[ FAILED ] TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate (72964 ms)
[----------] 4 tests from TabletCopyITest (97810 ms total)
[----------] 1 test from FaultFlags/BadTabletCopyITest
[ RUN ] FaultFlags/BadTabletCopyITest.TestBadCopy/1
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:1510: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] FaultFlags/BadTabletCopyITest.TestBadCopy/1 (8 ms)
[----------] 1 test from FaultFlags/BadTabletCopyITest (8 ms total)
[----------] Global test environment tear-down
[==========] 5 tests from 2 test suites ran. (97819 ms total)
[ PASSED ] 1 test.
[ SKIPPED ] 3 tests, listed below:
[ SKIPPED ] TabletCopyITest.TestRejectRogueLeader
[ SKIPPED ] TabletCopyITest.TestDeleteLeaderDuringTabletCopyStressTest
[ SKIPPED ] FaultFlags/BadTabletCopyITest.TestBadCopy/1
[ FAILED ] 1 test, listed below:
[ FAILED ] TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate
1 FAILED TEST