Note: This is test shard 1 of 6.
[==========] Running 5 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 4 tests from TabletCopyITest
[ RUN ] TabletCopyITest.TestRejectRogueLeader
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:172: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] TabletCopyITest.TestRejectRogueLeader (10 ms)
[ RUN ] TabletCopyITest.TestDeleteLeaderDuringTabletCopyStressTest
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:727: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] TabletCopyITest.TestDeleteLeaderDuringTabletCopyStressTest (6 ms)
[ RUN ] TabletCopyITest.TestTabletCopyThrottling
2025-06-26T13:57:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T13:57:47Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250626 13:57:47.960390 9576 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
/tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.62:33281
--webserver_interface=127.9.90.62
--webserver_port=0
--builtin_ntp_servers=127.9.90.20:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.62:33281
--master_tombstone_evicted_tablet_replicas=false with env {}
W20250626 13:57:48.253104 9585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:57:48.253743 9585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:57:48.254220 9585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:57:48.285070 9585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 13:57:48.285394 9585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:57:48.285668 9585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 13:57:48.285915 9585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 13:57:48.321584 9585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.20:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/wal
--master_tombstone_evicted_tablet_replicas=false
--ipki_ca_key_size=768
--master_addresses=127.9.90.62:33281
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.62:33281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:57:48.322974 9585 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:57:48.324697 9585 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:57:48.340852 9594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:57:48.342347 9585 server_base.cc:1048] running on GCE node
W20250626 13:57:48.340814 9591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:48.341564 9592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:57:49.531560 9585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:57:49.534395 9585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:57:49.535785 9585 hybrid_clock.cc:648] HybridClock initialized: now 1750946269535745 us; error 57 us; skew 500 ppm
I20250626 13:57:49.536602 9585 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:57:49.542910 9585 webserver.cc:469] Webserver started at http://127.9.90.62:39847/ using document root <none> and password file <none>
I20250626 13:57:49.543887 9585 fs_manager.cc:362] Metadata directory not provided
I20250626 13:57:49.544085 9585 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:57:49.544589 9585 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:57:49.549221 9585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data/instance:
uuid: "ef4a7218a9904c8faab612104d9e9113"
format_stamp: "Formatted at 2025-06-26 13:57:49 on dist-test-slave-92f0"
I20250626 13:57:49.550326 9585 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/wal/instance:
uuid: "ef4a7218a9904c8faab612104d9e9113"
format_stamp: "Formatted at 2025-06-26 13:57:49 on dist-test-slave-92f0"
I20250626 13:57:49.557649 9585 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250626 13:57:49.563385 9601 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:49.564482 9585 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 13:57:49.564841 9585 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data,/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/wal
uuid: "ef4a7218a9904c8faab612104d9e9113"
format_stamp: "Formatted at 2025-06-26 13:57:49 on dist-test-slave-92f0"
I20250626 13:57:49.565162 9585 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:57:49.628432 9585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:57:49.629873 9585 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:57:49.630275 9585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:57:49.709429 9585 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.62:33281
I20250626 13:57:49.709498 9652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.90.62:33281 every 8 connection(s)
I20250626 13:57:49.712117 9585 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data/info.pb
I20250626 13:57:49.717634 9653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:57:49.718997 9576 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu as pid 9585
I20250626 13:57:49.719415 9576 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/wal/instance
I20250626 13:57:49.736889 9653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113: Bootstrap starting.
I20250626 13:57:49.742898 9653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113: Neither blocks nor log segments found. Creating new log.
I20250626 13:57:49.745694 9653 log.cc:826] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113: Log is configured to *not* fsync() on all Append() calls
I20250626 13:57:49.751506 9653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113: No bootstrap required, opened a new log
I20250626 13:57:49.772859 9653 raft_consensus.cc:357] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a7218a9904c8faab612104d9e9113" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 33281 } }
I20250626 13:57:49.773658 9653 raft_consensus.cc:383] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:57:49.773910 9653 raft_consensus.cc:738] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef4a7218a9904c8faab612104d9e9113, State: Initialized, Role: FOLLOWER
I20250626 13:57:49.774658 9653 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a7218a9904c8faab612104d9e9113" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 33281 } }
I20250626 13:57:49.775136 9653 raft_consensus.cc:397] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:57:49.775360 9653 raft_consensus.cc:491] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:57:49.775674 9653 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:57:49.780539 9653 raft_consensus.cc:513] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a7218a9904c8faab612104d9e9113" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 33281 } }
I20250626 13:57:49.781286 9653 leader_election.cc:304] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef4a7218a9904c8faab612104d9e9113; no voters:
I20250626 13:57:49.782915 9653 leader_election.cc:290] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:57:49.785159 9658 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:57:49.787869 9653 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 13:57:49.788123 9658 raft_consensus.cc:695] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 1 LEADER]: Becoming Leader. State: Replica: ef4a7218a9904c8faab612104d9e9113, State: Running, Role: LEADER
I20250626 13:57:49.788965 9658 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a7218a9904c8faab612104d9e9113" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 33281 } }
I20250626 13:57:49.803812 9659 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef4a7218a9904c8faab612104d9e9113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a7218a9904c8faab612104d9e9113" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 33281 } } }
I20250626 13:57:49.804577 9659 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [sys.catalog]: This master's current role is: LEADER
I20250626 13:57:49.806715 9660 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef4a7218a9904c8faab612104d9e9113. Latest consensus state: current_term: 1 leader_uuid: "ef4a7218a9904c8faab612104d9e9113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a7218a9904c8faab612104d9e9113" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 33281 } } }
I20250626 13:57:49.810247 9660 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [sys.catalog]: This master's current role is: LEADER
I20250626 13:57:49.812403 9668 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 13:57:49.825459 9668 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 13:57:49.844527 9668 catalog_manager.cc:1349] Generated new cluster ID: b94ed09f8a844e489e3e844485e9a484
I20250626 13:57:49.844791 9668 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 13:57:49.861702 9668 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 13:57:49.863111 9668 catalog_manager.cc:1506] Loading token signing keys...
I20250626 13:57:49.883342 9668 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113: Generated new TSK 0
I20250626 13:57:49.884465 9668 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 13:57:49.906142 9576 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
/tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.1:0
--local_ip_for_outbound_sockets=127.9.90.1
--webserver_interface=127.9.90.1
--webserver_port=0
--tserver_master_addrs=127.9.90.62:33281
--builtin_ntp_servers=127.9.90.20:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--num_tablets_to_copy_simultaneously=1 with env {}
W20250626 13:57:50.231388 9677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:57:50.231966 9677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:57:50.232650 9677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:57:50.264370 9677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:57:50.265478 9677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.1
I20250626 13:57:50.301349 9677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.20:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.1
--webserver_port=0
--tserver_master_addrs=127.9.90.62:33281
--num_tablets_to_copy_simultaneously=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.1
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:57:50.302713 9677 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:57:50.304551 9677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:57:50.323757 9684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:50.324189 9686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:50.327873 9683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:51.503723 9685 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:57:51.503866 9677 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:57:51.508303 9677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:57:51.510991 9677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:57:51.512493 9677 hybrid_clock.cc:648] HybridClock initialized: now 1750946271512439 us; error 58 us; skew 500 ppm
I20250626 13:57:51.513558 9677 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:57:51.521114 9677 webserver.cc:469] Webserver started at http://127.9.90.1:43399/ using document root <none> and password file <none>
I20250626 13:57:51.522325 9677 fs_manager.cc:362] Metadata directory not provided
I20250626 13:57:51.522598 9677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:57:51.523224 9677 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:57:51.527858 9677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/data/instance:
uuid: "73a1bdf6d24d4917a92b5c94c1cf365b"
format_stamp: "Formatted at 2025-06-26 13:57:51 on dist-test-slave-92f0"
I20250626 13:57:51.528951 9677 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/wal/instance:
uuid: "73a1bdf6d24d4917a92b5c94c1cf365b"
format_stamp: "Formatted at 2025-06-26 13:57:51 on dist-test-slave-92f0"
I20250626 13:57:51.536377 9677 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250626 13:57:51.543720 9693 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:51.544831 9677 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20250626 13:57:51.545141 9677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/wal
uuid: "73a1bdf6d24d4917a92b5c94c1cf365b"
format_stamp: "Formatted at 2025-06-26 13:57:51 on dist-test-slave-92f0"
I20250626 13:57:51.545497 9677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:57:51.606457 9677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:57:51.608019 9677 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:57:51.608461 9677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:57:51.611513 9677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:57:51.615741 9677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:57:51.615938 9677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:51.616233 9677 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:57:51.616387 9677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:51.773819 9677 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.1:44111
I20250626 13:57:51.773918 9805 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.90.1:44111 every 8 connection(s)
I20250626 13:57:51.777179 9677 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/data/info.pb
I20250626 13:57:51.786909 9576 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu as pid 9677
I20250626 13:57:51.787281 9576 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-0/wal/instance
I20250626 13:57:51.792870 9576 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
/tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.2:0
--local_ip_for_outbound_sockets=127.9.90.2
--webserver_interface=127.9.90.2
--webserver_port=0
--tserver_master_addrs=127.9.90.62:33281
--builtin_ntp_servers=127.9.90.20:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--num_tablets_to_copy_simultaneously=1 with env {}
I20250626 13:57:51.800284 9806 heartbeater.cc:344] Connected to a master server at 127.9.90.62:33281
I20250626 13:57:51.800834 9806 heartbeater.cc:461] Registering TS with master...
I20250626 13:57:51.802698 9806 heartbeater.cc:507] Master 127.9.90.62:33281 requested a full tablet report, sending...
I20250626 13:57:51.806010 9618 ts_manager.cc:194] Registered new tserver with Master: 73a1bdf6d24d4917a92b5c94c1cf365b (127.9.90.1:44111)
I20250626 13:57:51.809129 9618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.1:41177
W20250626 13:57:52.100370 9810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:57:52.100857 9810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:57:52.101344 9810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:57:52.132381 9810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:57:52.133209 9810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.2
I20250626 13:57:52.167865 9810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.20:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.2
--webserver_port=0
--tserver_master_addrs=127.9.90.62:33281
--num_tablets_to_copy_simultaneously=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.2
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:57:52.169145 9810 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:57:52.170755 9810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:57:52.188517 9817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:57:52.814014 9806 heartbeater.cc:499] Master 127.9.90.62:33281 was elected leader, sending a full tablet report...
W20250626 13:57:52.189558 9819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:53.361089 9818 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:57:53.361480 9810 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250626 13:57:52.188524 9816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:57:53.366369 9810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:57:53.369205 9810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:57:53.370641 9810 hybrid_clock.cc:648] HybridClock initialized: now 1750946273370607 us; error 63 us; skew 500 ppm
I20250626 13:57:53.371510 9810 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:57:53.378125 9810 webserver.cc:469] Webserver started at http://127.9.90.2:38369/ using document root <none> and password file <none>
I20250626 13:57:53.379046 9810 fs_manager.cc:362] Metadata directory not provided
I20250626 13:57:53.379318 9810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:57:53.379797 9810 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:57:53.384908 9810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data/instance:
uuid: "d1abec696a6646e69480089ecc381245"
format_stamp: "Formatted at 2025-06-26 13:57:53 on dist-test-slave-92f0"
I20250626 13:57:53.386014 9810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/wal/instance:
uuid: "d1abec696a6646e69480089ecc381245"
format_stamp: "Formatted at 2025-06-26 13:57:53 on dist-test-slave-92f0"
I20250626 13:57:53.393198 9810 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250626 13:57:53.399348 9826 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:53.400516 9810 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 13:57:53.400832 9810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/wal
uuid: "d1abec696a6646e69480089ecc381245"
format_stamp: "Formatted at 2025-06-26 13:57:53 on dist-test-slave-92f0"
I20250626 13:57:53.401165 9810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:57:53.458851 9810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:57:53.460851 9810 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:57:53.461319 9810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:57:53.463817 9810 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:57:53.468274 9810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:57:53.468499 9810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:53.468744 9810 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:57:53.468905 9810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:53.623096 9810 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.2:41503
I20250626 13:57:53.623222 9938 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.90.2:41503 every 8 connection(s)
I20250626 13:57:53.625804 9810 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data/info.pb
I20250626 13:57:53.634210 9576 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu as pid 9810
I20250626 13:57:53.634621 9576 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/wal/instance
I20250626 13:57:53.649221 9939 heartbeater.cc:344] Connected to a master server at 127.9.90.62:33281
I20250626 13:57:53.649758 9939 heartbeater.cc:461] Registering TS with master...
I20250626 13:57:53.650765 9939 heartbeater.cc:507] Master 127.9.90.62:33281 requested a full tablet report, sending...
I20250626 13:57:53.653090 9618 ts_manager.cc:194] Registered new tserver with Master: d1abec696a6646e69480089ecc381245 (127.9.90.2:41503)
I20250626 13:57:53.654475 9618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.2:45293
I20250626 13:57:53.655237 9576 external_mini_cluster.cc:934] 2 TS(s) registered with all masters
I20250626 13:57:53.690644 9576 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu with pid 9810
I20250626 13:57:53.713796 9576 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu with pid 9585
I20250626 13:57:53.743934 9576 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
/tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.62:33281
--webserver_interface=127.9.90.62
--webserver_port=39847
--builtin_ntp_servers=127.9.90.20:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.62:33281
--master_tombstone_evicted_tablet_replicas=false with env {}
W20250626 13:57:53.822484 9806 heartbeater.cc:646] Failed to heartbeat to 127.9.90.62:33281 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.90.62:33281: connect: Connection refused (error 111)
W20250626 13:57:55.292474 9951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:57:55.293108 9951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:57:55.293594 9951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:57:55.325476 9951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 13:57:55.325839 9951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:57:55.326112 9951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 13:57:55.326371 9951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 13:57:55.364027 9951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.20:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/wal
--master_tombstone_evicted_tablet_replicas=false
--ipki_ca_key_size=768
--master_addresses=127.9.90.62:33281
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.62:33281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.62
--webserver_port=39847
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:57:55.365429 9951 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:57:55.367167 9951 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:57:55.382400 9960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:55.382392 9958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:55.383296 9957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:57:56.565651 9959 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:57:56.565796 9951 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:57:56.569828 9951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:57:56.572456 9951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:57:56.573817 9951 hybrid_clock.cc:648] HybridClock initialized: now 1750946276573777 us; error 59 us; skew 500 ppm
I20250626 13:57:56.574606 9951 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:57:56.585234 9951 webserver.cc:469] Webserver started at http://127.9.90.62:39847/ using document root <none> and password file <none>
I20250626 13:57:56.586171 9951 fs_manager.cc:362] Metadata directory not provided
I20250626 13:57:56.586395 9951 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:57:56.594214 9951 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250626 13:57:56.598886 9968 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:57:56.600018 9951 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20250626 13:57:56.600319 9951 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data,/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/wal
uuid: "ef4a7218a9904c8faab612104d9e9113"
format_stamp: "Formatted at 2025-06-26 13:57:49 on dist-test-slave-92f0"
I20250626 13:57:56.602281 9951 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:57:56.661098 9951 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:57:56.662609 9951 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:57:56.663017 9951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:57:56.733424 9951 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.62:33281
I20250626 13:57:56.733530 10019 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.90.62:33281 every 8 connection(s)
I20250626 13:57:56.736328 9951 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/master-0/data/info.pb
I20250626 13:57:56.744640 9576 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu as pid 9951
I20250626 13:57:56.746613 10020 sys_catalog.cc:263] Verifying existing consensus state
I20250626 13:57:56.754575 10020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113: Bootstrap starting.
I20250626 13:57:56.801223 10020 log.cc:826] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113: Log is configured to *not* fsync() on all Append() calls
I20250626 13:57:56.817090 10020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:57:56.817874 10020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113: Bootstrap complete.
I20250626 13:57:56.839743 10020 raft_consensus.cc:357] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a7218a9904c8faab612104d9e9113" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 33281 } }
I20250626 13:57:56.842722 10020 raft_consensus.cc:738] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef4a7218a9904c8faab612104d9e9113, State: Initialized, Role: FOLLOWER
I20250626 13:57:56.843145 9806 heartbeater.cc:344] Connected to a master server at 127.9.90.62:33281
I20250626 13:57:56.843613 10020 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a7218a9904c8faab612104d9e9113" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 33281 } }
I20250626 13:57:56.844187 10020 raft_consensus.cc:397] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:57:56.844481 10020 raft_consensus.cc:491] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:57:56.844797 10020 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 1 FOLLOWER]: Advancing to term 2
I20250626 13:57:56.850762 10020 raft_consensus.cc:513] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a7218a9904c8faab612104d9e9113" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 33281 } }
I20250626 13:57:56.851681 10020 leader_election.cc:304] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef4a7218a9904c8faab612104d9e9113; no voters:
I20250626 13:57:56.853664 10020 leader_election.cc:290] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250626 13:57:56.854107 10026 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 13:57:56.856712 10026 raft_consensus.cc:695] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [term 2 LEADER]: Becoming Leader. State: Replica: ef4a7218a9904c8faab612104d9e9113, State: Running, Role: LEADER
I20250626 13:57:56.857503 10026 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a7218a9904c8faab612104d9e9113" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 33281 } }
I20250626 13:57:56.858421 10020 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 13:57:56.868062 10028 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef4a7218a9904c8faab612104d9e9113. Latest consensus state: current_term: 2 leader_uuid: "ef4a7218a9904c8faab612104d9e9113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a7218a9904c8faab612104d9e9113" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 33281 } } }
I20250626 13:57:56.868814 10028 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [sys.catalog]: This master's current role is: LEADER
I20250626 13:57:56.869926 10027 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ef4a7218a9904c8faab612104d9e9113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4a7218a9904c8faab612104d9e9113" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 33281 } } }
I20250626 13:57:56.870434 10027 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113 [sys.catalog]: This master's current role is: LEADER
I20250626 13:57:56.872418 10036 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 13:57:56.886559 10036 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 13:57:56.891656 10036 catalog_manager.cc:1261] Loaded cluster ID: b94ed09f8a844e489e3e844485e9a484
I20250626 13:57:56.891950 10036 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 13:57:56.899312 10036 catalog_manager.cc:1506] Loading token signing keys...
I20250626 13:57:56.904067 10036 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ef4a7218a9904c8faab612104d9e9113: Loaded TSK: 0
I20250626 13:57:56.905651 10036 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 13:57:57.851311 9985 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" instance_seqno: 1750946271736790) as {username='slave'} at 127.9.90.1:55117; Asking this server to re-register.
I20250626 13:57:57.852867 9806 heartbeater.cc:461] Registering TS with master...
I20250626 13:57:57.853420 9806 heartbeater.cc:507] Master 127.9.90.62:33281 requested a full tablet report, sending...
I20250626 13:57:57.855713 9985 ts_manager.cc:194] Registered new tserver with Master: 73a1bdf6d24d4917a92b5c94c1cf365b (127.9.90.1:44111)
I20250626 13:57:57.857715 9576 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250626 13:57:57.858090 9576 test_util.cc:276] Using random seed: -219571439
I20250626 13:57:57.914603 9985 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56468:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 13:57:57.979830 9741 tablet_service.cc:1468] Processing CreateTablet for tablet 1aacb69070df4b04844add935d88a48c (DEFAULT_TABLE table=test-workload [id=57ff1490d85e436eb6724b0ea5964038]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 13:57:57.979841 9740 tablet_service.cc:1468] Processing CreateTablet for tablet 3b7987a9eca34f43a9befc607121a68a (DEFAULT_TABLE table=test-workload [id=57ff1490d85e436eb6724b0ea5964038]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 13:57:57.980125 9738 tablet_service.cc:1468] Processing CreateTablet for tablet 73ed8a803e544fe190d6b145c790c580 (DEFAULT_TABLE table=test-workload [id=57ff1490d85e436eb6724b0ea5964038]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 13:57:57.980789 9739 tablet_service.cc:1468] Processing CreateTablet for tablet b6581f4311fd4dceb376128a8fdf1cd4 (DEFAULT_TABLE table=test-workload [id=57ff1490d85e436eb6724b0ea5964038]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 13:57:57.982780 9741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aacb69070df4b04844add935d88a48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:57:57.983636 9740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7987a9eca34f43a9befc607121a68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:57:57.984228 9739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6581f4311fd4dceb376128a8fdf1cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:57:57.989797 9738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73ed8a803e544fe190d6b145c790c580. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:57:58.017648 10057 tablet_bootstrap.cc:492] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b: Bootstrap starting.
I20250626 13:57:58.025641 10057 tablet_bootstrap.cc:654] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b: Neither blocks nor log segments found. Creating new log.
I20250626 13:57:58.028038 10057 log.cc:826] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b: Log is configured to *not* fsync() on all Append() calls
I20250626 13:57:58.033653 10057 tablet_bootstrap.cc:492] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b: No bootstrap required, opened a new log
I20250626 13:57:58.034195 10057 ts_tablet_manager.cc:1397] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20250626 13:57:58.060912 10057 raft_consensus.cc:357] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.061728 10057 raft_consensus.cc:383] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:57:58.062033 10057 raft_consensus.cc:738] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1bdf6d24d4917a92b5c94c1cf365b, State: Initialized, Role: FOLLOWER
I20250626 13:57:58.063345 10057 consensus_queue.cc:260] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.064121 10057 raft_consensus.cc:397] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:57:58.064479 10057 raft_consensus.cc:491] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:57:58.064883 10057 raft_consensus.cc:3058] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:57:58.071669 10057 raft_consensus.cc:513] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.072794 10057 leader_election.cc:304] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73a1bdf6d24d4917a92b5c94c1cf365b; no voters:
I20250626 13:57:58.074923 10057 leader_election.cc:290] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:57:58.075394 10059 raft_consensus.cc:2802] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:57:58.079710 10057 ts_tablet_manager.cc:1428] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b: Time spent starting tablet: real 0.045s user 0.037s sys 0.008s
I20250626 13:57:58.080595 10057 tablet_bootstrap.cc:492] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b: Bootstrap starting.
I20250626 13:57:58.083323 10059 raft_consensus.cc:695] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 1 LEADER]: Becoming Leader. State: Replica: 73a1bdf6d24d4917a92b5c94c1cf365b, State: Running, Role: LEADER
I20250626 13:57:58.084084 10059 consensus_queue.cc:237] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.086690 10057 tablet_bootstrap.cc:654] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b: Neither blocks nor log segments found. Creating new log.
I20250626 13:57:58.098742 10057 tablet_bootstrap.cc:492] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b: No bootstrap required, opened a new log
I20250626 13:57:58.099170 10057 ts_tablet_manager.cc:1397] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20250626 13:57:58.100339 9985 catalog_manager.cc:5582] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1bdf6d24d4917a92b5c94c1cf365b (127.9.90.1). New cstate: current_term: 1 leader_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } health_report { overall_health: HEALTHY } } }
I20250626 13:57:58.101740 10057 raft_consensus.cc:357] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.102368 10057 raft_consensus.cc:383] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:57:58.102633 10057 raft_consensus.cc:738] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1bdf6d24d4917a92b5c94c1cf365b, State: Initialized, Role: FOLLOWER
I20250626 13:57:58.103217 10057 consensus_queue.cc:260] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.103876 10057 raft_consensus.cc:397] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:57:58.104179 10057 raft_consensus.cc:491] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:57:58.104480 10057 raft_consensus.cc:3058] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:57:58.110846 10057 raft_consensus.cc:513] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.111548 10057 leader_election.cc:304] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73a1bdf6d24d4917a92b5c94c1cf365b; no voters:
I20250626 13:57:58.112125 10057 leader_election.cc:290] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:57:58.112288 10059 raft_consensus.cc:2802] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:57:58.113101 10059 raft_consensus.cc:695] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b [term 1 LEADER]: Becoming Leader. State: Replica: 73a1bdf6d24d4917a92b5c94c1cf365b, State: Running, Role: LEADER
I20250626 13:57:58.113832 10057 ts_tablet_manager.cc:1428] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20250626 13:57:58.113776 10059 consensus_queue.cc:237] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.114667 10057 tablet_bootstrap.cc:492] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b: Bootstrap starting.
I20250626 13:57:58.121131 10057 tablet_bootstrap.cc:654] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b: Neither blocks nor log segments found. Creating new log.
I20250626 13:57:58.121865 9985 catalog_manager.cc:5582] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1bdf6d24d4917a92b5c94c1cf365b (127.9.90.1). New cstate: current_term: 1 leader_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } health_report { overall_health: HEALTHY } } }
I20250626 13:57:58.127146 10057 tablet_bootstrap.cc:492] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b: No bootstrap required, opened a new log
I20250626 13:57:58.127624 10057 ts_tablet_manager.cc:1397] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.004s
I20250626 13:57:58.130040 10057 raft_consensus.cc:357] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.130589 10057 raft_consensus.cc:383] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:57:58.130867 10057 raft_consensus.cc:738] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1bdf6d24d4917a92b5c94c1cf365b, State: Initialized, Role: FOLLOWER
I20250626 13:57:58.131937 10057 consensus_queue.cc:260] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.132579 10057 raft_consensus.cc:397] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:57:58.132871 10057 raft_consensus.cc:491] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:57:58.133188 10057 raft_consensus.cc:3058] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:57:58.139565 10057 raft_consensus.cc:513] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.140246 10057 leader_election.cc:304] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73a1bdf6d24d4917a92b5c94c1cf365b; no voters:
I20250626 13:57:58.140872 10057 leader_election.cc:290] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:57:58.140998 10059 raft_consensus.cc:2802] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:57:58.141880 10059 raft_consensus.cc:695] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b [term 1 LEADER]: Becoming Leader. State: Replica: 73a1bdf6d24d4917a92b5c94c1cf365b, State: Running, Role: LEADER
I20250626 13:57:58.142403 10059 consensus_queue.cc:237] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.142596 10057 ts_tablet_manager.cc:1428] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20250626 13:57:58.143375 10057 tablet_bootstrap.cc:492] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b: Bootstrap starting.
I20250626 13:57:58.149320 9985 catalog_manager.cc:5582] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1bdf6d24d4917a92b5c94c1cf365b (127.9.90.1). New cstate: current_term: 1 leader_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } health_report { overall_health: HEALTHY } } }
I20250626 13:57:58.149869 10057 tablet_bootstrap.cc:654] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b: Neither blocks nor log segments found. Creating new log.
I20250626 13:57:58.155952 10057 tablet_bootstrap.cc:492] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b: No bootstrap required, opened a new log
I20250626 13:57:58.156481 10057 ts_tablet_manager.cc:1397] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.009s
I20250626 13:57:58.158922 10057 raft_consensus.cc:357] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.159495 10057 raft_consensus.cc:383] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:57:58.159879 10057 raft_consensus.cc:738] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a1bdf6d24d4917a92b5c94c1cf365b, State: Initialized, Role: FOLLOWER
I20250626 13:57:58.160481 10057 consensus_queue.cc:260] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.160939 10057 raft_consensus.cc:397] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:57:58.161273 10057 raft_consensus.cc:491] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:57:58.161562 10057 raft_consensus.cc:3058] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:57:58.166427 10057 raft_consensus.cc:513] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.167191 10057 leader_election.cc:304] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73a1bdf6d24d4917a92b5c94c1cf365b; no voters:
I20250626 13:57:58.167909 10059 raft_consensus.cc:2802] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:57:58.168246 10059 raft_consensus.cc:695] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b [term 1 LEADER]: Becoming Leader. State: Replica: 73a1bdf6d24d4917a92b5c94c1cf365b, State: Running, Role: LEADER
I20250626 13:57:58.168823 10059 consensus_queue.cc:237] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:57:58.171241 10057 leader_election.cc:290] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:57:58.173434 10057 ts_tablet_manager.cc:1428] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b: Time spent starting tablet: real 0.017s user 0.010s sys 0.002s
I20250626 13:57:58.175133 9985 catalog_manager.cc:5582] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b reported cstate change: term changed from 0 to 1, leader changed from <none> to 73a1bdf6d24d4917a92b5c94c1cf365b (127.9.90.1). New cstate: current_term: 1 leader_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } health_report { overall_health: HEALTHY } } }
W20250626 13:57:59.723603 10067 log.cc:927] Time spent T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250626 13:58:02.752061 9576 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
/tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.2:41503
--local_ip_for_outbound_sockets=127.9.90.2
--tserver_master_addrs=127.9.90.62:33281
--webserver_port=38369
--webserver_interface=127.9.90.2
--builtin_ntp_servers=127.9.90.20:37975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--num_tablets_to_copy_simultaneously=1 with env {}
W20250626 13:58:03.084393 10091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:03.084877 10091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:03.085361 10091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:03.117247 10091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:03.118687 10091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.2
I20250626 13:58:03.153247 10091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.20:37975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.2:41503
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.2
--webserver_port=38369
--tserver_master_addrs=127.9.90.62:33281
--num_tablets_to_copy_simultaneously=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.2
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:03.154543 10091 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:03.156172 10091 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:03.175076 10098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:03.175175 10097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:03.177520 10091 server_base.cc:1048] running on GCE node
W20250626 13:58:03.175086 10100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:04.348513 10091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:04.350826 10091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:04.352205 10091 hybrid_clock.cc:648] HybridClock initialized: now 1750946284352158 us; error 62 us; skew 500 ppm
I20250626 13:58:04.353076 10091 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:04.359710 10091 webserver.cc:469] Webserver started at http://127.9.90.2:38369/ using document root <none> and password file <none>
I20250626 13:58:04.360670 10091 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:04.360915 10091 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:04.369088 10091 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250626 13:58:04.374152 10107 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:04.375294 10091 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250626 13:58:04.375663 10091 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/wal
uuid: "d1abec696a6646e69480089ecc381245"
format_stamp: "Formatted at 2025-06-26 13:57:53 on dist-test-slave-92f0"
I20250626 13:58:04.377660 10091 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:04.432269 10091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:04.433753 10091 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:04.434193 10091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:04.437280 10091 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:04.441761 10091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:04.442041 10091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:04.442312 10091 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:04.442484 10091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:04.598807 10091 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.2:41503
I20250626 13:58:04.598927 10219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.90.2:41503 every 8 connection(s)
I20250626 13:58:04.601438 10091 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750946267712444-9576-0/minicluster-data/ts-1/data/info.pb
I20250626 13:58:04.611156 9576 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu as pid 10091
I20250626 13:58:04.625463 10220 heartbeater.cc:344] Connected to a master server at 127.9.90.62:33281
I20250626 13:58:04.625990 10220 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:04.627260 10220 heartbeater.cc:507] Master 127.9.90.62:33281 requested a full tablet report, sending...
I20250626 13:58:04.630285 9985 ts_manager.cc:194] Registered new tserver with Master: d1abec696a6646e69480089ecc381245 (127.9.90.2:41503)
I20250626 13:58:04.630573 10226 ts_tablet_manager.cc:927] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245: Initiating tablet copy from peer 73a1bdf6d24d4917a92b5c94c1cf365b (127.9.90.1:44111)
I20250626 13:58:04.631911 9985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.2:41877
I20250626 13:58:04.633845 10226 tablet_copy_client.cc:323] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245: tablet copy: Beginning tablet copy session from remote peer at address 127.9.90.1:44111
I20250626 13:58:04.642666 9781 tablet_copy_service.cc:140] P 73a1bdf6d24d4917a92b5c94c1cf365b: Received BeginTabletCopySession request for tablet 1aacb69070df4b04844add935d88a48c from peer d1abec696a6646e69480089ecc381245 ({username='slave'} at 127.9.90.2:41085)
I20250626 13:58:04.643149 9781 tablet_copy_service.cc:161] P 73a1bdf6d24d4917a92b5c94c1cf365b: Beginning new tablet copy session on tablet 1aacb69070df4b04844add935d88a48c from peer d1abec696a6646e69480089ecc381245 at {username='slave'} at 127.9.90.2:41085: session id = d1abec696a6646e69480089ecc381245-1aacb69070df4b04844add935d88a48c
I20250626 13:58:04.648875 9781 tablet_copy_source_session.cc:215] T 1aacb69070df4b04844add935d88a48c P 73a1bdf6d24d4917a92b5c94c1cf365b: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 13:58:04.653839 10226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aacb69070df4b04844add935d88a48c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:04.672885 10226 tablet_copy_client.cc:806] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245: tablet copy: Starting download of 0 data blocks...
I20250626 13:58:04.673460 10226 tablet_copy_client.cc:670] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245: tablet copy: Starting download of 1 WAL segments...
I20250626 13:58:04.687484 10226 tablet_copy_client.cc:538] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 13:58:04.694454 10226 tablet_bootstrap.cc:492] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245: Bootstrap starting.
I20250626 13:58:04.848069 10226 log.cc:826] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:05.635139 10220 heartbeater.cc:499] Master 127.9.90.62:33281 was elected leader, sending a full tablet report...
I20250626 13:58:05.791245 10226 tablet_bootstrap.cc:492] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245: Bootstrap replayed 1/1 log segments. Stats: ops{read=214 overwritten=0 applied=214 ignored=0} inserts{seen=2622 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:58:05.792186 10226 tablet_bootstrap.cc:492] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245: Bootstrap complete.
I20250626 13:58:05.792964 10226 ts_tablet_manager.cc:1397] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245: Time spent bootstrapping tablet: real 1.099s user 1.088s sys 0.010s
I20250626 13:58:05.807912 10226 raft_consensus.cc:357] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:58:05.808521 10226 raft_consensus.cc:738] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: d1abec696a6646e69480089ecc381245, State: Initialized, Role: NON_PARTICIPANT
I20250626 13:58:05.809222 10226 consensus_queue.cc:260] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 214, Last appended: 1.214, Last appended by leader: 214, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:58:05.812264 10226 ts_tablet_manager.cc:1428] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245: Time spent starting tablet: real 0.019s user 0.020s sys 0.000s
I20250626 13:58:05.813815 9781 tablet_copy_service.cc:342] P 73a1bdf6d24d4917a92b5c94c1cf365b: Request end of tablet copy session d1abec696a6646e69480089ecc381245-1aacb69070df4b04844add935d88a48c received from {username='slave'} at 127.9.90.2:41085
I20250626 13:58:05.814226 9781 tablet_copy_service.cc:434] P 73a1bdf6d24d4917a92b5c94c1cf365b: ending tablet copy session d1abec696a6646e69480089ecc381245-1aacb69070df4b04844add935d88a48c on tablet 1aacb69070df4b04844add935d88a48c with peer d1abec696a6646e69480089ecc381245
I20250626 13:58:05.817441 10226 ts_tablet_manager.cc:927] T 73ed8a803e544fe190d6b145c790c580 P d1abec696a6646e69480089ecc381245: Initiating tablet copy from peer 73a1bdf6d24d4917a92b5c94c1cf365b (127.9.90.1:44111)
I20250626 13:58:05.819036 10226 tablet_copy_client.cc:323] T 73ed8a803e544fe190d6b145c790c580 P d1abec696a6646e69480089ecc381245: tablet copy: Beginning tablet copy session from remote peer at address 127.9.90.1:44111
I20250626 13:58:05.820169 9781 tablet_copy_service.cc:140] P 73a1bdf6d24d4917a92b5c94c1cf365b: Received BeginTabletCopySession request for tablet 73ed8a803e544fe190d6b145c790c580 from peer d1abec696a6646e69480089ecc381245 ({username='slave'} at 127.9.90.2:41085)
I20250626 13:58:05.820544 9781 tablet_copy_service.cc:161] P 73a1bdf6d24d4917a92b5c94c1cf365b: Beginning new tablet copy session on tablet 73ed8a803e544fe190d6b145c790c580 from peer d1abec696a6646e69480089ecc381245 at {username='slave'} at 127.9.90.2:41085: session id = d1abec696a6646e69480089ecc381245-73ed8a803e544fe190d6b145c790c580
I20250626 13:58:05.825011 9781 tablet_copy_source_session.cc:215] T 73ed8a803e544fe190d6b145c790c580 P 73a1bdf6d24d4917a92b5c94c1cf365b: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 13:58:05.826963 10226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73ed8a803e544fe190d6b145c790c580. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:05.836129 10226 tablet_copy_client.cc:806] T 73ed8a803e544fe190d6b145c790c580 P d1abec696a6646e69480089ecc381245: tablet copy: Starting download of 0 data blocks...
I20250626 13:58:05.836540 10226 tablet_copy_client.cc:670] T 73ed8a803e544fe190d6b145c790c580 P d1abec696a6646e69480089ecc381245: tablet copy: Starting download of 1 WAL segments...
I20250626 13:58:05.848285 10226 tablet_copy_client.cc:538] T 73ed8a803e544fe190d6b145c790c580 P d1abec696a6646e69480089ecc381245: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 13:58:05.854244 10226 tablet_bootstrap.cc:492] T 73ed8a803e544fe190d6b145c790c580 P d1abec696a6646e69480089ecc381245: Bootstrap starting.
I20250626 13:58:06.929277 10226 tablet_bootstrap.cc:492] T 73ed8a803e544fe190d6b145c790c580 P d1abec696a6646e69480089ecc381245: Bootstrap replayed 1/1 log segments. Stats: ops{read=214 overwritten=0 applied=214 ignored=0} inserts{seen=2719 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:58:06.930025 10226 tablet_bootstrap.cc:492] T 73ed8a803e544fe190d6b145c790c580 P d1abec696a6646e69480089ecc381245: Bootstrap complete.
I20250626 13:58:06.930564 10226 ts_tablet_manager.cc:1397] T 73ed8a803e544fe190d6b145c790c580 P d1abec696a6646e69480089ecc381245: Time spent bootstrapping tablet: real 1.076s user 1.051s sys 0.024s
I20250626 13:58:06.932569 10226 raft_consensus.cc:357] T 73ed8a803e544fe190d6b145c790c580 P d1abec696a6646e69480089ecc381245 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:58:06.932924 10226 raft_consensus.cc:738] T 73ed8a803e544fe190d6b145c790c580 P d1abec696a6646e69480089ecc381245 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: d1abec696a6646e69480089ecc381245, State: Initialized, Role: NON_PARTICIPANT
I20250626 13:58:06.933310 10226 consensus_queue.cc:260] T 73ed8a803e544fe190d6b145c790c580 P d1abec696a6646e69480089ecc381245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 214, Last appended: 1.214, Last appended by leader: 214, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:58:06.935603 10226 ts_tablet_manager.cc:1428] T 73ed8a803e544fe190d6b145c790c580 P d1abec696a6646e69480089ecc381245: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20250626 13:58:06.937016 9781 tablet_copy_service.cc:342] P 73a1bdf6d24d4917a92b5c94c1cf365b: Request end of tablet copy session d1abec696a6646e69480089ecc381245-73ed8a803e544fe190d6b145c790c580 received from {username='slave'} at 127.9.90.2:41085
I20250626 13:58:06.937393 9781 tablet_copy_service.cc:434] P 73a1bdf6d24d4917a92b5c94c1cf365b: ending tablet copy session d1abec696a6646e69480089ecc381245-73ed8a803e544fe190d6b145c790c580 on tablet 73ed8a803e544fe190d6b145c790c580 with peer d1abec696a6646e69480089ecc381245
W20250626 13:58:06.939503 10226 ts_tablet_manager.cc:726] T 73ed8a803e544fe190d6b145c790c580 P d1abec696a6646e69480089ecc381245: Tablet Copy: Invalid argument: Leader has replica of tablet 73ed8a803e544fe190d6b145c790c580 with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
I20250626 13:58:06.945489 10226 ts_tablet_manager.cc:927] T b6581f4311fd4dceb376128a8fdf1cd4 P d1abec696a6646e69480089ecc381245: Initiating tablet copy from peer 73a1bdf6d24d4917a92b5c94c1cf365b (127.9.90.1:44111)
I20250626 13:58:06.946951 10226 tablet_copy_client.cc:323] T b6581f4311fd4dceb376128a8fdf1cd4 P d1abec696a6646e69480089ecc381245: tablet copy: Beginning tablet copy session from remote peer at address 127.9.90.1:44111
I20250626 13:58:06.948140 9781 tablet_copy_service.cc:140] P 73a1bdf6d24d4917a92b5c94c1cf365b: Received BeginTabletCopySession request for tablet b6581f4311fd4dceb376128a8fdf1cd4 from peer d1abec696a6646e69480089ecc381245 ({username='slave'} at 127.9.90.2:41085)
I20250626 13:58:06.948503 9781 tablet_copy_service.cc:161] P 73a1bdf6d24d4917a92b5c94c1cf365b: Beginning new tablet copy session on tablet b6581f4311fd4dceb376128a8fdf1cd4 from peer d1abec696a6646e69480089ecc381245 at {username='slave'} at 127.9.90.2:41085: session id = d1abec696a6646e69480089ecc381245-b6581f4311fd4dceb376128a8fdf1cd4
I20250626 13:58:06.952927 9781 tablet_copy_source_session.cc:215] T b6581f4311fd4dceb376128a8fdf1cd4 P 73a1bdf6d24d4917a92b5c94c1cf365b: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 13:58:06.955018 10226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6581f4311fd4dceb376128a8fdf1cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:06.963071 10226 tablet_copy_client.cc:806] T b6581f4311fd4dceb376128a8fdf1cd4 P d1abec696a6646e69480089ecc381245: tablet copy: Starting download of 0 data blocks...
I20250626 13:58:06.963624 10226 tablet_copy_client.cc:670] T b6581f4311fd4dceb376128a8fdf1cd4 P d1abec696a6646e69480089ecc381245: tablet copy: Starting download of 1 WAL segments...
I20250626 13:58:06.975797 10226 tablet_copy_client.cc:538] T b6581f4311fd4dceb376128a8fdf1cd4 P d1abec696a6646e69480089ecc381245: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 13:58:06.983944 10226 tablet_bootstrap.cc:492] T b6581f4311fd4dceb376128a8fdf1cd4 P d1abec696a6646e69480089ecc381245: Bootstrap starting.
I20250626 13:58:08.059309 10226 tablet_bootstrap.cc:492] T b6581f4311fd4dceb376128a8fdf1cd4 P d1abec696a6646e69480089ecc381245: Bootstrap replayed 1/1 log segments. Stats: ops{read=214 overwritten=0 applied=214 ignored=0} inserts{seen=2599 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:58:08.060082 10226 tablet_bootstrap.cc:492] T b6581f4311fd4dceb376128a8fdf1cd4 P d1abec696a6646e69480089ecc381245: Bootstrap complete.
I20250626 13:58:08.060540 10226 ts_tablet_manager.cc:1397] T b6581f4311fd4dceb376128a8fdf1cd4 P d1abec696a6646e69480089ecc381245: Time spent bootstrapping tablet: real 1.077s user 1.055s sys 0.020s
I20250626 13:58:08.062119 10226 raft_consensus.cc:357] T b6581f4311fd4dceb376128a8fdf1cd4 P d1abec696a6646e69480089ecc381245 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:58:08.062467 10226 raft_consensus.cc:738] T b6581f4311fd4dceb376128a8fdf1cd4 P d1abec696a6646e69480089ecc381245 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: d1abec696a6646e69480089ecc381245, State: Initialized, Role: NON_PARTICIPANT
I20250626 13:58:08.062836 10226 consensus_queue.cc:260] T b6581f4311fd4dceb376128a8fdf1cd4 P d1abec696a6646e69480089ecc381245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 214, Last appended: 1.214, Last appended by leader: 214, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:58:08.065160 10226 ts_tablet_manager.cc:1428] T b6581f4311fd4dceb376128a8fdf1cd4 P d1abec696a6646e69480089ecc381245: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20250626 13:58:08.066583 9781 tablet_copy_service.cc:342] P 73a1bdf6d24d4917a92b5c94c1cf365b: Request end of tablet copy session d1abec696a6646e69480089ecc381245-b6581f4311fd4dceb376128a8fdf1cd4 received from {username='slave'} at 127.9.90.2:41085
I20250626 13:58:08.066980 9781 tablet_copy_service.cc:434] P 73a1bdf6d24d4917a92b5c94c1cf365b: ending tablet copy session d1abec696a6646e69480089ecc381245-b6581f4311fd4dceb376128a8fdf1cd4 on tablet b6581f4311fd4dceb376128a8fdf1cd4 with peer d1abec696a6646e69480089ecc381245
W20250626 13:58:08.069952 10226 ts_tablet_manager.cc:726] T 1aacb69070df4b04844add935d88a48c P d1abec696a6646e69480089ecc381245: Tablet Copy: Invalid argument: Leader has replica of tablet 1aacb69070df4b04844add935d88a48c with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
I20250626 13:58:08.074923 10226 ts_tablet_manager.cc:927] T 3b7987a9eca34f43a9befc607121a68a P d1abec696a6646e69480089ecc381245: Initiating tablet copy from peer 73a1bdf6d24d4917a92b5c94c1cf365b (127.9.90.1:44111)
I20250626 13:58:08.076102 10226 tablet_copy_client.cc:323] T 3b7987a9eca34f43a9befc607121a68a P d1abec696a6646e69480089ecc381245: tablet copy: Beginning tablet copy session from remote peer at address 127.9.90.1:44111
I20250626 13:58:08.077311 9781 tablet_copy_service.cc:140] P 73a1bdf6d24d4917a92b5c94c1cf365b: Received BeginTabletCopySession request for tablet 3b7987a9eca34f43a9befc607121a68a from peer d1abec696a6646e69480089ecc381245 ({username='slave'} at 127.9.90.2:41085)
I20250626 13:58:08.077726 9781 tablet_copy_service.cc:161] P 73a1bdf6d24d4917a92b5c94c1cf365b: Beginning new tablet copy session on tablet 3b7987a9eca34f43a9befc607121a68a from peer d1abec696a6646e69480089ecc381245 at {username='slave'} at 127.9.90.2:41085: session id = d1abec696a6646e69480089ecc381245-3b7987a9eca34f43a9befc607121a68a
I20250626 13:58:08.082129 9781 tablet_copy_source_session.cc:215] T 3b7987a9eca34f43a9befc607121a68a P 73a1bdf6d24d4917a92b5c94c1cf365b: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 13:58:08.084285 10226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7987a9eca34f43a9befc607121a68a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:08.092849 10226 tablet_copy_client.cc:806] T 3b7987a9eca34f43a9befc607121a68a P d1abec696a6646e69480089ecc381245: tablet copy: Starting download of 0 data blocks...
I20250626 13:58:08.093266 10226 tablet_copy_client.cc:670] T 3b7987a9eca34f43a9befc607121a68a P d1abec696a6646e69480089ecc381245: tablet copy: Starting download of 1 WAL segments...
I20250626 13:58:08.106326 10226 tablet_copy_client.cc:538] T 3b7987a9eca34f43a9befc607121a68a P d1abec696a6646e69480089ecc381245: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 13:58:08.113054 10226 tablet_bootstrap.cc:492] T 3b7987a9eca34f43a9befc607121a68a P d1abec696a6646e69480089ecc381245: Bootstrap starting.
I20250626 13:58:09.185247 10226 tablet_bootstrap.cc:492] T 3b7987a9eca34f43a9befc607121a68a P d1abec696a6646e69480089ecc381245: Bootstrap replayed 1/1 log segments. Stats: ops{read=214 overwritten=0 applied=214 ignored=0} inserts{seen=2710 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 13:58:09.185986 10226 tablet_bootstrap.cc:492] T 3b7987a9eca34f43a9befc607121a68a P d1abec696a6646e69480089ecc381245: Bootstrap complete.
I20250626 13:58:09.186517 10226 ts_tablet_manager.cc:1397] T 3b7987a9eca34f43a9befc607121a68a P d1abec696a6646e69480089ecc381245: Time spent bootstrapping tablet: real 1.074s user 1.051s sys 0.020s
I20250626 13:58:09.188180 10226 raft_consensus.cc:357] T 3b7987a9eca34f43a9befc607121a68a P d1abec696a6646e69480089ecc381245 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:58:09.188550 10226 raft_consensus.cc:738] T 3b7987a9eca34f43a9befc607121a68a P d1abec696a6646e69480089ecc381245 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: d1abec696a6646e69480089ecc381245, State: Initialized, Role: NON_PARTICIPANT
I20250626 13:58:09.188953 10226 consensus_queue.cc:260] T 3b7987a9eca34f43a9befc607121a68a P d1abec696a6646e69480089ecc381245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 214, Last appended: 1.214, Last appended by leader: 214, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a1bdf6d24d4917a92b5c94c1cf365b" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 44111 } }
I20250626 13:58:09.190765 10226 ts_tablet_manager.cc:1428] T 3b7987a9eca34f43a9befc607121a68a P d1abec696a6646e69480089ecc381245: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20250626 13:58:09.192240 9781 tablet_copy_service.cc:342] P 73a1bdf6d24d4917a92b5c94c1cf365b: Request end of tablet copy session d1abec696a6646e69480089ecc381245-3b7987a9eca34f43a9befc607121a68a received from {username='slave'} at 127.9.90.2:41085
I20250626 13:58:09.192600 9781 tablet_copy_service.cc:434] P 73a1bdf6d24d4917a92b5c94c1cf365b: ending tablet copy session d1abec696a6646e69480089ecc381245-3b7987a9eca34f43a9befc607121a68a on tablet 3b7987a9eca34f43a9befc607121a68a with peer d1abec696a6646e69480089ecc381245
W20250626 13:58:09.195225 10226 ts_tablet_manager.cc:726] T b6581f4311fd4dceb376128a8fdf1cd4 P d1abec696a6646e69480089ecc381245: Tablet Copy: Invalid argument: Leader has replica of tablet b6581f4311fd4dceb376128a8fdf1cd4 with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
W20250626 13:58:09.200076 10226 ts_tablet_manager.cc:726] T 3b7987a9eca34f43a9befc607121a68a P d1abec696a6646e69480089ecc381245: Tablet Copy: Invalid argument: Leader has replica of tablet 3b7987a9eca34f43a9befc607121a68a with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
I20250626 13:58:09.203549 9576 tablet_copy-itest.cc:1252] Number of Service unavailable responses: 1631
I20250626 13:58:09.203842 9576 tablet_copy-itest.cc:1253] Number of in progress responses: 790
I20250626 13:58:09.205963 9576 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu with pid 9677
I20250626 13:58:09.256433 9576 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu with pid 10091
I20250626 13:58:09.287559 9576 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu with pid 9951
2025-06-26T13:58:09Z chronyd exiting
[ OK ] TabletCopyITest.TestTabletCopyThrottling (21449 ms)
[ RUN ] TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate
2025-06-26T13:58:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-26T13:58:09Z Disabled control of system clock
I20250626 13:58:09.379837 9576 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
/tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.62:38455
--webserver_interface=127.9.90.62
--webserver_port=0
--builtin_ntp_servers=127.9.90.20:41399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.90.62:38455 with env {}
W20250626 13:58:09.677160 10245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:09.677728 10245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:09.678133 10245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:09.709187 10245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250626 13:58:09.709501 10245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:09.709709 10245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250626 13:58:09.709909 10245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250626 13:58:09.746069 10245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.20:41399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.90.62:38455
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.90.62:38455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.90.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:09.747388 10245 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:09.749053 10245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:09.765126 10251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:09.765380 10254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:09.765540 10252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:10.951009 10253 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250626 13:58:10.951143 10245 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:58:10.955749 10245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:10.958925 10245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:10.960366 10245 hybrid_clock.cc:648] HybridClock initialized: now 1750946290960323 us; error 62 us; skew 500 ppm
I20250626 13:58:10.961196 10245 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:10.973428 10245 webserver.cc:469] Webserver started at http://127.9.90.62:41831/ using document root <none> and password file <none>
I20250626 13:58:10.974391 10245 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:10.974591 10245 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:10.975044 10245 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:10.980010 10245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/data/instance:
uuid: "810014de92a447fcae20e13f275e8785"
format_stamp: "Formatted at 2025-06-26 13:58:10 on dist-test-slave-92f0"
I20250626 13:58:10.981092 10245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/wal/instance:
uuid: "810014de92a447fcae20e13f275e8785"
format_stamp: "Formatted at 2025-06-26 13:58:10 on dist-test-slave-92f0"
I20250626 13:58:10.988282 10245 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250626 13:58:10.993741 10261 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:10.994786 10245 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250626 13:58:10.995088 10245 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/data,/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/wal
uuid: "810014de92a447fcae20e13f275e8785"
format_stamp: "Formatted at 2025-06-26 13:58:10 on dist-test-slave-92f0"
I20250626 13:58:10.995390 10245 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:11.065084 10245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:11.066522 10245 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:11.066933 10245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:11.136087 10245 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.62:38455
I20250626 13:58:11.136173 10312 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.90.62:38455 every 8 connection(s)
I20250626 13:58:11.138824 10245 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/data/info.pb
I20250626 13:58:11.140996 9576 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu as pid 10245
I20250626 13:58:11.141571 9576 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/master-0/wal/instance
I20250626 13:58:11.145001 10313 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:11.170715 10313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785: Bootstrap starting.
I20250626 13:58:11.176147 10313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:11.177879 10313 log.cc:826] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:11.182157 10313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785: No bootstrap required, opened a new log
I20250626 13:58:11.199065 10313 raft_consensus.cc:357] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810014de92a447fcae20e13f275e8785" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 38455 } }
I20250626 13:58:11.199862 10313 raft_consensus.cc:383] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:11.200170 10313 raft_consensus.cc:738] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 810014de92a447fcae20e13f275e8785, State: Initialized, Role: FOLLOWER
I20250626 13:58:11.200987 10313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810014de92a447fcae20e13f275e8785" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 38455 } }
I20250626 13:58:11.201494 10313 raft_consensus.cc:397] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 13:58:11.201812 10313 raft_consensus.cc:491] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 13:58:11.202111 10313 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:11.206249 10313 raft_consensus.cc:513] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810014de92a447fcae20e13f275e8785" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 38455 } }
I20250626 13:58:11.206991 10313 leader_election.cc:304] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 810014de92a447fcae20e13f275e8785; no voters:
I20250626 13:58:11.208910 10313 leader_election.cc:290] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 13:58:11.209617 10318 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:58:11.212006 10318 raft_consensus.cc:695] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [term 1 LEADER]: Becoming Leader. State: Replica: 810014de92a447fcae20e13f275e8785, State: Running, Role: LEADER
I20250626 13:58:11.212836 10318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810014de92a447fcae20e13f275e8785" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 38455 } }
I20250626 13:58:11.213898 10313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 13:58:11.223048 10320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 810014de92a447fcae20e13f275e8785. Latest consensus state: current_term: 1 leader_uuid: "810014de92a447fcae20e13f275e8785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810014de92a447fcae20e13f275e8785" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 38455 } } }
I20250626 13:58:11.224104 10320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [sys.catalog]: This master's current role is: LEADER
I20250626 13:58:11.223977 10319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "810014de92a447fcae20e13f275e8785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "810014de92a447fcae20e13f275e8785" member_type: VOTER last_known_addr { host: "127.9.90.62" port: 38455 } } }
I20250626 13:58:11.224624 10319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785 [sys.catalog]: This master's current role is: LEADER
I20250626 13:58:11.227924 10326 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 13:58:11.239884 10326 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 13:58:11.257267 10326 catalog_manager.cc:1349] Generated new cluster ID: f0db69b7b2854d13924f950fa51011b2
I20250626 13:58:11.257524 10326 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 13:58:11.275624 10326 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 13:58:11.277365 10326 catalog_manager.cc:1506] Loading token signing keys...
I20250626 13:58:11.288892 10326 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 810014de92a447fcae20e13f275e8785: Generated new TSK 0
I20250626 13:58:11.289839 10326 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 13:58:11.317031 9576 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
/tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.1:0
--local_ip_for_outbound_sockets=127.9.90.1
--webserver_interface=127.9.90.1
--webserver_port=0
--tserver_master_addrs=127.9.90.62:38455
--builtin_ntp_servers=127.9.90.20:41399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
W20250626 13:58:11.622046 10337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:11.622573 10337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:11.622993 10337 flags.cc:425] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20250626 13:58:11.623203 10337 flags.cc:425] Enabled unsafe flag: --enable_flush_memrowset=false
W20250626 13:58:11.623570 10337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:11.655263 10337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:11.656260 10337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.1
I20250626 13:58:11.693481 10337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.20:41399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.90.1
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.9.90.62:38455
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.1
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:11.694831 10337 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:11.696537 10337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:11.713951 10344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:11.716539 10343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:11.718082 10337 server_base.cc:1048] running on GCE node
W20250626 13:58:11.717516 10346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:12.915737 10337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:12.918543 10337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:12.919972 10337 hybrid_clock.cc:648] HybridClock initialized: now 1750946292919918 us; error 74 us; skew 500 ppm
I20250626 13:58:12.920809 10337 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:12.928391 10337 webserver.cc:469] Webserver started at http://127.9.90.1:45295/ using document root <none> and password file <none>
I20250626 13:58:12.929339 10337 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:12.929667 10337 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:12.930171 10337 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:12.934851 10337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/data/instance:
uuid: "7c092ed73dd64feab004f5c1a90a7ec4"
format_stamp: "Formatted at 2025-06-26 13:58:12 on dist-test-slave-92f0"
I20250626 13:58:12.935997 10337 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/wal/instance:
uuid: "7c092ed73dd64feab004f5c1a90a7ec4"
format_stamp: "Formatted at 2025-06-26 13:58:12 on dist-test-slave-92f0"
I20250626 13:58:12.944212 10337 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20250626 13:58:12.950578 10353 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:12.951826 10337 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250626 13:58:12.952142 10337 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/data,/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/wal
uuid: "7c092ed73dd64feab004f5c1a90a7ec4"
format_stamp: "Formatted at 2025-06-26 13:58:12 on dist-test-slave-92f0"
I20250626 13:58:12.952478 10337 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:13.008625 10337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:13.010134 10337 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:13.010589 10337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:13.013373 10337 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:13.018337 10337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:13.018549 10337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:13.018826 10337 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:13.018983 10337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:13.162760 10337 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.1:37993
I20250626 13:58:13.162873 10465 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.90.1:37993 every 8 connection(s)
I20250626 13:58:13.165373 10337 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/data/info.pb
I20250626 13:58:13.171777 9576 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu as pid 10337
I20250626 13:58:13.172179 9576 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-0/wal/instance
I20250626 13:58:13.178398 9576 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
/tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.2:0
--local_ip_for_outbound_sockets=127.9.90.2
--webserver_interface=127.9.90.2
--webserver_port=0
--tserver_master_addrs=127.9.90.62:38455
--builtin_ntp_servers=127.9.90.20:41399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
I20250626 13:58:13.187268 10466 heartbeater.cc:344] Connected to a master server at 127.9.90.62:38455
I20250626 13:58:13.187844 10466 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:13.189213 10466 heartbeater.cc:507] Master 127.9.90.62:38455 requested a full tablet report, sending...
I20250626 13:58:13.192303 10278 ts_manager.cc:194] Registered new tserver with Master: 7c092ed73dd64feab004f5c1a90a7ec4 (127.9.90.1:37993)
I20250626 13:58:13.194399 10278 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.1:36165
W20250626 13:58:13.482793 10470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:13.483314 10470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:13.483754 10470 flags.cc:425] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20250626 13:58:13.483969 10470 flags.cc:425] Enabled unsafe flag: --enable_flush_memrowset=false
W20250626 13:58:13.484284 10470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:13.515165 10470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:13.516233 10470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.2
I20250626 13:58:13.551806 10470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.20:41399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.90.2
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.9.90.62:38455
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.2
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:13.553241 10470 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:13.554911 10470 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:13.569448 10476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:14.198836 10466 heartbeater.cc:499] Master 127.9.90.62:38455 was elected leader, sending a full tablet report...
W20250626 13:58:13.569454 10477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:15.106467 10470 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.539s user 0.456s sys 1.051s
W20250626 13:58:14.972038 10475 debug-util.cc:398] Leaking SignalData structure 0x7b0800012f40 after lost signal to thread 10470
W20250626 13:58:15.107913 10470 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.541s user 0.457s sys 1.051s
W20250626 13:58:15.109211 10479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:15.112213 10478 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1541 milliseconds
I20250626 13:58:15.112299 10470 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250626 13:58:15.113494 10470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:15.115741 10470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:15.117142 10470 hybrid_clock.cc:648] HybridClock initialized: now 1750946295117094 us; error 54 us; skew 500 ppm
I20250626 13:58:15.117992 10470 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:15.124169 10470 webserver.cc:469] Webserver started at http://127.9.90.2:46787/ using document root <none> and password file <none>
I20250626 13:58:15.125113 10470 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:15.125329 10470 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:15.125766 10470 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:15.130172 10470 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/data/instance:
uuid: "9822c3a0d69d4061af449a06cb2e7c50"
format_stamp: "Formatted at 2025-06-26 13:58:15 on dist-test-slave-92f0"
I20250626 13:58:15.131351 10470 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/wal/instance:
uuid: "9822c3a0d69d4061af449a06cb2e7c50"
format_stamp: "Formatted at 2025-06-26 13:58:15 on dist-test-slave-92f0"
I20250626 13:58:15.139917 10470 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20250626 13:58:15.147119 10486 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:15.148413 10470 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 13:58:15.148802 10470 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/data,/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/wal
uuid: "9822c3a0d69d4061af449a06cb2e7c50"
format_stamp: "Formatted at 2025-06-26 13:58:15 on dist-test-slave-92f0"
I20250626 13:58:15.149226 10470 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:15.196561 10470 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:15.198002 10470 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:15.198410 10470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:15.200858 10470 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:15.204950 10470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:15.205255 10470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:15.205556 10470 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:15.205716 10470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:15.338552 10470 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.2:42365
I20250626 13:58:15.338675 10598 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.90.2:42365 every 8 connection(s)
I20250626 13:58:15.341110 10470 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/data/info.pb
I20250626 13:58:15.342074 9576 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu as pid 10470
I20250626 13:58:15.342527 9576 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-1/wal/instance
I20250626 13:58:15.350570 9576 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
/tmp/dist-test-taskraC6OB/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.90.3:0
--local_ip_for_outbound_sockets=127.9.90.3
--webserver_interface=127.9.90.3
--webserver_port=0
--tserver_master_addrs=127.9.90.62:38455
--builtin_ntp_servers=127.9.90.20:41399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
I20250626 13:58:15.362572 10599 heartbeater.cc:344] Connected to a master server at 127.9.90.62:38455
I20250626 13:58:15.363130 10599 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:15.364578 10599 heartbeater.cc:507] Master 127.9.90.62:38455 requested a full tablet report, sending...
I20250626 13:58:15.367349 10278 ts_manager.cc:194] Registered new tserver with Master: 9822c3a0d69d4061af449a06cb2e7c50 (127.9.90.2:42365)
I20250626 13:58:15.368631 10278 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.2:53497
W20250626 13:58:15.650717 10603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250626 13:58:15.651250 10603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250626 13:58:15.651711 10603 flags.cc:425] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20250626 13:58:15.651930 10603 flags.cc:425] Enabled unsafe flag: --enable_flush_memrowset=false
W20250626 13:58:15.652256 10603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250626 13:58:15.683265 10603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250626 13:58:15.684309 10603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.90.3
I20250626 13:58:15.720474 10603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.90.20:41399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.90.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.90.3
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.9.90.62:38455
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.90.3
--log_dir=/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 26 Jun 2025 13:43:22 UTC on 5fd53c4cbb9d
build id 6781
TSAN enabled
I20250626 13:58:15.721840 10603 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 13:58:15.723556 10603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 13:58:15.741195 10609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:16.372691 10599 heartbeater.cc:499] Master 127.9.90.62:38455 was elected leader, sending a full tablet report...
I20250626 13:58:15.742853 10603 server_base.cc:1048] running on GCE node
W20250626 13:58:15.741822 10610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 13:58:15.741333 10612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 13:58:16.917217 10603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250626 13:58:16.919449 10603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250626 13:58:16.920851 10603 hybrid_clock.cc:648] HybridClock initialized: now 1750946296920812 us; error 57 us; skew 500 ppm
I20250626 13:58:16.921656 10603 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 13:58:16.930932 10603 webserver.cc:469] Webserver started at http://127.9.90.3:46489/ using document root <none> and password file <none>
I20250626 13:58:16.931926 10603 fs_manager.cc:362] Metadata directory not provided
I20250626 13:58:16.932143 10603 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 13:58:16.932615 10603 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 13:58:16.937012 10603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/data/instance:
uuid: "ebbfab9e778e469980d686587e3cdb81"
format_stamp: "Formatted at 2025-06-26 13:58:16 on dist-test-slave-92f0"
I20250626 13:58:16.938220 10603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/wal/instance:
uuid: "ebbfab9e778e469980d686587e3cdb81"
format_stamp: "Formatted at 2025-06-26 13:58:16 on dist-test-slave-92f0"
I20250626 13:58:16.945389 10603 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.002s
I20250626 13:58:16.950930 10619 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:16.952109 10603 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250626 13:58:16.952435 10603 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/data,/tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/wal
uuid: "ebbfab9e778e469980d686587e3cdb81"
format_stamp: "Formatted at 2025-06-26 13:58:16 on dist-test-slave-92f0"
I20250626 13:58:16.952780 10603 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 13:58:17.015923 10603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 13:58:17.017392 10603 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 13:58:17.017845 10603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 13:58:17.020399 10603 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 13:58:17.024430 10603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 13:58:17.024634 10603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:17.024907 10603 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 13:58:17.025063 10603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 13:58:17.165514 10603 rpc_server.cc:307] RPC server started. Bound to: 127.9.90.3:41769
I20250626 13:58:17.165717 10731 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.90.3:41769 every 8 connection(s)
I20250626 13:58:17.168154 10603 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/data/info.pb
I20250626 13:58:17.174044 9576 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu as pid 10603
I20250626 13:58:17.174538 9576 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0/minicluster-data/ts-2/wal/instance
I20250626 13:58:17.190891 10732 heartbeater.cc:344] Connected to a master server at 127.9.90.62:38455
I20250626 13:58:17.191347 10732 heartbeater.cc:461] Registering TS with master...
I20250626 13:58:17.192548 10732 heartbeater.cc:507] Master 127.9.90.62:38455 requested a full tablet report, sending...
I20250626 13:58:17.194870 10277 ts_manager.cc:194] Registered new tserver with Master: ebbfab9e778e469980d686587e3cdb81 (127.9.90.3:41769)
I20250626 13:58:17.196275 10277 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.90.3:44899
I20250626 13:58:17.209213 9576 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250626 13:58:17.237825 9576 test_util.cc:276] Using random seed: -200191708
I20250626 13:58:17.277467 10277 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53152:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 13:58:17.279974 10277 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 13:58:17.330157 10667 tablet_service.cc:1468] Processing CreateTablet for tablet 080deb44c32c465aa51ae828ee4eb021 (DEFAULT_TABLE table=test-workload [id=e91532e0bd5645e4b9afecfcfb806c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:17.332427 10667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 080deb44c32c465aa51ae828ee4eb021. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:17.334685 10534 tablet_service.cc:1468] Processing CreateTablet for tablet 080deb44c32c465aa51ae828ee4eb021 (DEFAULT_TABLE table=test-workload [id=e91532e0bd5645e4b9afecfcfb806c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:17.336671 10534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 080deb44c32c465aa51ae828ee4eb021. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:17.337344 10401 tablet_service.cc:1468] Processing CreateTablet for tablet 080deb44c32c465aa51ae828ee4eb021 (DEFAULT_TABLE table=test-workload [id=e91532e0bd5645e4b9afecfcfb806c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 13:58:17.342978 10401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 080deb44c32c465aa51ae828ee4eb021. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 13:58:17.371062 10757 tablet_bootstrap.cc:492] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50: Bootstrap starting.
I20250626 13:58:17.374672 10756 tablet_bootstrap.cc:492] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81: Bootstrap starting.
I20250626 13:58:17.376403 10758 tablet_bootstrap.cc:492] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4: Bootstrap starting.
I20250626 13:58:17.381116 10757 tablet_bootstrap.cc:654] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:17.383786 10757 log.cc:826] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:17.385195 10758 tablet_bootstrap.cc:654] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:17.387969 10758 log.cc:826] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:17.390417 10756 tablet_bootstrap.cc:654] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81: Neither blocks nor log segments found. Creating new log.
I20250626 13:58:17.393244 10756 log.cc:826] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81: Log is configured to *not* fsync() on all Append() calls
I20250626 13:58:17.394570 10757 tablet_bootstrap.cc:492] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50: No bootstrap required, opened a new log
I20250626 13:58:17.394985 10758 tablet_bootstrap.cc:492] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4: No bootstrap required, opened a new log
I20250626 13:58:17.395123 10757 ts_tablet_manager.cc:1397] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.004s
I20250626 13:58:17.395768 10758 ts_tablet_manager.cc:1397] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.012s
I20250626 13:58:17.398886 10756 tablet_bootstrap.cc:492] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81: No bootstrap required, opened a new log
I20250626 13:58:17.399389 10756 ts_tablet_manager.cc:1397] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.003s
I20250626 13:58:17.413964 10758 raft_consensus.cc:357] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbfab9e778e469980d686587e3cdb81" member_type: VOTER last_known_addr { host: "127.9.90.3" port: 41769 } } peers { permanent_uuid: "9822c3a0d69d4061af449a06cb2e7c50" member_type: VOTER last_known_addr { host: "127.9.90.2" port: 42365 } } peers { permanent_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 37993 } }
I20250626 13:58:17.414705 10758 raft_consensus.cc:383] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:17.414973 10758 raft_consensus.cc:738] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c092ed73dd64feab004f5c1a90a7ec4, State: Initialized, Role: FOLLOWER
I20250626 13:58:17.415836 10758 consensus_queue.cc:260] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbfab9e778e469980d686587e3cdb81" member_type: VOTER last_known_addr { host: "127.9.90.3" port: 41769 } } peers { permanent_uuid: "9822c3a0d69d4061af449a06cb2e7c50" member_type: VOTER last_known_addr { host: "127.9.90.2" port: 42365 } } peers { permanent_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 37993 } }
I20250626 13:58:17.422606 10758 ts_tablet_manager.cc:1428] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4: Time spent starting tablet: real 0.027s user 0.019s sys 0.006s
I20250626 13:58:17.423228 10757 raft_consensus.cc:357] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbfab9e778e469980d686587e3cdb81" member_type: VOTER last_known_addr { host: "127.9.90.3" port: 41769 } } peers { permanent_uuid: "9822c3a0d69d4061af449a06cb2e7c50" member_type: VOTER last_known_addr { host: "127.9.90.2" port: 42365 } } peers { permanent_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 37993 } }
I20250626 13:58:17.424141 10757 raft_consensus.cc:383] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:17.424450 10757 raft_consensus.cc:738] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9822c3a0d69d4061af449a06cb2e7c50, State: Initialized, Role: FOLLOWER
W20250626 13:58:17.425071 10467 tablet.cc:2378] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:58:17.425446 10757 consensus_queue.cc:260] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbfab9e778e469980d686587e3cdb81" member_type: VOTER last_known_addr { host: "127.9.90.3" port: 41769 } } peers { permanent_uuid: "9822c3a0d69d4061af449a06cb2e7c50" member_type: VOTER last_known_addr { host: "127.9.90.2" port: 42365 } } peers { permanent_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 37993 } }
I20250626 13:58:17.427863 10756 raft_consensus.cc:357] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbfab9e778e469980d686587e3cdb81" member_type: VOTER last_known_addr { host: "127.9.90.3" port: 41769 } } peers { permanent_uuid: "9822c3a0d69d4061af449a06cb2e7c50" member_type: VOTER last_known_addr { host: "127.9.90.2" port: 42365 } } peers { permanent_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 37993 } }
I20250626 13:58:17.428798 10756 raft_consensus.cc:383] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 13:58:17.429123 10756 raft_consensus.cc:738] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbfab9e778e469980d686587e3cdb81, State: Initialized, Role: FOLLOWER
I20250626 13:58:17.429734 10757 ts_tablet_manager.cc:1428] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50: Time spent starting tablet: real 0.034s user 0.025s sys 0.007s
I20250626 13:58:17.430188 10756 consensus_queue.cc:260] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbfab9e778e469980d686587e3cdb81" member_type: VOTER last_known_addr { host: "127.9.90.3" port: 41769 } } peers { permanent_uuid: "9822c3a0d69d4061af449a06cb2e7c50" member_type: VOTER last_known_addr { host: "127.9.90.2" port: 42365 } } peers { permanent_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 37993 } }
I20250626 13:58:17.434092 10732 heartbeater.cc:499] Master 127.9.90.62:38455 was elected leader, sending a full tablet report...
I20250626 13:58:17.434322 10756 ts_tablet_manager.cc:1428] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81: Time spent starting tablet: real 0.035s user 0.029s sys 0.002s
W20250626 13:58:17.598049 10600 tablet.cc:2378] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250626 13:58:17.673499 10733 tablet.cc:2378] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 13:58:17.789992 10762 raft_consensus.cc:491] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 13:58:17.790467 10762 raft_consensus.cc:513] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbfab9e778e469980d686587e3cdb81" member_type: VOTER last_known_addr { host: "127.9.90.3" port: 41769 } } peers { permanent_uuid: "9822c3a0d69d4061af449a06cb2e7c50" member_type: VOTER last_known_addr { host: "127.9.90.2" port: 42365 } } peers { permanent_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 37993 } }
I20250626 13:58:17.792892 10762 leader_election.cc:290] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebbfab9e778e469980d686587e3cdb81 (127.9.90.3:41769), 9822c3a0d69d4061af449a06cb2e7c50 (127.9.90.2:42365)
I20250626 13:58:17.805197 10687 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "080deb44c32c465aa51ae828ee4eb021" candidate_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbfab9e778e469980d686587e3cdb81" is_pre_election: true
I20250626 13:58:17.805696 10554 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "080deb44c32c465aa51ae828ee4eb021" candidate_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9822c3a0d69d4061af449a06cb2e7c50" is_pre_election: true
I20250626 13:58:17.806089 10687 raft_consensus.cc:2466] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c092ed73dd64feab004f5c1a90a7ec4 in term 0.
I20250626 13:58:17.806528 10554 raft_consensus.cc:2466] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c092ed73dd64feab004f5c1a90a7ec4 in term 0.
I20250626 13:58:17.807401 10355 leader_election.cc:304] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c092ed73dd64feab004f5c1a90a7ec4, ebbfab9e778e469980d686587e3cdb81; no voters:
I20250626 13:58:17.808280 10762 raft_consensus.cc:2802] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 13:58:17.808560 10762 raft_consensus.cc:491] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 13:58:17.808807 10762 raft_consensus.cc:3058] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:17.813172 10762 raft_consensus.cc:513] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbfab9e778e469980d686587e3cdb81" member_type: VOTER last_known_addr { host: "127.9.90.3" port: 41769 } } peers { permanent_uuid: "9822c3a0d69d4061af449a06cb2e7c50" member_type: VOTER last_known_addr { host: "127.9.90.2" port: 42365 } } peers { permanent_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 37993 } }
I20250626 13:58:17.814486 10762 leader_election.cc:290] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [CANDIDATE]: Term 1 election: Requested vote from peers ebbfab9e778e469980d686587e3cdb81 (127.9.90.3:41769), 9822c3a0d69d4061af449a06cb2e7c50 (127.9.90.2:42365)
I20250626 13:58:17.815237 10687 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "080deb44c32c465aa51ae828ee4eb021" candidate_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbfab9e778e469980d686587e3cdb81"
I20250626 13:58:17.815387 10554 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "080deb44c32c465aa51ae828ee4eb021" candidate_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9822c3a0d69d4061af449a06cb2e7c50"
I20250626 13:58:17.815737 10687 raft_consensus.cc:3058] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:17.815953 10554 raft_consensus.cc:3058] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50 [term 0 FOLLOWER]: Advancing to term 1
I20250626 13:58:17.820338 10554 raft_consensus.cc:2466] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c092ed73dd64feab004f5c1a90a7ec4 in term 1.
I20250626 13:58:17.820338 10687 raft_consensus.cc:2466] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c092ed73dd64feab004f5c1a90a7ec4 in term 1.
I20250626 13:58:17.821364 10357 leader_election.cc:304] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c092ed73dd64feab004f5c1a90a7ec4, 9822c3a0d69d4061af449a06cb2e7c50; no voters:
I20250626 13:58:17.822041 10762 raft_consensus.cc:2802] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 13:58:17.823534 10762 raft_consensus.cc:695] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [term 1 LEADER]: Becoming Leader. State: Replica: 7c092ed73dd64feab004f5c1a90a7ec4, State: Running, Role: LEADER
I20250626 13:58:17.824390 10762 consensus_queue.cc:237] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbfab9e778e469980d686587e3cdb81" member_type: VOTER last_known_addr { host: "127.9.90.3" port: 41769 } } peers { permanent_uuid: "9822c3a0d69d4061af449a06cb2e7c50" member_type: VOTER last_known_addr { host: "127.9.90.2" port: 42365 } } peers { permanent_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 37993 } }
I20250626 13:58:17.835281 10278 catalog_manager.cc:5582] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c092ed73dd64feab004f5c1a90a7ec4 (127.9.90.1). New cstate: current_term: 1 leader_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbfab9e778e469980d686587e3cdb81" member_type: VOTER last_known_addr { host: "127.9.90.3" port: 41769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9822c3a0d69d4061af449a06cb2e7c50" member_type: VOTER last_known_addr { host: "127.9.90.2" port: 42365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c092ed73dd64feab004f5c1a90a7ec4" member_type: VOTER last_known_addr { host: "127.9.90.1" port: 37993 } health_report { overall_health: HEALTHY } } }
I20250626 13:58:17.920979 10554 raft_consensus.cc:1273] T 080deb44c32c465aa51ae828ee4eb021 P 9822c3a0d69d4061af449a06cb2e7c50 [term 1 FOLLOWER]: Refusing update from remote peer 7c092ed73dd64feab004f5c1a90a7ec4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 13:58:17.921325 10687 raft_consensus.cc:1273] T 080deb44c32c465aa51ae828ee4eb021 P ebbfab9e778e469980d686587e3cdb81 [term 1 FOLLOWER]: Refusing update from remote peer 7c092ed73dd64feab004f5c1a90a7ec4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 13:58:17.922356 10762 consensus_queue.cc:1035] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9822c3a0d69d4061af449a06cb2e7c50" member_type: VOTER last_known_addr { host: "127.9.90.2" port: 42365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 13:58:17.923131 10767 consensus_queue.cc:1035] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbfab9e778e469980d686587e3cdb81" member_type: VOTER last_known_addr { host: "127.9.90.3" port: 41769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250626 13:58:17.950212 10775 mvcc.cc:204] Tried to move back new op lower bound from 7171876036269547520 to 7171876035895832576. Current Snapshot: MvccSnapshot[applied={T|T < 7171876036269547520}]
I20250626 13:58:17.953963 10776 mvcc.cc:204] Tried to move back new op lower bound from 7171876036269547520 to 7171876035895832576. Current Snapshot: MvccSnapshot[applied={T|T < 7171876036269547520}]
I20250626 13:58:17.959769 10778 mvcc.cc:204] Tried to move back new op lower bound from 7171876036269547520 to 7171876035895832576. Current Snapshot: MvccSnapshot[applied={T|T < 7171876036269547520}]
I20250626 13:58:18.007716 10421 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "080deb44c32c465aa51ae828ee4eb021"
dest_uuid: "7c092ed73dd64feab004f5c1a90a7ec4"
from {username='slave'} at 127.0.0.1:49776
I20250626 13:58:18.008250 10421 raft_consensus.cc:474] T 080deb44c32c465aa51ae828ee4eb021 P 7c092ed73dd64feab004f5c1a90a7ec4 [term 1 LEADER]: Not starting forced leader election -- already a leader
W20250626 13:58:43.278663 10595 debug-util.cc:398] Leaking SignalData structure 0x7b08000c7920 after lost signal to thread 10471
W20250626 13:58:43.284864 10595 debug-util.cc:398] Leaking SignalData structure 0x7b08000c6600 after lost signal to thread 10598
W20250626 13:59:10.894737 10728 debug-util.cc:398] Leaking SignalData structure 0x7b08000b2fe0 after lost signal to thread 10604
W20250626 13:59:10.895937 10728 debug-util.cc:398] Leaking SignalData structure 0x7b0800036f00 after lost signal to thread 10731
W20250626 13:59:12.874323 10309 debug-util.cc:398] Leaking SignalData structure 0x7b08000a8e80 after lost signal to thread 10246
W20250626 13:59:12.875051 10309 debug-util.cc:398] Leaking SignalData structure 0x7b080009f3a0 after lost signal to thread 10312
W20250626 13:59:14.012948 10792 meta_cache.cc:1261] Time spent looking up entry by key: real 0.053s user 0.000s sys 0.000s
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:2151: Failure
Failed
Bad status: Timed out: Timed out waiting for number of WAL segments on tablet 080deb44c32c465aa51ae828ee4eb021 on TS 0 to be 6. Found 5
I20250626 13:59:18.168229 9576 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20250626 13:59:18.168689 9576 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 7c092ed73dd64feab004f5c1a90a7ec4 and pid 10337
************************ BEGIN STACKS **************************
[New LWP 10338]
[New LWP 10339]
[New LWP 10340]
[New LWP 10341]
[New LWP 10342]
[New LWP 10349]
[New LWP 10350]
[New LWP 10351]
[New LWP 10354]
[New LWP 10355]
[New LWP 10356]
[New LWP 10357]
[New LWP 10358]
[New LWP 10359]
[New LWP 10360]
[New LWP 10361]
[New LWP 10362]
[New LWP 10363]
[New LWP 10364]
[New LWP 10365]
[New LWP 10366]
[New LWP 10367]
[New LWP 10368]
[New LWP 10369]
[New LWP 10370]
[New LWP 10371]
[New LWP 10372]
[New LWP 10373]
[New LWP 10374]
[New LWP 10375]
[New LWP 10376]
[New LWP 10377]
[New LWP 10378]
[New LWP 10379]
[New LWP 10380]
[New LWP 10381]
[New LWP 10382]
[New LWP 10383]
[New LWP 10384]
[New LWP 10385]
[New LWP 10386]
[New LWP 10387]
[New LWP 10388]
[New LWP 10389]
[New LWP 10390]
[New LWP 10391]
[New LWP 10392]
[New LWP 10393]
[New LWP 10394]
[New LWP 10395]
[New LWP 10396]
[New LWP 10397]
[New LWP 10398]
[New LWP 10399]
[New LWP 10400]
[New LWP 10401]
[New LWP 10402]
[New LWP 10403]
[New LWP 10404]
[New LWP 10405]
[New LWP 10406]
[New LWP 10407]
[New LWP 10408]
[New LWP 10409]
[New LWP 10410]
[New LWP 10411]
[New LWP 10412]
[New LWP 10413]
[New LWP 10414]
[New LWP 10415]
[New LWP 10416]
[New LWP 10417]
[New LWP 10418]
[New LWP 10419]
[New LWP 10420]
[New LWP 10421]
[New LWP 10422]
[New LWP 10423]
[New LWP 10424]
[New LWP 10425]
[New LWP 10426]
[New LWP 10427]
[New LWP 10428]
[New LWP 10429]
[New LWP 10430]
[New LWP 10431]
[New LWP 10432]
[New LWP 10433]
[New LWP 10434]
[New LWP 10435]
[New LWP 10436]
[New LWP 10437]
[New LWP 10438]
[New LWP 10439]
[New LWP 10440]
[New LWP 10441]
[New LWP 10442]
[New LWP 10443]
[New LWP 10444]
[New LWP 10445]
[New LWP 10446]
[New LWP 10447]
[New LWP 10448]
[New LWP 10449]
[New LWP 10450]
[New LWP 10451]
[New LWP 10452]
[New LWP 10453]
[New LWP 10454]
[New LWP 10455]
[New LWP 10456]
[New LWP 10457]
[New LWP 10458]
[New LWP 10459]
[New LWP 10460]
[New LWP 10461]
[New LWP 10462]
[New LWP 10463]
[New LWP 10464]
[New LWP 10465]
[New LWP 10466]
[New LWP 10467]
[New LWP 10768]
[New LWP 10957]
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
0x00007fd61f5b5d50 in ?? ()
Id Target Id Frame
* 1 LWP 10337 "kudu" 0x00007fd61f5b5d50 in ?? ()
2 LWP 10338 "kudu" 0x00007fd61a9797a0 in ?? ()
3 LWP 10339 "kudu" 0x00007fd61f5b1fb9 in ?? ()
4 LWP 10340 "kudu" 0x00007fd61f5b1fb9 in ?? ()
5 LWP 10341 "kudu" 0x00007fd61f5b1fb9 in ?? ()
6 LWP 10342 "kernel-watcher-" 0x00007fd61f5b1fb9 in ?? ()
7 LWP 10349 "ntp client-1034" 0x00007fd61f5b59e2 in ?? ()
8 LWP 10350 "file cache-evic" 0x00007fd61f5b1fb9 in ?? ()
9 LWP 10351 "sq_acceptor" 0x00007fd61a9a9cb9 in ?? ()
10 LWP 10354 "rpc reactor-103" 0x00007fd61a9b6a47 in ?? ()
11 LWP 10355 "rpc reactor-103" 0x00007fd61a9b6a47 in ?? ()
12 LWP 10356 "rpc reactor-103" 0x00007fd61a9b6a47 in ?? ()
13 LWP 10357 "rpc reactor-103" 0x00007fd61a9b6a47 in ?? ()
14 LWP 10358 "MaintenanceMgr " 0x00007fd61f5b1ad3 in ?? ()
15 LWP 10359 "txn-status-mana" 0x00007fd61f5b1fb9 in ?? ()
16 LWP 10360 "collect_and_rem" 0x00007fd61f5b1fb9 in ?? ()
17 LWP 10361 "tc-session-exp-" 0x00007fd61f5b1fb9 in ?? ()
18 LWP 10362 "rpc worker-1036" 0x00007fd61f5b1ad3 in ?? ()
19 LWP 10363 "rpc worker-1036" 0x00007fd61f5b1ad3 in ?? ()
20 LWP 10364 "rpc worker-1036" 0x00007fd61f5b1ad3 in ?? ()
21 LWP 10365 "rpc worker-1036" 0x00007fd61f5b1ad3 in ?? ()
22 LWP 10366 "rpc worker-1036" 0x00007fd61f5b1ad3 in ?? ()
23 LWP 10367 "rpc worker-1036" 0x00007fd61f5b1ad3 in ?? ()
24 LWP 10368 "rpc worker-1036" 0x00007fd61f5b1ad3 in ?? ()
25 LWP 10369 "rpc worker-1036" 0x00007fd61f5b1ad3 in ?? ()
26 LWP 10370 "rpc worker-1037" 0x00007fd61f5b1ad3 in ?? ()
27 LWP 10371 "rpc worker-1037" 0x00007fd61f5b1ad3 in ?? ()
28 LWP 10372 "rpc worker-1037" 0x00007fd61f5b1ad3 in ?? ()
29 LWP 10373 "rpc worker-1037" 0x00007fd61f5b1ad3 in ?? ()
30 LWP 10374 "rpc worker-1037" 0x00007fd61f5b1ad3 in ?? ()
31 LWP 10375 "rpc worker-1037" 0x00007fd61f5b1ad3 in ?? ()
32 LWP 10376 "rpc worker-1037" 0x00007fd61f5b1ad3 in ?? ()
33 LWP 10377 "rpc worker-1037" 0x00007fd61f5b1ad3 in ?? ()
34 LWP 10378 "rpc worker-1037" 0x00007fd61f5b1ad3 in ?? ()
35 LWP 10379 "rpc worker-1037" 0x00007fd61f5b1ad3 in ?? ()
36 LWP 10380 "rpc worker-1038" 0x00007fd61f5b1ad3 in ?? ()
37 LWP 10381 "rpc worker-1038" 0x00007fd61f5b1ad3 in ?? ()
38 LWP 10382 "rpc worker-1038" 0x00007fd61f5b1ad3 in ?? ()
39 LWP 10383 "rpc worker-1038" 0x00007fd61f5b1ad3 in ?? ()
40 LWP 10384 "rpc worker-1038" 0x00007fd61f5b1ad3 in ?? ()
41 LWP 10385 "rpc worker-1038" 0x00007fd61f5b1ad3 in ?? ()
42 LWP 10386 "rpc worker-1038" 0x00007fd61f5b1ad3 in ?? ()
43 LWP 10387 "rpc worker-1038" 0x00007fd61f5b1ad3 in ?? ()
44 LWP 10388 "rpc worker-1038" 0x00007fd61f5b1ad3 in ?? ()
45 LWP 10389 "rpc worker-1038" 0x00007fd61f5b1ad3 in ?? ()
46 LWP 10390 "rpc worker-1039" 0x00007fd61f5b1ad3 in ?? ()
47 LWP 10391 "rpc worker-1039" 0x00007fd61f5b1ad3 in ?? ()
48 LWP 10392 "rpc worker-1039" 0x00007fd61f5b1ad3 in ?? ()
49 LWP 10393 "rpc worker-1039" 0x00007fd61f5b1ad3 in ?? ()
50 LWP 10394 "rpc worker-1039" 0x00007fd61f5b1ad3 in ?? ()
51 LWP 10395 "rpc worker-1039" 0x00007fd61f5b1ad3 in ?? ()
52 LWP 10396 "rpc worker-1039" 0x00007fd61f5b1ad3 in ?? ()
53 LWP 10397 "rpc worker-1039" 0x00007fd61f5b1ad3 in ?? ()
54 LWP 10398 "rpc worker-1039" 0x00007fd61f5b1ad3 in ?? ()
55 LWP 10399 "rpc worker-1039" 0x00007fd61f5b1ad3 in ?? ()
56 LWP 10400 "rpc worker-1040" 0x00007fd61f5b1ad3 in ?? ()
57 LWP 10401 "rpc worker-1040" 0x00007fd61f5b1ad3 in ?? ()
58 LWP 10402 "rpc worker-1040" 0x00007fd61f5b1ad3 in ?? ()
59 LWP 10403 "rpc worker-1040" 0x00007fd61f5b1ad3 in ?? ()
60 LWP 10404 "rpc worker-1040" 0x00007fd61f5b1ad3 in ?? ()
61 LWP 10405 "rpc worker-1040" 0x00007fd61f5b1ad3 in ?? ()
62 LWP 10406 "rpc worker-1040" 0x00007fd61f5b1ad3 in ?? ()
63 LWP 10407 "rpc worker-1040" 0x00007fd61f5b1ad3 in ?? ()
64 LWP 10408 "rpc worker-1040" 0x00007fd61f5b1ad3 in ?? ()
65 LWP 10409 "rpc worker-1040" 0x00007fd61f5b1ad3 in ?? ()
66 LWP 10410 "rpc worker-1041" 0x00007fd61f5b1ad3 in ?? ()
67 LWP 10411 "rpc worker-1041" 0x00007fd61f5b1ad3 in ?? ()
68 LWP 10412 "rpc worker-1041" 0x00007fd61f5b1ad3 in ?? ()
69 LWP 10413 "rpc worker-1041" 0x00007fd61f5b1ad3 in ?? ()
70 LWP 10414 "rpc worker-1041" 0x00007fd61f5b1ad3 in ?? ()
71 LWP 10415 "rpc worker-1041" 0x00007fd61f5b1ad3 in ?? ()
72 LWP 10416 "rpc worker-1041" 0x00007fd61f5b1ad3 in ?? ()
73 LWP 10417 "rpc worker-1041" 0x00007fd61f5b1ad3 in ?? ()
74 LWP 10418 "rpc worker-1041" 0x00007fd61f5b1ad3 in ?? ()
75 LWP 10419 "rpc worker-1041" 0x00007fd61f5b1ad3 in ?? ()
76 LWP 10420 "rpc worker-1042" 0x00007fd61f5b1ad3 in ?? ()
77 LWP 10421 "rpc worker-1042" 0x00007fd61f5b1ad3 in ?? ()
78 LWP 10422 "rpc worker-1042" 0x00007fd61f5b1ad3 in ?? ()
79 LWP 10423 "rpc worker-1042" 0x00007fd61f5b1ad3 in ?? ()
80 LWP 10424 "rpc worker-1042" 0x00007fd61f5b1ad3 in ?? ()
81 LWP 10425 "rpc worker-1042" 0x00007fd61f5b1ad3 in ?? ()
82 LWP 10426 "rpc worker-1042" 0x00007fd61f5b1ad3 in ?? ()
83 LWP 10427 "rpc worker-1042" 0x00007fd61f5b1ad3 in ?? ()
84 LWP 10428 "rpc worker-1042" 0x00007fd61f5b1ad3 in ?? ()
85 LWP 10429 "rpc worker-1042" 0x00007fd61f5b1ad3 in ?? ()
86 LWP 10430 "rpc worker-1043" 0x00007fd61f5b1ad3 in ?? ()
87 LWP 10431 "rpc worker-1043" 0x00007fd61f5b1ad3 in ?? ()
88 LWP 10432 "rpc worker-1043" 0x00007fd61f5b1ad3 in ?? ()
89 LWP 10433 "rpc worker-1043" 0x00007fd61f5b1ad3 in ?? ()
90 LWP 10434 "rpc worker-1043" 0x00007fd61f5b1ad3 in ?? ()
91 LWP 10435 "rpc worker-1043" 0x00007fd61f5b1ad3 in ?? ()
92 LWP 10436 "rpc worker-1043" 0x00007fd61f5b1ad3 in ?? ()
93 LWP 10437 "rpc worker-1043" 0x00007fd61f5b1ad3 in ?? ()
94 LWP 10438 "rpc worker-1043" 0x00007fd61f5b1ad3 in ?? ()
95 LWP 10439 "rpc worker-1043" 0x00007fd61f5b1ad3 in ?? ()
96 LWP 10440 "rpc worker-1044" 0x00007fd61f5b1ad3 in ?? ()
97 LWP 10441 "rpc worker-1044" 0x00007fd61f5b1ad3 in ?? ()
98 LWP 10442 "rpc worker-1044" 0x00007fd61f5b1ad3 in ?? ()
99 LWP 10443 "rpc worker-1044" 0x00007fd61f5b1ad3 in ?? ()
100 LWP 10444 "rpc worker-1044" 0x00007fd61f5b1ad3 in ?? ()
101 LWP 10445 "rpc worker-1044" 0x00007fd61f5b1ad3 in ?? ()
102 LWP 10446 "rpc worker-1044" 0x00007fd61f5b1ad3 in ?? ()
103 LWP 10447 "rpc worker-1044" 0x00007fd61f5b1ad3 in ?? ()
104 LWP 10448 "rpc worker-1044" 0x00007fd61f5b1ad3 in ?? ()
105 LWP 10449 "rpc worker-1044" 0x00007fd61f5b1ad3 in ?? ()
106 LWP 10450 "rpc worker-1045" 0x00007fd61f5b1ad3 in ?? ()
107 LWP 10451 "rpc worker-1045" 0x00007fd61f5b1ad3 in ?? ()
108 LWP 10452 "rpc worker-1045" 0x00007fd61f5b1ad3 in ?? ()
109 LWP 10453 "rpc worker-1045" 0x00007fd61f5b1ad3 in ?? ()
110 LWP 10454 "rpc worker-1045" 0x00007fd61f5b1ad3 in ?? ()
111 LWP 10455 "rpc worker-1045" 0x00007fd61f5b1ad3 in ?? ()
112 LWP 10456 "rpc worker-1045" 0x00007fd61f5b1ad3 in ?? ()
113 LWP 10457 "rpc worker-1045" 0x00007fd61f5b1ad3 in ?? ()
114 LWP 10458 "rpc worker-1045" 0x00007fd61f5b1ad3 in ?? ()
115 LWP 10459 "rpc worker-1045" 0x00007fd61f5b1ad3 in ?? ()
116 LWP 10460 "rpc worker-1046" 0x00007fd61f5b1ad3 in ?? ()
117 LWP 10461 "rpc worker-1046" 0x00007fd61f5b1ad3 in ?? ()
118 LWP 10462 "diag-logger-104" 0x00007fd61f5b1fb9 in ?? ()
119 LWP 10463 "result-tracker-" 0x00007fd61f5b1fb9 in ?? ()
120 LWP 10464 "excess-log-dele" 0x00007fd61f5b1fb9 in ?? ()
121 LWP 10465 "acceptor-10465" 0x00007fd61a9b80c7 in ?? ()
122 LWP 10466 "heartbeat-10466" 0x00007fd61f5b1fb9 in ?? ()
123 LWP 10467 "maintenance_sch" 0x00007fd61f5b1fb9 in ?? ()
124 LWP 10768 "wal-append [wor" 0x00007fd61f5b1fb9 in ?? ()
125 LWP 10957 "raft [worker]-1" 0x00007fd61f5b1fb9 in ?? ()
Thread 125 (LWP 10957):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 124 (LWP 10768):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x00007b10000583f0 in ?? ()
#2 0x00000000000012d8 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b6400060018 in ?? ()
#5 0x00007fd5cfdbb440 in ?? ()
#6 0x00000000000025b0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 123 (LWP 10467):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x00007b0100000000 in ?? ()
#2 0x0000000000000106 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007fd5d37b96c0 in ?? ()
#6 0x000000000000020c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 10466):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 10465):
#0 0x00007fd61a9b80c7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 10464):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x00007fd5d4fbc940 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffc9e4946c0 in ?? ()
#5 0x00007fd5d4fbc7b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 10463):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x0000000085352f80 in ?? ()
#2 0x0000000000000041 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001008 in ?? ()
#5 0x00007fd5d57bd800 in ?? ()
#6 0x0000000000000082 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 10462):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x00007fd618a36008 in ?? ()
#2 0x0000000000000041 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000c90 in ?? ()
#5 0x00007fd5d5fbe750 in ?? ()
#6 0x0000000000000082 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 10461):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 10460):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 10459):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 10458):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 10457):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 10456):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 10455):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 10454):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 10453):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 10452):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 10451):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 10450):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 10449):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 10448):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 10447):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 10446):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 10445):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 10444):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 10443):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 10442):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 10441):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 10440):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 10439):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 10438):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 10437):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 10436):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 10435):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 10434):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 10433):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 10432):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 10431):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 10430):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 10429):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 10428):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 10427):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 10426):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 10425):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 10424):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 10423):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 10422):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 10421):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24001147cc in ?? ()
#4 0x00007fd5eb1ba710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd5eb1ba730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x0000000000000000 in ?? ()
Thread 76 (LWP 10420):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 75 (LWP 10419):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 10418):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 10417):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 10416):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 10415):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 10414):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 10413):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 10412):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 10411):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 10410):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 10409):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 10408):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 10407):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 10406):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 10405):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 10404):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 10403):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 10402):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 10401):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000b902c in ?? ()
#4 0x00007fd5f55bc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd5f55bc730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x007f0400000026c1 in ?? ()
#9 0x00007fd61f5b1770 in ?? ()
#10 0x00007fd5f55bc730 in ?? ()
#11 0x0002008300000dfe in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 10400):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 10399):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 10398):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 10397):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 10396):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 10395):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 10394):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 10393):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 10392):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 10391):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 10390):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 10389):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 10388):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 10387):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 10386):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 10385):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 10384):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 10383):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 10382):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 10381):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x00000000000004c3 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240005ffec in ?? ()
#4 0x00007fd5ff9be710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd5ff9be730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007fd61f5b1770 in ?? ()
#10 0x00007fd5ff9be730 in ?? ()
#11 0x00007fd5d2704ca8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 36 (LWP 10380):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000376 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240005d7f8 in ?? ()
#4 0x00007fd6003b6710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd6003b6730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 10379):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x000000000000015c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b2400058ff8 in ?? ()
#4 0x00007fd600bb7710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd600bb7730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 34 (LWP 10378):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 10377):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 10376):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 10375):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 10374):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 10373):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 10372):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 10371):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 10370):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 10369):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 10368):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 10367):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 10366):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 10365):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 10364):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 10363):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 10362):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 10361):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x0000000017a335b8 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800003a00 in ?? ()
#5 0x00007fd609f92700 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 10360):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x00007fd60a7939a8 in ?? ()
#2 0x000000000000000d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400037198 in ?? ()
#5 0x00007fd60a793840 in ?? ()
#6 0x000000000000001a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 10359):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007fd60af94410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 10358):
#0 0x00007fd61f5b1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 10357):
#0 0x00007fd61a9b6a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 10356):
#0 0x00007fd61a9b6a47 in ?? ()
#1 0x00007b280002a088 in ?? ()
#2 0x0044e00000094d23 in ?? ()
#3 0x00007fd60c797500 in ?? ()
#4 0x00007fd60c798b80 in ?? ()
#5 0x00007fd60c797500 in ?? ()
#6 0x0000000000000011 in ?? ()
#7 0x00007b5800001800 in ?? ()
#8 0x0000000000488695 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007fd618232000 in ?? ()
#10 0x0000000000488599 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007fd60c798b80 in ?? ()
#12 0x00007fd61d40f069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007fd622b901a0 in ?? ()
#15 0x00007b4c00002f90 in ?? ()
#16 0x00007b4c00002f98 in ?? ()
#17 0x00007fd60c7977a0 in ?? ()
#18 0x00007b4400033d00 in ?? ()
#19 0x00007fd60c797cd0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 11 (LWP 10355):
#0 0x00007fd61a9b6a47 in ?? ()
#1 0x00007b28000050a8 in ?? ()
#2 0x0040e00002201cc0 in ?? ()
#3 0x00007fd60cf98500 in ?? ()
#4 0x00007fd60cf99b80 in ?? ()
#5 0x00007fd60cf98500 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x00007b5800001200 in ?? ()
#8 0x0000000000488695 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007fd618244000 in ?? ()
#10 0x0000000000488599 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007fd60cf99b80 in ?? ()
#12 0x00007fd61d40f069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007fd622b901a0 in ?? ()
#15 0x00007b4c000026d0 in ?? ()
#16 0x00007b4c000026d8 in ?? ()
#17 0x00007fd60cf987a0 in ?? ()
#18 0x00007b44000363c0 in ?? ()
#19 0x00007fd60cf98cd0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 10 (LWP 10354):
#0 0x00007fd61a9b6a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 9 (LWP 10351):
#0 0x00007fd61a9a9cb9 in ?? ()
#1 0x00007fd6137bcc10 in ?? ()
#2 0x00007b0400009010 in ?? ()
#3 0x00007fd6137bdb80 in ?? ()
#4 0x00007fd6137bcc10 in ?? ()
#5 0x00007b0400009010 in ?? ()
#6 0x00000000004888a3 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007fd618410000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007fd6137bdb80 in ?? ()
#10 0x00007fd62438eb28 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 8 (LWP 10350):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 7 (LWP 10349):
#0 0x00007fd61f5b59e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 10342):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x00007fd6147bea40 in ?? ()
#2 0x000000000000014e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000361d8 in ?? ()
#5 0x00007fd6147be5d0 in ?? ()
#6 0x000000000000029c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 10341):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 10340):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 10339):
#0 0x00007fd61f5b1fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 10338):
#0 0x00007fd61a9797a0 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 10337):
#0 0x00007fd61f5b5d50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x0000000000467b2b in __sanitizer::internal_alloc_placeholder ()
#3 0x00007fd619bd7cc0 in ?? ()
#4 0x00007fd619bd7cc0 in ?? ()
#5 0x00007ffc9e4944d0 in ?? ()
#6 0x000000000048aef4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000a94680 in ?? ()
#9 0x00007fd619bd7cc0 in ?? ()
#10 0x00007fd61daddf0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250626 13:59:19.141009 9576 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID 9822c3a0d69d4061af449a06cb2e7c50 and pid 10470
************************ BEGIN STACKS **************************
[New LWP 10471]
[New LWP 10472]
[New LWP 10473]
[New LWP 10474]
[New LWP 10475]
[New LWP 10482]
[New LWP 10483]
[New LWP 10484]
[New LWP 10487]
[New LWP 10488]
[New LWP 10489]
[New LWP 10490]
[New LWP 10491]
[New LWP 10492]
[New LWP 10493]
[New LWP 10494]
[New LWP 10495]
[New LWP 10496]
[New LWP 10497]
[New LWP 10498]
[New LWP 10499]
[New LWP 10500]
[New LWP 10501]
[New LWP 10502]
[New LWP 10503]
[New LWP 10504]
[New LWP 10505]
[New LWP 10506]
[New LWP 10507]
[New LWP 10508]
[New LWP 10509]
[New LWP 10510]
[New LWP 10511]
[New LWP 10512]
[New LWP 10513]
[New LWP 10514]
[New LWP 10515]
[New LWP 10516]
[New LWP 10517]
[New LWP 10518]
[New LWP 10519]
[New LWP 10520]
[New LWP 10521]
[New LWP 10522]
[New LWP 10523]
[New LWP 10524]
[New LWP 10525]
[New LWP 10526]
[New LWP 10527]
[New LWP 10528]
[New LWP 10529]
[New LWP 10530]
[New LWP 10531]
[New LWP 10532]
[New LWP 10533]
[New LWP 10534]
[New LWP 10535]
[New LWP 10536]
[New LWP 10537]
[New LWP 10538]
[New LWP 10539]
[New LWP 10540]
[New LWP 10541]
[New LWP 10542]
[New LWP 10543]
[New LWP 10544]
[New LWP 10545]
[New LWP 10546]
[New LWP 10547]
[New LWP 10548]
[New LWP 10549]
[New LWP 10550]
[New LWP 10551]
[New LWP 10552]
[New LWP 10553]
[New LWP 10554]
[New LWP 10555]
[New LWP 10556]
[New LWP 10557]
[New LWP 10558]
[New LWP 10559]
[New LWP 10560]
[New LWP 10561]
[New LWP 10562]
[New LWP 10563]
[New LWP 10564]
[New LWP 10565]
[New LWP 10566]
[New LWP 10567]
[New LWP 10568]
[New LWP 10569]
[New LWP 10570]
[New LWP 10571]
[New LWP 10572]
[New LWP 10573]
[New LWP 10574]
[New LWP 10575]
[New LWP 10576]
[New LWP 10577]
[New LWP 10578]
[New LWP 10579]
[New LWP 10580]
[New LWP 10581]
[New LWP 10582]
[New LWP 10583]
[New LWP 10584]
[New LWP 10585]
[New LWP 10586]
[New LWP 10587]
[New LWP 10588]
[New LWP 10589]
[New LWP 10590]
[New LWP 10591]
[New LWP 10592]
[New LWP 10593]
[New LWP 10594]
[New LWP 10595]
[New LWP 10596]
[New LWP 10597]
[New LWP 10598]
[New LWP 10599]
[New LWP 10600]
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
0x00007f821ae15d50 in ?? ()
Id Target Id Frame
* 1 LWP 10470 "kudu" 0x00007f821ae15d50 in ?? ()
2 LWP 10471 "kudu" 0x00007f82161d97a0 in ?? ()
3 LWP 10472 "kudu" 0x00007f821ae11fb9 in ?? ()
4 LWP 10473 "kudu" 0x00007f821ae11fb9 in ?? ()
5 LWP 10474 "kudu" 0x00007f821ae11fb9 in ?? ()
6 LWP 10475 "kernel-watcher-" 0x00007f821ae11fb9 in ?? ()
7 LWP 10482 "ntp client-1048" 0x00007f821ae159e2 in ?? ()
8 LWP 10483 "file cache-evic" 0x00007f821ae11fb9 in ?? ()
9 LWP 10484 "sq_acceptor" 0x00007f8216209cb9 in ?? ()
10 LWP 10487 "rpc reactor-104" 0x00007f8216216a47 in ?? ()
11 LWP 10488 "rpc reactor-104" 0x00007f8216216a47 in ?? ()
12 LWP 10489 "rpc reactor-104" 0x00007f8216216a47 in ?? ()
13 LWP 10490 "rpc reactor-104" 0x00007f8216216a47 in ?? ()
14 LWP 10491 "MaintenanceMgr " 0x00007f821ae11ad3 in ?? ()
15 LWP 10492 "txn-status-mana" 0x00007f821ae11fb9 in ?? ()
16 LWP 10493 "collect_and_rem" 0x00007f821ae11fb9 in ?? ()
17 LWP 10494 "tc-session-exp-" 0x00007f821ae11fb9 in ?? ()
18 LWP 10495 "rpc worker-1049" 0x00007f821ae11ad3 in ?? ()
19 LWP 10496 "rpc worker-1049" 0x00007f821ae11ad3 in ?? ()
20 LWP 10497 "rpc worker-1049" 0x00007f821ae11ad3 in ?? ()
21 LWP 10498 "rpc worker-1049" 0x00007f821ae11ad3 in ?? ()
22 LWP 10499 "rpc worker-1049" 0x00007f821ae11ad3 in ?? ()
23 LWP 10500 "rpc worker-1050" 0x00007f821ae11ad3 in ?? ()
24 LWP 10501 "rpc worker-1050" 0x00007f821ae11ad3 in ?? ()
25 LWP 10502 "rpc worker-1050" 0x00007f821ae11ad3 in ?? ()
26 LWP 10503 "rpc worker-1050" 0x00007f821ae11ad3 in ?? ()
27 LWP 10504 "rpc worker-1050" 0x00007f821ae11ad3 in ?? ()
28 LWP 10505 "rpc worker-1050" 0x00007f821ae11ad3 in ?? ()
29 LWP 10506 "rpc worker-1050" 0x00007f821ae11ad3 in ?? ()
30 LWP 10507 "rpc worker-1050" 0x00007f821ae11ad3 in ?? ()
31 LWP 10508 "rpc worker-1050" 0x00007f821ae11ad3 in ?? ()
32 LWP 10509 "rpc worker-1050" 0x00007f821ae11ad3 in ?? ()
33 LWP 10510 "rpc worker-1051" 0x00007f821ae11ad3 in ?? ()
34 LWP 10511 "rpc worker-1051" 0x00007f821ae11ad3 in ?? ()
35 LWP 10512 "rpc worker-1051" 0x00007f821ae11ad3 in ?? ()
36 LWP 10513 "rpc worker-1051" 0x00007f821ae11ad3 in ?? ()
37 LWP 10514 "rpc worker-1051" 0x00007f821ae11ad3 in ?? ()
38 LWP 10515 "rpc worker-1051" 0x00007f821ae11ad3 in ?? ()
39 LWP 10516 "rpc worker-1051" 0x00007f821ae11ad3 in ?? ()
40 LWP 10517 "rpc worker-1051" 0x00007f821ae11ad3 in ?? ()
41 LWP 10518 "rpc worker-1051" 0x00007f821ae11ad3 in ?? ()
42 LWP 10519 "rpc worker-1051" 0x00007f821ae11ad3 in ?? ()
43 LWP 10520 "rpc worker-1052" 0x00007f821ae11ad3 in ?? ()
44 LWP 10521 "rpc worker-1052" 0x00007f821ae11ad3 in ?? ()
45 LWP 10522 "rpc worker-1052" 0x00007f821ae11ad3 in ?? ()
46 LWP 10523 "rpc worker-1052" 0x00007f821ae11ad3 in ?? ()
47 LWP 10524 "rpc worker-1052" 0x00007f821ae11ad3 in ?? ()
48 LWP 10525 "rpc worker-1052" 0x00007f821ae11ad3 in ?? ()
49 LWP 10526 "rpc worker-1052" 0x00007f821ae11ad3 in ?? ()
50 LWP 10527 "rpc worker-1052" 0x00007f821ae11ad3 in ?? ()
51 LWP 10528 "rpc worker-1052" 0x00007f821ae11ad3 in ?? ()
52 LWP 10529 "rpc worker-1052" 0x00007f821ae11ad3 in ?? ()
53 LWP 10530 "rpc worker-1053" 0x00007f821ae11ad3 in ?? ()
54 LWP 10531 "rpc worker-1053" 0x00007f821ae11ad3 in ?? ()
55 LWP 10532 "rpc worker-1053" 0x00007f821ae11ad3 in ?? ()
56 LWP 10533 "rpc worker-1053" 0x00007f821ae11ad3 in ?? ()
57 LWP 10534 "rpc worker-1053" 0x00007f821ae11ad3 in ?? ()
58 LWP 10535 "rpc worker-1053" 0x00007f821ae11ad3 in ?? ()
59 LWP 10536 "rpc worker-1053" 0x00007f821ae11ad3 in ?? ()
60 LWP 10537 "rpc worker-1053" 0x00007f821ae11ad3 in ?? ()
61 LWP 10538 "rpc worker-1053" 0x00007f821ae11ad3 in ?? ()
62 LWP 10539 "rpc worker-1053" 0x00007f821ae11ad3 in ?? ()
63 LWP 10540 "rpc worker-1054" 0x00007f821ae11ad3 in ?? ()
64 LWP 10541 "rpc worker-1054" 0x00007f821ae11ad3 in ?? ()
65 LWP 10542 "rpc worker-1054" 0x00007f821ae11ad3 in ?? ()
66 LWP 10543 "rpc worker-1054" 0x00007f821ae11ad3 in ?? ()
67 LWP 10544 "rpc worker-1054" 0x00007f821ae11ad3 in ?? ()
68 LWP 10545 "rpc worker-1054" 0x00007f821ae11ad3 in ?? ()
69 LWP 10546 "rpc worker-1054" 0x00007f821ae11ad3 in ?? ()
70 LWP 10547 "rpc worker-1054" 0x00007f821ae11ad3 in ?? ()
71 LWP 10548 "rpc worker-1054" 0x00007f821ae11ad3 in ?? ()
72 LWP 10549 "rpc worker-1054" 0x00007f821ae11ad3 in ?? ()
73 LWP 10550 "rpc worker-1055" 0x00007f821ae11ad3 in ?? ()
74 LWP 10551 "rpc worker-1055" 0x00007f821ae11ad3 in ?? ()
75 LWP 10552 "rpc worker-1055" 0x00007f821ae11ad3 in ?? ()
76 LWP 10553 "rpc worker-1055" 0x00007f821ae11ad3 in ?? ()
77 LWP 10554 "rpc worker-1055" 0x00007f821ae11ad3 in ?? ()
78 LWP 10555 "rpc worker-1055" 0x00007f821ae11ad3 in ?? ()
79 LWP 10556 "rpc worker-1055" 0x00007f821ae11ad3 in ?? ()
80 LWP 10557 "rpc worker-1055" 0x00007f821ae11ad3 in ?? ()
81 LWP 10558 "rpc worker-1055" 0x00007f821ae11ad3 in ?? ()
82 LWP 10559 "rpc worker-1055" 0x00007f821ae11ad3 in ?? ()
83 LWP 10560 "rpc worker-1056" 0x00007f821ae11ad3 in ?? ()
84 LWP 10561 "rpc worker-1056" 0x00007f821ae11ad3 in ?? ()
85 LWP 10562 "rpc worker-1056" 0x00007f821ae11ad3 in ?? ()
86 LWP 10563 "rpc worker-1056" 0x00007f821ae11ad3 in ?? ()
87 LWP 10564 "rpc worker-1056" 0x00007f821ae11ad3 in ?? ()
88 LWP 10565 "rpc worker-1056" 0x00007f821ae11ad3 in ?? ()
89 LWP 10566 "rpc worker-1056" 0x00007f821ae11ad3 in ?? ()
90 LWP 10567 "rpc worker-1056" 0x00007f821ae11ad3 in ?? ()
91 LWP 10568 "rpc worker-1056" 0x00007f821ae11ad3 in ?? ()
92 LWP 10569 "rpc worker-1056" 0x00007f821ae11ad3 in ?? ()
93 LWP 10570 "rpc worker-1057" 0x00007f821ae11ad3 in ?? ()
94 LWP 10571 "rpc worker-1057" 0x00007f821ae11ad3 in ?? ()
95 LWP 10572 "rpc worker-1057" 0x00007f821ae11ad3 in ?? ()
96 LWP 10573 "rpc worker-1057" 0x00007f821ae11ad3 in ?? ()
97 LWP 10574 "rpc worker-1057" 0x00007f821ae11ad3 in ?? ()
98 LWP 10575 "rpc worker-1057" 0x00007f821ae11ad3 in ?? ()
99 LWP 10576 "rpc worker-1057" 0x00007f821ae11ad3 in ?? ()
100 LWP 10577 "rpc worker-1057" 0x00007f821ae11ad3 in ?? ()
101 LWP 10578 "rpc worker-1057" 0x00007f821ae11ad3 in ?? ()
102 LWP 10579 "rpc worker-1057" 0x00007f821ae11ad3 in ?? ()
103 LWP 10580 "rpc worker-1058" 0x00007f821ae11ad3 in ?? ()
104 LWP 10581 "rpc worker-1058" 0x00007f821ae11ad3 in ?? ()
105 LWP 10582 "rpc worker-1058" 0x00007f821ae11ad3 in ?? ()
106 LWP 10583 "rpc worker-1058" 0x00007f821ae11ad3 in ?? ()
107 LWP 10584 "rpc worker-1058" 0x00007f821ae11ad3 in ?? ()
108 LWP 10585 "rpc worker-1058" 0x00007f821ae11ad3 in ?? ()
109 LWP 10586 "rpc worker-1058" 0x00007f821ae11ad3 in ?? ()
110 LWP 10587 "rpc worker-1058" 0x00007f821ae11ad3 in ?? ()
111 LWP 10588 "rpc worker-1058" 0x00007f821ae11ad3 in ?? ()
112 LWP 10589 "rpc worker-1058" 0x00007f821ae11ad3 in ?? ()
113 LWP 10590 "rpc worker-1059" 0x00007f821ae11ad3 in ?? ()
114 LWP 10591 "rpc worker-1059" 0x00007f821ae11ad3 in ?? ()
115 LWP 10592 "rpc worker-1059" 0x00007f821ae11ad3 in ?? ()
116 LWP 10593 "rpc worker-1059" 0x00007f821ae11ad3 in ?? ()
117 LWP 10594 "rpc worker-1059" 0x00007f821ae11ad3 in ?? ()
118 LWP 10595 "diag-logger-105" 0x00007f821ae11fb9 in ?? ()
119 LWP 10596 "result-tracker-" 0x00007f821ae11fb9 in ?? ()
120 LWP 10597 "excess-log-dele" 0x00007f821ae11fb9 in ?? ()
121 LWP 10598 "acceptor-10598" 0x00007f82162180c7 in ?? ()
122 LWP 10599 "heartbeat-10599" 0x00007f821ae11fb9 in ?? ()
123 LWP 10600 "maintenance_sch" 0x00007f821ae11fb9 in ?? ()
Thread 123 (LWP 10600):
#0 0x00007f821ae11fb9 in ?? ()
#1 0x00007b0100000000 in ?? ()
#2 0x0000000000000101 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007f81cf0b96c0 in ?? ()
#6 0x0000000000000202 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 10599):
#0 0x00007f821ae11fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 10598):
#0 0x00007f82162180c7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 10597):
#0 0x00007f821ae11fb9 in ?? ()
#1 0x00007f81d08bc940 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffeff6e1040 in ?? ()
#5 0x00007f81d08bc7b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 10596):
#0 0x00007f821ae11fb9 in ?? ()
#1 0x0000000085352f80 in ?? ()
#2 0x0000000000000040 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001008 in ?? ()
#5 0x00007f81d10bd800 in ?? ()
#6 0x0000000000000080 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 10595):
#0 0x00007f821ae11fb9 in ?? ()
#1 0x00007f8214188008 in ?? ()
#2 0x000000000000003b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000590 in ?? ()
#5 0x00007f81d18be750 in ?? ()
#6 0x0000000000000076 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 10594):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 10593):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 10592):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 10591):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 10590):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 10589):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 10588):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 10587):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 10586):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 10585):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 10584):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 10583):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 10582):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 10581):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 10580):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 10579):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 10578):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 10577):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 10576):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 10575):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 10574):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 10573):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 10572):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 10571):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 10570):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 10569):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 10568):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 10567):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 10566):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 10565):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 10564):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 10563):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 10562):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 10561):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 10560):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 10559):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 10558):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 10557):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 10556):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 10555):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 10554):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x000000000000091c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b24001147c8 in ?? ()
#4 0x00007f81e6aba710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f81e6aba730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 76 (LWP 10553):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x00000000000009bf in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240010ffcc in ?? ()
#4 0x00007f81e72bb710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f81e72bb730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f821ae11770 in ?? ()
#10 0x00007f81e72bb730 in ?? ()
#11 0x00007f81cb2e5678 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 10552):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 10551):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 10550):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 10549):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 10548):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 10547):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 10546):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 10545):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 10544):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 10543):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 10542):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 10541):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 10540):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 10539):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 10538):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 10537):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 10536):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 10535):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 10534):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000b902c in ?? ()
#4 0x00007f81f0ebc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f81f0ebc730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x007f0400000026c1 in ?? ()
#9 0x00007f821ae11770 in ?? ()
#10 0x00007f81f0ebc730 in ?? ()
#11 0x0002008300000dfe in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 10533):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 10532):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 10531):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 10530):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 10529):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 10528):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 10527):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 10526):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 10525):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 10524):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 10523):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 10522):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 10521):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 10520):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 10519):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 10518):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 10517):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 10516):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 10515):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 10514):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240005ffec in ?? ()
#4 0x00007f81fb2be710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f81fb2be730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f821ae11770 in ?? ()
#10 0x00007f81fb2be730 in ?? ()
#11 0x00007f8213269c48 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 36 (LWP 10513):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 10512):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 10511):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 10510):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 10509):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 10508):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 10507):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 10506):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 10505):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 10504):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 10503):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 10502):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 10501):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 10500):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 10499):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 10498):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 10497):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 10496):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 10495):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 10494):
#0 0x00007f821ae11fb9 in ?? ()
#1 0x0000000017a335b8 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800003a00 in ?? ()
#5 0x00007f8205892700 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 10493):
#0 0x00007f821ae11fb9 in ?? ()
#1 0x00007f82060939a8 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400037198 in ?? ()
#5 0x00007f8206093840 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 10492):
#0 0x00007f821ae11fb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007f8206894410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 10491):
#0 0x00007f821ae11ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 10490):
#0 0x00007f8216216a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 10489):
#0 0x00007f8216216a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 11 (LWP 10488):
#0 0x00007f8216216a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 10 (LWP 10487):
#0 0x00007f8216216a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 9 (LWP 10484):
#0 0x00007f8216209cb9 in ?? ()
#1 0x00007f820f0bcc10 in ?? ()
#2 0x00007b040000a050 in ?? ()
#3 0x00007f820f0bdb80 in ?? ()
#4 0x00007f820f0bcc10 in ?? ()
#5 0x00007b040000a050 in ?? ()
#6 0x00000000004888a3 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007f8213bbc000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007f820f0bdb80 in ?? ()
#10 0x00007f821fbeeb28 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 8 (LWP 10483):
#0 0x00007f821ae11fb9 in ?? ()
#1 0x0000600000000000 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400034018 in ?? ()
#5 0x00007f820e8bb7f0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 10482):
#0 0x00007f821ae159e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 10475):
#0 0x00007f821ae11fb9 in ?? ()
#1 0x00007f82100bea40 in ?? ()
#2 0x0000000000000144 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000361d8 in ?? ()
#5 0x00007f82100be5d0 in ?? ()
#6 0x0000000000000288 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 10474):
#0 0x00007f821ae11fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 10473):
#0 0x00007f821ae11fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 10472):
#0 0x00007f821ae11fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 10471):
#0 0x00007f82161d97a0 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 10470):
#0 0x00007f821ae15d50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x0000000000467b2b in __sanitizer::internal_alloc_placeholder ()
#3 0x00007f8215437cc0 in ?? ()
#4 0x00007f8215437cc0 in ?? ()
#5 0x00007ffeff6e0e50 in ?? ()
#6 0x000000000048aef4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000a98d3f in ?? ()
#9 0x00007f8215437cc0 in ?? ()
#10 0x00007f821933df0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250626 13:59:20.089684 9576 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID ebbfab9e778e469980d686587e3cdb81 and pid 10603
************************ BEGIN STACKS **************************
[New LWP 10604]
[New LWP 10605]
[New LWP 10606]
[New LWP 10607]
[New LWP 10608]
[New LWP 10615]
[New LWP 10616]
[New LWP 10617]
[New LWP 10620]
[New LWP 10621]
[New LWP 10622]
[New LWP 10623]
[New LWP 10624]
[New LWP 10625]
[New LWP 10626]
[New LWP 10627]
[New LWP 10628]
[New LWP 10629]
[New LWP 10630]
[New LWP 10631]
[New LWP 10632]
[New LWP 10633]
[New LWP 10634]
[New LWP 10635]
[New LWP 10636]
[New LWP 10637]
[New LWP 10638]
[New LWP 10639]
[New LWP 10640]
[New LWP 10641]
[New LWP 10642]
[New LWP 10643]
[New LWP 10644]
[New LWP 10645]
[New LWP 10646]
[New LWP 10647]
[New LWP 10648]
[New LWP 10649]
[New LWP 10650]
[New LWP 10651]
[New LWP 10652]
[New LWP 10653]
[New LWP 10654]
[New LWP 10655]
[New LWP 10656]
[New LWP 10657]
[New LWP 10658]
[New LWP 10659]
[New LWP 10660]
[New LWP 10661]
[New LWP 10662]
[New LWP 10663]
[New LWP 10664]
[New LWP 10665]
[New LWP 10666]
[New LWP 10667]
[New LWP 10668]
[New LWP 10669]
[New LWP 10670]
[New LWP 10671]
[New LWP 10672]
[New LWP 10673]
[New LWP 10674]
[New LWP 10675]
[New LWP 10676]
[New LWP 10677]
[New LWP 10678]
[New LWP 10679]
[New LWP 10680]
[New LWP 10681]
[New LWP 10682]
[New LWP 10683]
[New LWP 10684]
[New LWP 10685]
[New LWP 10686]
[New LWP 10687]
[New LWP 10688]
[New LWP 10689]
[New LWP 10690]
[New LWP 10691]
[New LWP 10692]
[New LWP 10693]
[New LWP 10694]
[New LWP 10695]
[New LWP 10696]
[New LWP 10697]
[New LWP 10698]
[New LWP 10699]
[New LWP 10700]
[New LWP 10701]
[New LWP 10702]
[New LWP 10703]
[New LWP 10704]
[New LWP 10705]
[New LWP 10706]
[New LWP 10707]
[New LWP 10708]
[New LWP 10709]
[New LWP 10710]
[New LWP 10711]
[New LWP 10712]
[New LWP 10713]
[New LWP 10714]
[New LWP 10715]
[New LWP 10716]
[New LWP 10717]
[New LWP 10718]
[New LWP 10719]
[New LWP 10720]
[New LWP 10721]
[New LWP 10722]
[New LWP 10723]
[New LWP 10724]
[New LWP 10725]
[New LWP 10726]
[New LWP 10727]
[New LWP 10728]
[New LWP 10729]
[New LWP 10730]
[New LWP 10731]
[New LWP 10732]
[New LWP 10733]
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
0x00007f439ca08d50 in ?? ()
Id Target Id Frame
* 1 LWP 10603 "kudu" 0x00007f439ca08d50 in ?? ()
2 LWP 10604 "kudu" 0x00007f4397dcc7a0 in ?? ()
3 LWP 10605 "kudu" 0x00007f439ca04fb9 in ?? ()
4 LWP 10606 "kudu" 0x00007f439ca04fb9 in ?? ()
5 LWP 10607 "kudu" 0x00007f439ca04fb9 in ?? ()
6 LWP 10608 "kernel-watcher-" 0x00007f439ca04fb9 in ?? ()
7 LWP 10615 "ntp client-1061" 0x00007f439ca089e2 in ?? ()
8 LWP 10616 "file cache-evic" 0x00007f439ca04fb9 in ?? ()
9 LWP 10617 "sq_acceptor" 0x00007f4397dfccb9 in ?? ()
10 LWP 10620 "rpc reactor-106" 0x00007f4397e09a47 in ?? ()
11 LWP 10621 "rpc reactor-106" 0x00007f4397e09a47 in ?? ()
12 LWP 10622 "rpc reactor-106" 0x00007f4397e09a47 in ?? ()
13 LWP 10623 "rpc reactor-106" 0x00007f4397e09a47 in ?? ()
14 LWP 10624 "MaintenanceMgr " 0x00007f439ca04ad3 in ?? ()
15 LWP 10625 "txn-status-mana" 0x00007f439ca04fb9 in ?? ()
16 LWP 10626 "collect_and_rem" 0x00007f439ca04fb9 in ?? ()
17 LWP 10627 "tc-session-exp-" 0x00007f439ca04fb9 in ?? ()
18 LWP 10628 "rpc worker-1062" 0x00007f439ca04ad3 in ?? ()
19 LWP 10629 "rpc worker-1062" 0x00007f439ca04ad3 in ?? ()
20 LWP 10630 "rpc worker-1063" 0x00007f439ca04ad3 in ?? ()
21 LWP 10631 "rpc worker-1063" 0x00007f439ca04ad3 in ?? ()
22 LWP 10632 "rpc worker-1063" 0x00007f439ca04ad3 in ?? ()
23 LWP 10633 "rpc worker-1063" 0x00007f439ca04ad3 in ?? ()
24 LWP 10634 "rpc worker-1063" 0x00007f439ca04ad3 in ?? ()
25 LWP 10635 "rpc worker-1063" 0x00007f439ca04ad3 in ?? ()
26 LWP 10636 "rpc worker-1063" 0x00007f439ca04ad3 in ?? ()
27 LWP 10637 "rpc worker-1063" 0x00007f439ca04ad3 in ?? ()
28 LWP 10638 "rpc worker-1063" 0x00007f439ca04ad3 in ?? ()
29 LWP 10639 "rpc worker-1063" 0x00007f439ca04ad3 in ?? ()
30 LWP 10640 "rpc worker-1064" 0x00007f439ca04ad3 in ?? ()
31 LWP 10641 "rpc worker-1064" 0x00007f439ca04ad3 in ?? ()
32 LWP 10642 "rpc worker-1064" 0x00007f439ca04ad3 in ?? ()
33 LWP 10643 "rpc worker-1064" 0x00007f439ca04ad3 in ?? ()
34 LWP 10644 "rpc worker-1064" 0x00007f439ca04ad3 in ?? ()
35 LWP 10645 "rpc worker-1064" 0x00007f439ca04ad3 in ?? ()
36 LWP 10646 "rpc worker-1064" 0x00007f439ca04ad3 in ?? ()
37 LWP 10647 "rpc worker-1064" 0x00007f439ca04ad3 in ?? ()
38 LWP 10648 "rpc worker-1064" 0x00007f439ca04ad3 in ?? ()
39 LWP 10649 "rpc worker-1064" 0x00007f439ca04ad3 in ?? ()
40 LWP 10650 "rpc worker-1065" 0x00007f439ca04ad3 in ?? ()
41 LWP 10651 "rpc worker-1065" 0x00007f439ca04ad3 in ?? ()
42 LWP 10652 "rpc worker-1065" 0x00007f439ca04ad3 in ?? ()
43 LWP 10653 "rpc worker-1065" 0x00007f439ca04ad3 in ?? ()
44 LWP 10654 "rpc worker-1065" 0x00007f439ca04ad3 in ?? ()
45 LWP 10655 "rpc worker-1065" 0x00007f439ca04ad3 in ?? ()
46 LWP 10656 "rpc worker-1065" 0x00007f439ca04ad3 in ?? ()
47 LWP 10657 "rpc worker-1065" 0x00007f439ca04ad3 in ?? ()
48 LWP 10658 "rpc worker-1065" 0x00007f439ca04ad3 in ?? ()
49 LWP 10659 "rpc worker-1065" 0x00007f439ca04ad3 in ?? ()
50 LWP 10660 "rpc worker-1066" 0x00007f439ca04ad3 in ?? ()
51 LWP 10661 "rpc worker-1066" 0x00007f439ca04ad3 in ?? ()
52 LWP 10662 "rpc worker-1066" 0x00007f439ca04ad3 in ?? ()
53 LWP 10663 "rpc worker-1066" 0x00007f439ca04ad3 in ?? ()
54 LWP 10664 "rpc worker-1066" 0x00007f439ca04ad3 in ?? ()
55 LWP 10665 "rpc worker-1066" 0x00007f439ca04ad3 in ?? ()
56 LWP 10666 "rpc worker-1066" 0x00007f439ca04ad3 in ?? ()
57 LWP 10667 "rpc worker-1066" 0x00007f439ca04ad3 in ?? ()
58 LWP 10668 "rpc worker-1066" 0x00007f439ca04ad3 in ?? ()
59 LWP 10669 "rpc worker-1066" 0x00007f439ca04ad3 in ?? ()
60 LWP 10670 "rpc worker-1067" 0x00007f439ca04ad3 in ?? ()
61 LWP 10671 "rpc worker-1067" 0x00007f439ca04ad3 in ?? ()
62 LWP 10672 "rpc worker-1067" 0x00007f439ca04ad3 in ?? ()
63 LWP 10673 "rpc worker-1067" 0x00007f439ca04ad3 in ?? ()
64 LWP 10674 "rpc worker-1067" 0x00007f439ca04ad3 in ?? ()
65 LWP 10675 "rpc worker-1067" 0x00007f439ca04ad3 in ?? ()
66 LWP 10676 "rpc worker-1067" 0x00007f439ca04ad3 in ?? ()
67 LWP 10677 "rpc worker-1067" 0x00007f439ca04ad3 in ?? ()
68 LWP 10678 "rpc worker-1067" 0x00007f439ca04ad3 in ?? ()
69 LWP 10679 "rpc worker-1067" 0x00007f439ca04ad3 in ?? ()
70 LWP 10680 "rpc worker-1068" 0x00007f439ca04ad3 in ?? ()
71 LWP 10681 "rpc worker-1068" 0x00007f439ca04ad3 in ?? ()
72 LWP 10682 "rpc worker-1068" 0x00007f439ca04ad3 in ?? ()
73 LWP 10683 "rpc worker-1068" 0x00007f439ca04ad3 in ?? ()
74 LWP 10684 "rpc worker-1068" 0x00007f439ca04ad3 in ?? ()
75 LWP 10685 "rpc worker-1068" 0x00007f439ca04ad3 in ?? ()
76 LWP 10686 "rpc worker-1068" 0x00007f439ca04ad3 in ?? ()
77 LWP 10687 "rpc worker-1068" 0x00007f439ca04ad3 in ?? ()
78 LWP 10688 "rpc worker-1068" 0x00007f439ca04ad3 in ?? ()
79 LWP 10689 "rpc worker-1068" 0x00007f439ca04ad3 in ?? ()
80 LWP 10690 "rpc worker-1069" 0x00007f439ca04ad3 in ?? ()
81 LWP 10691 "rpc worker-1069" 0x00007f439ca04ad3 in ?? ()
82 LWP 10692 "rpc worker-1069" 0x00007f439ca04ad3 in ?? ()
83 LWP 10693 "rpc worker-1069" 0x00007f439ca04ad3 in ?? ()
84 LWP 10694 "rpc worker-1069" 0x00007f439ca04ad3 in ?? ()
85 LWP 10695 "rpc worker-1069" 0x00007f439ca04ad3 in ?? ()
86 LWP 10696 "rpc worker-1069" 0x00007f439ca04ad3 in ?? ()
87 LWP 10697 "rpc worker-1069" 0x00007f439ca04ad3 in ?? ()
88 LWP 10698 "rpc worker-1069" 0x00007f439ca04ad3 in ?? ()
89 LWP 10699 "rpc worker-1069" 0x00007f439ca04ad3 in ?? ()
90 LWP 10700 "rpc worker-1070" 0x00007f439ca04ad3 in ?? ()
91 LWP 10701 "rpc worker-1070" 0x00007f439ca04ad3 in ?? ()
92 LWP 10702 "rpc worker-1070" 0x00007f439ca04ad3 in ?? ()
93 LWP 10703 "rpc worker-1070" 0x00007f439ca04ad3 in ?? ()
94 LWP 10704 "rpc worker-1070" 0x00007f439ca04ad3 in ?? ()
95 LWP 10705 "rpc worker-1070" 0x00007f439ca04ad3 in ?? ()
96 LWP 10706 "rpc worker-1070" 0x00007f439ca04ad3 in ?? ()
97 LWP 10707 "rpc worker-1070" 0x00007f439ca04ad3 in ?? ()
98 LWP 10708 "rpc worker-1070" 0x00007f439ca04ad3 in ?? ()
99 LWP 10709 "rpc worker-1070" 0x00007f439ca04ad3 in ?? ()
100 LWP 10710 "rpc worker-1071" 0x00007f439ca04ad3 in ?? ()
101 LWP 10711 "rpc worker-1071" 0x00007f439ca04ad3 in ?? ()
102 LWP 10712 "rpc worker-1071" 0x00007f439ca04ad3 in ?? ()
103 LWP 10713 "rpc worker-1071" 0x00007f439ca04ad3 in ?? ()
104 LWP 10714 "rpc worker-1071" 0x00007f439ca04ad3 in ?? ()
105 LWP 10715 "rpc worker-1071" 0x00007f439ca04ad3 in ?? ()
106 LWP 10716 "rpc worker-1071" 0x00007f439ca04ad3 in ?? ()
107 LWP 10717 "rpc worker-1071" 0x00007f439ca04ad3 in ?? ()
108 LWP 10718 "rpc worker-1071" 0x00007f439ca04ad3 in ?? ()
109 LWP 10719 "rpc worker-1071" 0x00007f439ca04ad3 in ?? ()
110 LWP 10720 "rpc worker-1072" 0x00007f439ca04ad3 in ?? ()
111 LWP 10721 "rpc worker-1072" 0x00007f439ca04ad3 in ?? ()
112 LWP 10722 "rpc worker-1072" 0x00007f439ca04ad3 in ?? ()
113 LWP 10723 "rpc worker-1072" 0x00007f439ca04ad3 in ?? ()
114 LWP 10724 "rpc worker-1072" 0x00007f439ca04ad3 in ?? ()
115 LWP 10725 "rpc worker-1072" 0x00007f439ca04ad3 in ?? ()
116 LWP 10726 "rpc worker-1072" 0x00007f439ca04ad3 in ?? ()
117 LWP 10727 "rpc worker-1072" 0x00007f439ca04ad3 in ?? ()
118 LWP 10728 "diag-logger-107" 0x00007f439ca04fb9 in ?? ()
119 LWP 10729 "result-tracker-" 0x00007f439ca04fb9 in ?? ()
120 LWP 10730 "excess-log-dele" 0x00007f439ca04fb9 in ?? ()
121 LWP 10731 "acceptor-10731" 0x00007f4397e0b0c7 in ?? ()
122 LWP 10732 "heartbeat-10732" 0x00007f439ca04fb9 in ?? ()
123 LWP 10733 "maintenance_sch" 0x00007f439ca04fb9 in ?? ()
Thread 123 (LWP 10733):
#0 0x00007f439ca04fb9 in ?? ()
#1 0x00007b0100000000 in ?? ()
#2 0x00000000000000fe in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007f4350cb96c0 in ?? ()
#6 0x00000000000001fc in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 10732):
#0 0x00007f439ca04fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 10731):
#0 0x00007f4397e0b0c7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 10730):
#0 0x00007f439ca04fb9 in ?? ()
#1 0x00007f43524bc940 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffc2e3f5900 in ?? ()
#5 0x00007f43524bc7b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 10729):
#0 0x00007f439ca04fb9 in ?? ()
#1 0x0000000085352f80 in ?? ()
#2 0x000000000000003f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001008 in ?? ()
#5 0x00007f4352cbd800 in ?? ()
#6 0x000000000000007e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 10728):
#0 0x00007f439ca04fb9 in ?? ()
#1 0x00007f4395d88008 in ?? ()
#2 0x000000000000003b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000c90 in ?? ()
#5 0x00007f43534be750 in ?? ()
#6 0x0000000000000076 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 10727):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 10726):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 10725):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 10724):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 10723):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 10722):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 10721):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 10720):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 10719):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 10718):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 10717):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 10716):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 10715):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 10714):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 10713):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 10712):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 10711):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 10710):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 10709):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 10708):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 10707):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 10706):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 10705):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 10704):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 10703):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 10702):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 10701):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 10700):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 10699):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 10698):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 10697):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 10696):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 10695):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 10694):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 10693):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 10692):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 10691):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 10690):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 10689):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 10688):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 10687):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x00000000000009ec in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b24001147c8 in ?? ()
#4 0x00007f43686ba710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f43686ba730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 76 (LWP 10686):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000862 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240010ffc8 in ?? ()
#4 0x00007f4368ebb710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4368ebb730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 10685):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 10684):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 10683):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 10682):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 10681):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 10680):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 10679):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 10678):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 10677):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 10676):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 10675):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 10674):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 10673):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 10672):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 10671):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 10670):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 10669):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 10668):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 10667):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000b902c in ?? ()
#4 0x00007f4372abc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4372abc730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x007f0400000026c1 in ?? ()
#9 0x00007f439ca04770 in ?? ()
#10 0x00007f4372abc730 in ?? ()
#11 0x0002008300000dfe in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 10666):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 10665):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 10664):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 10663):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 10662):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 10661):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 10660):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 10659):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 10658):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 10657):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 10656):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 10655):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 10654):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 10653):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 10652):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 10651):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 10650):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 10649):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 10648):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 10647):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240005ffec in ?? ()
#4 0x00007f437cebe710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f437cebe730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f439ca04770 in ?? ()
#10 0x00007f437cebe730 in ?? ()
#11 0x00007f4394e69c48 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 36 (LWP 10646):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 10645):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 10644):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 10643):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 10642):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 10641):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 10640):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 10639):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 10638):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 10637):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 10636):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 10635):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 10634):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 10633):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 10632):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 10631):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 10630):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 10629):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 10628):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 10627):
#0 0x00007f439ca04fb9 in ?? ()
#1 0x0000000017a335b8 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800003a00 in ?? ()
#5 0x00007f4387492700 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 10626):
#0 0x00007f439ca04fb9 in ?? ()
#1 0x00007f4387c939a8 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400037198 in ?? ()
#5 0x00007f4387c93840 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 10625):
#0 0x00007f439ca04fb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007f4388494410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 10624):
#0 0x00007f439ca04ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 10623):
#0 0x00007f4397e09a47 in ?? ()
#1 0x00007b0c000d2840 in ?? ()
#2 0x0048e00001e35720 in ?? ()
#3 0x00007f4389496500 in ?? ()
#4 0x00007f4389497b80 in ?? ()
#5 0x00007f4389496500 in ?? ()
#6 0x0000000000000012 in ?? ()
#7 0x00007b5800001b00 in ?? ()
#8 0x0000000000488695 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f4395576000 in ?? ()
#10 0x0000000000488599 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f4389497b80 in ?? ()
#12 0x00007f439a862069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f439ffe31a0 in ?? ()
#15 0x00007b4c00003310 in ?? ()
#16 0x00007b4c00003318 in ?? ()
#17 0x00007f43894967a0 in ?? ()
#18 0x00007b4400034980 in ?? ()
#19 0x00007f4389496cd0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 12 (LWP 10622):
#0 0x00007f4397e09a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 11 (LWP 10621):
#0 0x00007f4397e09a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 10 (LWP 10620):
#0 0x00007f4397e09a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 9 (LWP 10617):
#0 0x00007f4397dfccb9 in ?? ()
#1 0x00007f4390cbcc10 in ?? ()
#2 0x00007b040000a850 in ?? ()
#3 0x00007f4390cbdb80 in ?? ()
#4 0x00007f4390cbcc10 in ?? ()
#5 0x00007b040000a850 in ?? ()
#6 0x00000000004888a3 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007f4395778000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007f4390cbdb80 in ?? ()
#10 0x00007f43a17e1b28 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 8 (LWP 10616):
#0 0x00007f439ca04fb9 in ?? ()
#1 0x0000600000000000 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400034018 in ?? ()
#5 0x00007f43904bb7f0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 10615):
#0 0x00007f439ca089e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 10608):
#0 0x00007f439ca04fb9 in ?? ()
#1 0x00007f4391cbea40 in ?? ()
#2 0x0000000000000143 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000361d8 in ?? ()
#5 0x00007f4391cbe5d0 in ?? ()
#6 0x0000000000000286 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 10607):
#0 0x00007f439ca04fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 10606):
#0 0x00007f439ca04fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 10605):
#0 0x00007f439ca04fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 10604):
#0 0x00007f4397dcc7a0 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 10603):
#0 0x00007f439ca08d50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x0000000000467b2b in __sanitizer::internal_alloc_placeholder ()
#3 0x00007f439702acc0 in ?? ()
#4 0x00007f439702acc0 in ?? ()
#5 0x00007ffc2e3f5710 in ?? ()
#6 0x000000000048aef4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000a9afb6 in ?? ()
#9 0x00007f439702acc0 in ?? ()
#10 0x00007f439af30f0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250626 13:59:21.033707 9576 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu with pid 10337
I20250626 13:59:21.081211 9576 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu with pid 10470
I20250626 13:59:21.124634 9576 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu with pid 10603
I20250626 13:59:21.176329 9576 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskraC6OB/build/tsan/bin/kudu with pid 10245
2025-06-26T13:59:21Z chronyd exiting
I20250626 13:59:21.231904 9576 test_util.cc:183] -----------------------------------------------
I20250626 13:59:21.232127 9576 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskraC6OB/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750946267712444-9576-0
[ FAILED ] TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate (71890 ms)
[----------] 4 tests from TabletCopyITest (93356 ms total)
[----------] 1 test from FaultFlags/BadTabletCopyITest
[ RUN ] FaultFlags/BadTabletCopyITest.TestBadCopy/1
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:1510: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] FaultFlags/BadTabletCopyITest.TestBadCopy/1 (7 ms)
[----------] 1 test from FaultFlags/BadTabletCopyITest (7 ms total)
[----------] Global test environment tear-down
[==========] 5 tests from 2 test suites ran. (93365 ms total)
[ PASSED ] 1 test.
[ SKIPPED ] 3 tests, listed below:
[ SKIPPED ] TabletCopyITest.TestRejectRogueLeader
[ SKIPPED ] TabletCopyITest.TestDeleteLeaderDuringTabletCopyStressTest
[ SKIPPED ] FaultFlags/BadTabletCopyITest.TestBadCopy/1
[ FAILED ] 1 test, listed below:
[ FAILED ] TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate
1 FAILED TEST