Note: This is test shard 1 of 6.
[==========] Running 5 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 4 tests from TabletCopyITest
[ RUN ] TabletCopyITest.TestRejectRogueLeader
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:172: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] TabletCopyITest.TestRejectRogueLeader (10 ms)
[ RUN ] TabletCopyITest.TestDeleteLeaderDuringTabletCopyStressTest
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:727: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] TabletCopyITest.TestDeleteLeaderDuringTabletCopyStressTest (6 ms)
[ RUN ] TabletCopyITest.TestTabletCopyThrottling
2025-06-27T01:57:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T01:57:22Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250627 01:57:22.016829 16802 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
/tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.104.190:35051
--webserver_interface=127.16.104.190
--webserver_port=0
--builtin_ntp_servers=127.16.104.148:44673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.104.190:35051
--master_tombstone_evicted_tablet_replicas=false with env {}
W20250627 01:57:22.321789 16811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 01:57:22.322412 16811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 01:57:22.322898 16811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 01:57:22.354452 16811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 01:57:22.354799 16811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 01:57:22.355074 16811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 01:57:22.355316 16811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 01:57:22.390450 16811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.104.148:44673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/wal
--master_tombstone_evicted_tablet_replicas=false
--ipki_ca_key_size=768
--master_addresses=127.16.104.190:35051
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.104.190:35051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.104.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 27 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6789
TSAN enabled
I20250627 01:57:22.391814 16811 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 01:57:22.393577 16811 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 01:57:22.410236 16818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 01:57:22.410267 16817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:22.412163 16811 server_base.cc:1048] running on GCE node
W20250627 01:57:22.411914 16820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:23.607738 16811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 01:57:23.611753 16811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 01:57:23.613253 16811 hybrid_clock.cc:648] HybridClock initialized: now 1750989443613194 us; error 57 us; skew 500 ppm
I20250627 01:57:23.614107 16811 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 01:57:23.621621 16811 webserver.cc:469] Webserver started at http://127.16.104.190:43117/ using document root <none> and password file <none>
I20250627 01:57:23.622574 16811 fs_manager.cc:362] Metadata directory not provided
I20250627 01:57:23.622769 16811 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 01:57:23.623204 16811 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 01:57:23.627656 16811 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data/instance:
uuid: "d867691000db4a76af1e1e6b8545682c"
format_stamp: "Formatted at 2025-06-27 01:57:23 on dist-test-slave-dvrl"
I20250627 01:57:23.628824 16811 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/wal/instance:
uuid: "d867691000db4a76af1e1e6b8545682c"
format_stamp: "Formatted at 2025-06-27 01:57:23 on dist-test-slave-dvrl"
I20250627 01:57:23.636441 16811 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250627 01:57:23.642746 16827 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:23.643870 16811 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250627 01:57:23.644227 16811 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data,/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/wal
uuid: "d867691000db4a76af1e1e6b8545682c"
format_stamp: "Formatted at 2025-06-27 01:57:23 on dist-test-slave-dvrl"
I20250627 01:57:23.644562 16811 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 01:57:23.696517 16811 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 01:57:23.698019 16811 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 01:57:23.698436 16811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 01:57:23.778090 16811 rpc_server.cc:307] RPC server started. Bound to: 127.16.104.190:35051
I20250627 01:57:23.778167 16878 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.104.190:35051 every 8 connection(s)
I20250627 01:57:23.780735 16811 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data/info.pb
I20250627 01:57:23.783155 16802 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu as pid 16811
I20250627 01:57:23.783746 16802 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/wal/instance
I20250627 01:57:23.787293 16879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 01:57:23.806617 16879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c: Bootstrap starting.
I20250627 01:57:23.814504 16879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c: Neither blocks nor log segments found. Creating new log.
I20250627 01:57:23.816987 16879 log.cc:826] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c: Log is configured to *not* fsync() on all Append() calls
I20250627 01:57:23.822371 16879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c: No bootstrap required, opened a new log
I20250627 01:57:23.841198 16879 raft_consensus.cc:357] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867691000db4a76af1e1e6b8545682c" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 35051 } }
I20250627 01:57:23.842396 16879 raft_consensus.cc:383] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 01:57:23.842648 16879 raft_consensus.cc:738] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d867691000db4a76af1e1e6b8545682c, State: Initialized, Role: FOLLOWER
I20250627 01:57:23.843420 16879 consensus_queue.cc:260] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867691000db4a76af1e1e6b8545682c" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 35051 } }
I20250627 01:57:23.843936 16879 raft_consensus.cc:397] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 01:57:23.844193 16879 raft_consensus.cc:491] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 01:57:23.844527 16879 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 0 FOLLOWER]: Advancing to term 1
I20250627 01:57:23.848604 16879 raft_consensus.cc:513] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867691000db4a76af1e1e6b8545682c" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 35051 } }
I20250627 01:57:23.849370 16879 leader_election.cc:304] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d867691000db4a76af1e1e6b8545682c; no voters:
I20250627 01:57:23.851397 16879 leader_election.cc:290] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 01:57:23.852104 16884 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 1 FOLLOWER]: Leader election won for term 1
I20250627 01:57:23.855573 16884 raft_consensus.cc:695] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 1 LEADER]: Becoming Leader. State: Replica: d867691000db4a76af1e1e6b8545682c, State: Running, Role: LEADER
I20250627 01:57:23.856272 16884 consensus_queue.cc:237] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867691000db4a76af1e1e6b8545682c" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 35051 } }
I20250627 01:57:23.856914 16879 sys_catalog.cc:564] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [sys.catalog]: configured and running, proceeding with master startup.
I20250627 01:57:23.870431 16885 sys_catalog.cc:455] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d867691000db4a76af1e1e6b8545682c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867691000db4a76af1e1e6b8545682c" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 35051 } } }
I20250627 01:57:23.870326 16886 sys_catalog.cc:455] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d867691000db4a76af1e1e6b8545682c. Latest consensus state: current_term: 1 leader_uuid: "d867691000db4a76af1e1e6b8545682c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867691000db4a76af1e1e6b8545682c" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 35051 } } }
I20250627 01:57:23.871019 16885 sys_catalog.cc:458] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [sys.catalog]: This master's current role is: LEADER
I20250627 01:57:23.871264 16886 sys_catalog.cc:458] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [sys.catalog]: This master's current role is: LEADER
I20250627 01:57:23.876623 16892 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 01:57:23.888801 16892 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 01:57:23.907478 16892 catalog_manager.cc:1349] Generated new cluster ID: 54ef5da74a114fed803c7a44cf794e0a
I20250627 01:57:23.907826 16892 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 01:57:23.946110 16892 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 01:57:23.947703 16892 catalog_manager.cc:1506] Loading token signing keys...
I20250627 01:57:23.966758 16892 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c: Generated new TSK 0
I20250627 01:57:23.967635 16892 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 01:57:23.982479 16802 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
/tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.104.129:0
--local_ip_for_outbound_sockets=127.16.104.129
--webserver_interface=127.16.104.129
--webserver_port=0
--tserver_master_addrs=127.16.104.190:35051
--builtin_ntp_servers=127.16.104.148:44673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--num_tablets_to_copy_simultaneously=1 with env {}
W20250627 01:57:24.296950 16903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 01:57:24.297514 16903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 01:57:24.298058 16903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 01:57:24.331660 16903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 01:57:24.332568 16903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.104.129
I20250627 01:57:24.368391 16903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.104.148:44673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.104.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.104.129
--webserver_port=0
--tserver_master_addrs=127.16.104.190:35051
--num_tablets_to_copy_simultaneously=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.104.129
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 27 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6789
TSAN enabled
I20250627 01:57:24.369926 16903 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 01:57:24.371608 16903 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 01:57:24.391399 16910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:24.396126 16903 server_base.cc:1048] running on GCE node
W20250627 01:57:24.394802 16909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 01:57:24.395773 16912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:25.598083 16903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 01:57:25.600791 16903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 01:57:25.602242 16903 hybrid_clock.cc:648] HybridClock initialized: now 1750989445602225 us; error 78 us; skew 500 ppm
I20250627 01:57:25.603056 16903 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 01:57:25.610539 16903 webserver.cc:469] Webserver started at http://127.16.104.129:39867/ using document root <none> and password file <none>
I20250627 01:57:25.611490 16903 fs_manager.cc:362] Metadata directory not provided
I20250627 01:57:25.611719 16903 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 01:57:25.612190 16903 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 01:57:25.617084 16903 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/data/instance:
uuid: "51d9045707a34a03b7fc6f5141a72cdf"
format_stamp: "Formatted at 2025-06-27 01:57:25 on dist-test-slave-dvrl"
I20250627 01:57:25.618247 16903 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/wal/instance:
uuid: "51d9045707a34a03b7fc6f5141a72cdf"
format_stamp: "Formatted at 2025-06-27 01:57:25 on dist-test-slave-dvrl"
I20250627 01:57:25.626421 16903 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20250627 01:57:25.633399 16919 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:25.634627 16903 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20250627 01:57:25.634984 16903 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/wal
uuid: "51d9045707a34a03b7fc6f5141a72cdf"
format_stamp: "Formatted at 2025-06-27 01:57:25 on dist-test-slave-dvrl"
I20250627 01:57:25.635326 16903 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 01:57:25.719795 16903 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 01:57:25.721308 16903 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 01:57:25.721755 16903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 01:57:25.724345 16903 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 01:57:25.728996 16903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 01:57:25.729211 16903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:25.729521 16903 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 01:57:25.729681 16903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:25.896867 16903 rpc_server.cc:307] RPC server started. Bound to: 127.16.104.129:46597
I20250627 01:57:25.897043 17031 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.104.129:46597 every 8 connection(s)
I20250627 01:57:25.899497 16903 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/data/info.pb
I20250627 01:57:25.907223 16802 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu as pid 16903
I20250627 01:57:25.907625 16802 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-0/wal/instance
I20250627 01:57:25.913375 16802 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
/tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.104.130:0
--local_ip_for_outbound_sockets=127.16.104.130
--webserver_interface=127.16.104.130
--webserver_port=0
--tserver_master_addrs=127.16.104.190:35051
--builtin_ntp_servers=127.16.104.148:44673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--num_tablets_to_copy_simultaneously=1 with env {}
I20250627 01:57:25.923754 17032 heartbeater.cc:344] Connected to a master server at 127.16.104.190:35051
I20250627 01:57:25.924245 17032 heartbeater.cc:461] Registering TS with master...
I20250627 01:57:25.925401 17032 heartbeater.cc:507] Master 127.16.104.190:35051 requested a full tablet report, sending...
I20250627 01:57:25.929064 16844 ts_manager.cc:194] Registered new tserver with Master: 51d9045707a34a03b7fc6f5141a72cdf (127.16.104.129:46597)
I20250627 01:57:25.931079 16844 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.104.129:59301
W20250627 01:57:26.238052 17036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 01:57:26.238585 17036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 01:57:26.239154 17036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 01:57:26.270913 17036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 01:57:26.271847 17036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.104.130
I20250627 01:57:26.306749 17036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.104.148:44673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.104.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.104.130
--webserver_port=0
--tserver_master_addrs=127.16.104.190:35051
--num_tablets_to_copy_simultaneously=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.104.130
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 27 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6789
TSAN enabled
I20250627 01:57:26.308116 17036 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 01:57:26.309820 17036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 01:57:26.328747 17042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:26.934928 17032 heartbeater.cc:499] Master 127.16.104.190:35051 was elected leader, sending a full tablet report...
W20250627 01:57:26.328825 17043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 01:57:26.330214 17045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:26.331697 17036 server_base.cc:1048] running on GCE node
I20250627 01:57:27.520991 17036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 01:57:27.523842 17036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 01:57:27.525326 17036 hybrid_clock.cc:648] HybridClock initialized: now 1750989447525286 us; error 66 us; skew 500 ppm
I20250627 01:57:27.526117 17036 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 01:57:27.533643 17036 webserver.cc:469] Webserver started at http://127.16.104.130:43189/ using document root <none> and password file <none>
I20250627 01:57:27.534543 17036 fs_manager.cc:362] Metadata directory not provided
I20250627 01:57:27.534735 17036 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 01:57:27.535138 17036 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 01:57:27.539553 17036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data/instance:
uuid: "b2454b3c7ee949e387890880e3129cc8"
format_stamp: "Formatted at 2025-06-27 01:57:27 on dist-test-slave-dvrl"
I20250627 01:57:27.540628 17036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/wal/instance:
uuid: "b2454b3c7ee949e387890880e3129cc8"
format_stamp: "Formatted at 2025-06-27 01:57:27 on dist-test-slave-dvrl"
I20250627 01:57:27.548363 17036 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20250627 01:57:27.554041 17052 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:27.555207 17036 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250627 01:57:27.555510 17036 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/wal
uuid: "b2454b3c7ee949e387890880e3129cc8"
format_stamp: "Formatted at 2025-06-27 01:57:27 on dist-test-slave-dvrl"
I20250627 01:57:27.555830 17036 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 01:57:27.614569 17036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 01:57:27.615998 17036 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 01:57:27.616410 17036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 01:57:27.619226 17036 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 01:57:27.623267 17036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 01:57:27.623521 17036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:27.623775 17036 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 01:57:27.623955 17036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:27.784013 17036 rpc_server.cc:307] RPC server started. Bound to: 127.16.104.130:44271
I20250627 01:57:27.784089 17164 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.104.130:44271 every 8 connection(s)
I20250627 01:57:27.787566 17036 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data/info.pb
I20250627 01:57:27.788472 16802 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu as pid 17036
I20250627 01:57:27.788841 16802 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/wal/instance
I20250627 01:57:27.817800 17165 heartbeater.cc:344] Connected to a master server at 127.16.104.190:35051
I20250627 01:57:27.818287 17165 heartbeater.cc:461] Registering TS with master...
I20250627 01:57:27.819303 17165 heartbeater.cc:507] Master 127.16.104.190:35051 requested a full tablet report, sending...
I20250627 01:57:27.821265 16843 ts_manager.cc:194] Registered new tserver with Master: b2454b3c7ee949e387890880e3129cc8 (127.16.104.130:44271)
I20250627 01:57:27.822571 16843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.104.130:34035
I20250627 01:57:27.833992 16802 external_mini_cluster.cc:934] 2 TS(s) registered with all masters
I20250627 01:57:27.869350 16802 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu with pid 17036
I20250627 01:57:27.892745 16802 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu with pid 16811
I20250627 01:57:27.924427 16802 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
/tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.104.190:35051
--webserver_interface=127.16.104.190
--webserver_port=43117
--builtin_ntp_servers=127.16.104.148:44673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.104.190:35051
--master_tombstone_evicted_tablet_replicas=false with env {}
W20250627 01:57:27.944339 17032 heartbeater.cc:646] Failed to heartbeat to 127.16.104.190:35051 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.104.190:35051: connect: Connection refused (error 111)
W20250627 01:57:29.472931 17177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 01:57:29.473577 17177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 01:57:29.474071 17177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 01:57:29.505578 17177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 01:57:29.505916 17177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 01:57:29.506174 17177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 01:57:29.506410 17177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 01:57:29.540822 17177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.104.148:44673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/wal
--master_tombstone_evicted_tablet_replicas=false
--ipki_ca_key_size=768
--master_addresses=127.16.104.190:35051
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.104.190:35051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.104.190
--webserver_port=43117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 27 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6789
TSAN enabled
I20250627 01:57:29.542191 17177 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 01:57:29.543807 17177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 01:57:29.558310 17183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 01:57:29.558389 17186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 01:57:29.558311 17184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:29.560560 17177 server_base.cc:1048] running on GCE node
I20250627 01:57:30.754074 17177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 01:57:30.756669 17177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 01:57:30.758057 17177 hybrid_clock.cc:648] HybridClock initialized: now 1750989450758027 us; error 48 us; skew 500 ppm
I20250627 01:57:30.758930 17177 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 01:57:30.765555 17177 webserver.cc:469] Webserver started at http://127.16.104.190:43117/ using document root <none> and password file <none>
I20250627 01:57:30.766543 17177 fs_manager.cc:362] Metadata directory not provided
I20250627 01:57:30.766767 17177 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 01:57:30.775002 17177 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250627 01:57:30.779966 17194 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:30.781128 17177 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250627 01:57:30.781574 17177 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data,/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/wal
uuid: "d867691000db4a76af1e1e6b8545682c"
format_stamp: "Formatted at 2025-06-27 01:57:23 on dist-test-slave-dvrl"
I20250627 01:57:30.783754 17177 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 01:57:30.838315 17177 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 01:57:30.839823 17177 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 01:57:30.840291 17177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 01:57:30.914911 17177 rpc_server.cc:307] RPC server started. Bound to: 127.16.104.190:35051
I20250627 01:57:30.915011 17245 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.104.190:35051 every 8 connection(s)
I20250627 01:57:30.917798 17177 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/master-0/data/info.pb
I20250627 01:57:30.921845 16802 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu as pid 17177
I20250627 01:57:30.930557 17246 sys_catalog.cc:263] Verifying existing consensus state
I20250627 01:57:30.936291 17246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c: Bootstrap starting.
I20250627 01:57:30.966004 17032 heartbeater.cc:344] Connected to a master server at 127.16.104.190:35051
I20250627 01:57:30.980858 17246 log.cc:826] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c: Log is configured to *not* fsync() on all Append() calls
I20250627 01:57:30.996695 17246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 01:57:30.997593 17246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c: Bootstrap complete.
I20250627 01:57:31.016760 17246 raft_consensus.cc:357] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867691000db4a76af1e1e6b8545682c" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 35051 } }
I20250627 01:57:31.019050 17246 raft_consensus.cc:738] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d867691000db4a76af1e1e6b8545682c, State: Initialized, Role: FOLLOWER
I20250627 01:57:31.019842 17246 consensus_queue.cc:260] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867691000db4a76af1e1e6b8545682c" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 35051 } }
I20250627 01:57:31.020413 17246 raft_consensus.cc:397] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 01:57:31.020694 17246 raft_consensus.cc:491] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 01:57:31.021052 17246 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 1 FOLLOWER]: Advancing to term 2
I20250627 01:57:31.027133 17246 raft_consensus.cc:513] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867691000db4a76af1e1e6b8545682c" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 35051 } }
I20250627 01:57:31.027915 17246 leader_election.cc:304] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d867691000db4a76af1e1e6b8545682c; no voters:
I20250627 01:57:31.029825 17246 leader_election.cc:290] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [CANDIDATE]: Term 2 election: Requested vote from peers
I20250627 01:57:31.030316 17252 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 2 FOLLOWER]: Leader election won for term 2
I20250627 01:57:31.033811 17252 raft_consensus.cc:695] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [term 2 LEADER]: Becoming Leader. State: Replica: d867691000db4a76af1e1e6b8545682c, State: Running, Role: LEADER
I20250627 01:57:31.034575 17246 sys_catalog.cc:564] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [sys.catalog]: configured and running, proceeding with master startup.
I20250627 01:57:31.034590 17252 consensus_queue.cc:237] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867691000db4a76af1e1e6b8545682c" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 35051 } }
I20250627 01:57:31.046402 17253 sys_catalog.cc:455] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d867691000db4a76af1e1e6b8545682c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867691000db4a76af1e1e6b8545682c" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 35051 } } }
I20250627 01:57:31.047683 17253 sys_catalog.cc:458] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [sys.catalog]: This master's current role is: LEADER
I20250627 01:57:31.050000 17254 sys_catalog.cc:455] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d867691000db4a76af1e1e6b8545682c. Latest consensus state: current_term: 2 leader_uuid: "d867691000db4a76af1e1e6b8545682c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d867691000db4a76af1e1e6b8545682c" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 35051 } } }
I20250627 01:57:31.050993 17254 sys_catalog.cc:458] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c [sys.catalog]: This master's current role is: LEADER
I20250627 01:57:31.051746 17260 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 01:57:31.063540 17260 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 01:57:31.068547 17260 catalog_manager.cc:1261] Loaded cluster ID: 54ef5da74a114fed803c7a44cf794e0a
I20250627 01:57:31.068919 17260 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 01:57:31.075759 17260 catalog_manager.cc:1506] Loading token signing keys...
I20250627 01:57:31.080838 17260 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d867691000db4a76af1e1e6b8545682c: Loaded TSK: 0
I20250627 01:57:31.082872 17260 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 01:57:31.974225 17211 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" instance_seqno: 1750989445856416) as {username='slave'} at 127.16.104.129:40289; Asking this server to re-register.
I20250627 01:57:31.975950 17032 heartbeater.cc:461] Registering TS with master...
I20250627 01:57:31.976648 17032 heartbeater.cc:507] Master 127.16.104.190:35051 requested a full tablet report, sending...
I20250627 01:57:31.979280 17211 ts_manager.cc:194] Registered new tserver with Master: 51d9045707a34a03b7fc6f5141a72cdf (127.16.104.129:46597)
I20250627 01:57:31.991299 16802 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250627 01:57:31.991921 16802 test_util.cc:276] Using random seed: 4889433
I20250627 01:57:32.048622 17211 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46688:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250627 01:57:32.111869 16965 tablet_service.cc:1468] Processing CreateTablet for tablet 009072bc698b442d96e8c2988ff9e05a (DEFAULT_TABLE table=test-workload [id=faaf3b5db7c24139861a9034e6b28b95]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250627 01:57:32.111858 16967 tablet_service.cc:1468] Processing CreateTablet for tablet 10b468f09d0749e8b6c723948379cb9b (DEFAULT_TABLE table=test-workload [id=faaf3b5db7c24139861a9034e6b28b95]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250627 01:57:32.112287 16964 tablet_service.cc:1468] Processing CreateTablet for tablet 20c59250b06040f3968dcd6c31060df3 (DEFAULT_TABLE table=test-workload [id=faaf3b5db7c24139861a9034e6b28b95]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250627 01:57:32.112887 16966 tablet_service.cc:1468] Processing CreateTablet for tablet 5e77f449263c4c1e838ed55fb80982b7 (DEFAULT_TABLE table=test-workload [id=faaf3b5db7c24139861a9034e6b28b95]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250627 01:57:32.114698 16965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009072bc698b442d96e8c2988ff9e05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 01:57:32.115847 16967 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10b468f09d0749e8b6c723948379cb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 01:57:32.117842 16964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20c59250b06040f3968dcd6c31060df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 01:57:32.118772 16966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e77f449263c4c1e838ed55fb80982b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 01:57:32.154810 17283 tablet_bootstrap.cc:492] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf: Bootstrap starting.
I20250627 01:57:32.210994 17283 tablet_bootstrap.cc:654] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf: Neither blocks nor log segments found. Creating new log.
I20250627 01:57:32.214406 17283 log.cc:826] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf: Log is configured to *not* fsync() on all Append() calls
I20250627 01:57:32.221174 17283 tablet_bootstrap.cc:492] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf: No bootstrap required, opened a new log
I20250627 01:57:32.221809 17283 ts_tablet_manager.cc:1397] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf: Time spent bootstrapping tablet: real 0.068s user 0.018s sys 0.000s
I20250627 01:57:32.252157 17283 raft_consensus.cc:357] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.253155 17283 raft_consensus.cc:383] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 01:57:32.253511 17283 raft_consensus.cc:738] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d9045707a34a03b7fc6f5141a72cdf, State: Initialized, Role: FOLLOWER
I20250627 01:57:32.254516 17283 consensus_queue.cc:260] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.255276 17283 raft_consensus.cc:397] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 01:57:32.255800 17283 raft_consensus.cc:491] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 01:57:32.256218 17283 raft_consensus.cc:3058] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Advancing to term 1
I20250627 01:57:32.320947 17283 raft_consensus.cc:513] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.322031 17283 leader_election.cc:304] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51d9045707a34a03b7fc6f5141a72cdf; no voters:
I20250627 01:57:32.324921 17283 leader_election.cc:290] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 01:57:32.325352 17285 raft_consensus.cc:2802] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf [term 1 FOLLOWER]: Leader election won for term 1
I20250627 01:57:32.327852 17285 raft_consensus.cc:695] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf [term 1 LEADER]: Becoming Leader. State: Replica: 51d9045707a34a03b7fc6f5141a72cdf, State: Running, Role: LEADER
I20250627 01:57:32.328823 17285 consensus_queue.cc:237] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.332208 17283 ts_tablet_manager.cc:1428] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf: Time spent starting tablet: real 0.110s user 0.046s sys 0.004s
I20250627 01:57:32.336284 17283 tablet_bootstrap.cc:492] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf: Bootstrap starting.
I20250627 01:57:32.343830 17283 tablet_bootstrap.cc:654] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf: Neither blocks nor log segments found. Creating new log.
I20250627 01:57:32.353881 17283 tablet_bootstrap.cc:492] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf: No bootstrap required, opened a new log
I20250627 01:57:32.353423 17211 catalog_manager.cc:5582] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 51d9045707a34a03b7fc6f5141a72cdf (127.16.104.129). New cstate: current_term: 1 leader_uuid: "51d9045707a34a03b7fc6f5141a72cdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } health_report { overall_health: HEALTHY } } }
I20250627 01:57:32.354382 17283 ts_tablet_manager.cc:1397] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20250627 01:57:32.359702 17283 raft_consensus.cc:357] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.360744 17283 raft_consensus.cc:383] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 01:57:32.361070 17283 raft_consensus.cc:738] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d9045707a34a03b7fc6f5141a72cdf, State: Initialized, Role: FOLLOWER
I20250627 01:57:32.362326 17283 consensus_queue.cc:260] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.363783 17283 raft_consensus.cc:397] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 01:57:32.364104 17283 raft_consensus.cc:491] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 01:57:32.364624 17283 raft_consensus.cc:3058] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Advancing to term 1
I20250627 01:57:32.420292 17283 raft_consensus.cc:513] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.421177 17283 leader_election.cc:304] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51d9045707a34a03b7fc6f5141a72cdf; no voters:
I20250627 01:57:32.422287 17283 leader_election.cc:290] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 01:57:32.422454 17285 raft_consensus.cc:2802] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf [term 1 FOLLOWER]: Leader election won for term 1
I20250627 01:57:32.422964 17285 raft_consensus.cc:695] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf [term 1 LEADER]: Becoming Leader. State: Replica: 51d9045707a34a03b7fc6f5141a72cdf, State: Running, Role: LEADER
I20250627 01:57:32.423748 17285 consensus_queue.cc:237] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.424386 17283 ts_tablet_manager.cc:1428] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf: Time spent starting tablet: real 0.069s user 0.016s sys 0.004s
I20250627 01:57:32.425509 17283 tablet_bootstrap.cc:492] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf: Bootstrap starting.
I20250627 01:57:32.432088 17283 tablet_bootstrap.cc:654] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf: Neither blocks nor log segments found. Creating new log.
I20250627 01:57:32.432796 17211 catalog_manager.cc:5582] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 51d9045707a34a03b7fc6f5141a72cdf (127.16.104.129). New cstate: current_term: 1 leader_uuid: "51d9045707a34a03b7fc6f5141a72cdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } health_report { overall_health: HEALTHY } } }
I20250627 01:57:32.437987 17283 tablet_bootstrap.cc:492] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf: No bootstrap required, opened a new log
I20250627 01:57:32.439363 17283 ts_tablet_manager.cc:1397] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20250627 01:57:32.441771 17283 raft_consensus.cc:357] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.442912 17283 raft_consensus.cc:383] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 01:57:32.443151 17283 raft_consensus.cc:738] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d9045707a34a03b7fc6f5141a72cdf, State: Initialized, Role: FOLLOWER
I20250627 01:57:32.443754 17283 consensus_queue.cc:260] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.444339 17283 raft_consensus.cc:397] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 01:57:32.444620 17283 raft_consensus.cc:491] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 01:57:32.444943 17283 raft_consensus.cc:3058] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Advancing to term 1
I20250627 01:57:32.521616 17283 raft_consensus.cc:513] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.522541 17283 leader_election.cc:304] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51d9045707a34a03b7fc6f5141a72cdf; no voters:
I20250627 01:57:32.523319 17283 leader_election.cc:290] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 01:57:32.523480 17285 raft_consensus.cc:2802] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf [term 1 FOLLOWER]: Leader election won for term 1
I20250627 01:57:32.524178 17285 raft_consensus.cc:695] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf [term 1 LEADER]: Becoming Leader. State: Replica: 51d9045707a34a03b7fc6f5141a72cdf, State: Running, Role: LEADER
I20250627 01:57:32.524863 17285 consensus_queue.cc:237] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.525494 17283 ts_tablet_manager.cc:1428] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf: Time spent starting tablet: real 0.086s user 0.008s sys 0.006s
I20250627 01:57:32.527040 17283 tablet_bootstrap.cc:492] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf: Bootstrap starting.
I20250627 01:57:32.532779 17211 catalog_manager.cc:5582] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 51d9045707a34a03b7fc6f5141a72cdf (127.16.104.129). New cstate: current_term: 1 leader_uuid: "51d9045707a34a03b7fc6f5141a72cdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } health_report { overall_health: HEALTHY } } }
I20250627 01:57:32.533978 17283 tablet_bootstrap.cc:654] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf: Neither blocks nor log segments found. Creating new log.
I20250627 01:57:32.540125 17283 tablet_bootstrap.cc:492] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf: No bootstrap required, opened a new log
I20250627 01:57:32.540666 17283 ts_tablet_manager.cc:1397] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20250627 01:57:32.543205 17283 raft_consensus.cc:357] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.544045 17283 raft_consensus.cc:383] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 01:57:32.544410 17283 raft_consensus.cc:738] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d9045707a34a03b7fc6f5141a72cdf, State: Initialized, Role: FOLLOWER
I20250627 01:57:32.545091 17283 consensus_queue.cc:260] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.545756 17283 raft_consensus.cc:397] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 01:57:32.546044 17283 raft_consensus.cc:491] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 01:57:32.546448 17283 raft_consensus.cc:3058] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf [term 0 FOLLOWER]: Advancing to term 1
I20250627 01:57:32.619946 17283 raft_consensus.cc:513] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.621055 17283 leader_election.cc:304] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51d9045707a34a03b7fc6f5141a72cdf; no voters:
I20250627 01:57:32.622067 17283 leader_election.cc:290] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 01:57:32.622229 17285 raft_consensus.cc:2802] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf [term 1 FOLLOWER]: Leader election won for term 1
I20250627 01:57:32.623836 17285 raft_consensus.cc:695] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf [term 1 LEADER]: Becoming Leader. State: Replica: 51d9045707a34a03b7fc6f5141a72cdf, State: Running, Role: LEADER
I20250627 01:57:32.624210 17283 ts_tablet_manager.cc:1428] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf: Time spent starting tablet: real 0.083s user 0.013s sys 0.000s
I20250627 01:57:32.624604 17285 consensus_queue.cc:237] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:32.631879 17211 catalog_manager.cc:5582] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 51d9045707a34a03b7fc6f5141a72cdf (127.16.104.129). New cstate: current_term: 1 leader_uuid: "51d9045707a34a03b7fc6f5141a72cdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } health_report { overall_health: HEALTHY } } }
W20250627 01:57:33.338119 17303 meta_cache.cc:1261] Time spent looking up entry by key: real 0.058s user 0.002s sys 0.000s
I20250627 01:57:37.273428 16802 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
/tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.104.130:44271
--local_ip_for_outbound_sockets=127.16.104.130
--tserver_master_addrs=127.16.104.190:35051
--webserver_port=43189
--webserver_interface=127.16.104.130
--builtin_ntp_servers=127.16.104.148:44673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--num_tablets_to_copy_simultaneously=1 with env {}
W20250627 01:57:37.593852 17320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 01:57:37.594327 17320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 01:57:37.594909 17320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 01:57:37.626379 17320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 01:57:37.627635 17320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.104.130
I20250627 01:57:37.662843 17320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.104.148:44673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.104.130:44271
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.104.130
--webserver_port=43189
--tserver_master_addrs=127.16.104.190:35051
--num_tablets_to_copy_simultaneously=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.104.130
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 27 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6789
TSAN enabled
I20250627 01:57:37.664113 17320 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 01:57:37.665772 17320 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 01:57:37.686075 17329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 01:57:37.686066 17327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:37.688055 17320 server_base.cc:1048] running on GCE node
W20250627 01:57:37.686504 17326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:38.862416 17320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 01:57:38.865126 17320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 01:57:38.866568 17320 hybrid_clock.cc:648] HybridClock initialized: now 1750989458866538 us; error 40 us; skew 500 ppm
I20250627 01:57:38.867442 17320 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 01:57:38.874150 17320 webserver.cc:469] Webserver started at http://127.16.104.130:43189/ using document root <none> and password file <none>
I20250627 01:57:38.875113 17320 fs_manager.cc:362] Metadata directory not provided
I20250627 01:57:38.875347 17320 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 01:57:38.883193 17320 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250627 01:57:38.888286 17336 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:38.889443 17320 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250627 01:57:38.889765 17320 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/wal
uuid: "b2454b3c7ee949e387890880e3129cc8"
format_stamp: "Formatted at 2025-06-27 01:57:27 on dist-test-slave-dvrl"
I20250627 01:57:38.891736 17320 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 01:57:38.954306 17320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 01:57:38.955909 17320 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 01:57:38.956344 17320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 01:57:38.959333 17320 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 01:57:38.963375 17320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 01:57:38.963613 17320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:38.963862 17320 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 01:57:38.964023 17320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:39.121809 17320 rpc_server.cc:307] RPC server started. Bound to: 127.16.104.130:44271
I20250627 01:57:39.121879 17448 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.104.130:44271 every 8 connection(s)
I20250627 01:57:39.124641 17320 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestTabletCopyThrottling.1750989441767164-16802-0/minicluster-data/ts-1/data/info.pb
I20250627 01:57:39.130100 16802 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu as pid 17320
I20250627 01:57:39.147974 17449 heartbeater.cc:344] Connected to a master server at 127.16.104.190:35051
I20250627 01:57:39.148491 17449 heartbeater.cc:461] Registering TS with master...
I20250627 01:57:39.149644 17449 heartbeater.cc:507] Master 127.16.104.190:35051 requested a full tablet report, sending...
I20250627 01:57:39.152647 17455 ts_tablet_manager.cc:927] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8: Initiating tablet copy from peer 51d9045707a34a03b7fc6f5141a72cdf (127.16.104.129:46597)
I20250627 01:57:39.152652 17211 ts_manager.cc:194] Registered new tserver with Master: b2454b3c7ee949e387890880e3129cc8 (127.16.104.130:44271)
I20250627 01:57:39.155050 17211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.104.130:60061
I20250627 01:57:39.155362 17455 tablet_copy_client.cc:323] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.104.129:46597
I20250627 01:57:39.164623 17007 tablet_copy_service.cc:140] P 51d9045707a34a03b7fc6f5141a72cdf: Received BeginTabletCopySession request for tablet 009072bc698b442d96e8c2988ff9e05a from peer b2454b3c7ee949e387890880e3129cc8 ({username='slave'} at 127.16.104.130:48963)
I20250627 01:57:39.165169 17007 tablet_copy_service.cc:161] P 51d9045707a34a03b7fc6f5141a72cdf: Beginning new tablet copy session on tablet 009072bc698b442d96e8c2988ff9e05a from peer b2454b3c7ee949e387890880e3129cc8 at {username='slave'} at 127.16.104.130:48963: session id = b2454b3c7ee949e387890880e3129cc8-009072bc698b442d96e8c2988ff9e05a
I20250627 01:57:39.173207 17007 tablet_copy_source_session.cc:215] T 009072bc698b442d96e8c2988ff9e05a P 51d9045707a34a03b7fc6f5141a72cdf: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 01:57:39.178416 17455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009072bc698b442d96e8c2988ff9e05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 01:57:39.197197 17455 tablet_copy_client.cc:806] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8: tablet copy: Starting download of 0 data blocks...
I20250627 01:57:39.197772 17455 tablet_copy_client.cc:670] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8: tablet copy: Starting download of 1 WAL segments...
I20250627 01:57:39.210903 17455 tablet_copy_client.cc:538] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 01:57:39.218266 17455 tablet_bootstrap.cc:492] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8: Bootstrap starting.
I20250627 01:57:39.341887 17455 log.cc:826] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8: Log is configured to *not* fsync() on all Append() calls
I20250627 01:57:40.159252 17449 heartbeater.cc:499] Master 127.16.104.190:35051 was elected leader, sending a full tablet report...
I20250627 01:57:40.312608 17455 tablet_bootstrap.cc:492] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=212 overwritten=0 applied=212 ignored=0} inserts{seen=2676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 01:57:40.313395 17455 tablet_bootstrap.cc:492] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8: Bootstrap complete.
I20250627 01:57:40.314076 17455 ts_tablet_manager.cc:1397] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8: Time spent bootstrapping tablet: real 1.096s user 1.054s sys 0.040s
I20250627 01:57:40.326823 17455 raft_consensus.cc:357] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:40.327673 17455 raft_consensus.cc:738] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: b2454b3c7ee949e387890880e3129cc8, State: Initialized, Role: NON_PARTICIPANT
I20250627 01:57:40.328336 17455 consensus_queue.cc:260] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 212, Last appended: 1.212, Last appended by leader: 212, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:40.331386 17455 ts_tablet_manager.cc:1428] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20250627 01:57:40.334239 17007 tablet_copy_service.cc:342] P 51d9045707a34a03b7fc6f5141a72cdf: Request end of tablet copy session b2454b3c7ee949e387890880e3129cc8-009072bc698b442d96e8c2988ff9e05a received from {username='slave'} at 127.16.104.130:48963
I20250627 01:57:40.334589 17007 tablet_copy_service.cc:434] P 51d9045707a34a03b7fc6f5141a72cdf: ending tablet copy session b2454b3c7ee949e387890880e3129cc8-009072bc698b442d96e8c2988ff9e05a on tablet 009072bc698b442d96e8c2988ff9e05a with peer b2454b3c7ee949e387890880e3129cc8
W20250627 01:57:40.339021 17455 ts_tablet_manager.cc:726] T 009072bc698b442d96e8c2988ff9e05a P b2454b3c7ee949e387890880e3129cc8: Tablet Copy: Invalid argument: Leader has replica of tablet 009072bc698b442d96e8c2988ff9e05a with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
I20250627 01:57:40.343603 17455 ts_tablet_manager.cc:927] T 10b468f09d0749e8b6c723948379cb9b P b2454b3c7ee949e387890880e3129cc8: Initiating tablet copy from peer 51d9045707a34a03b7fc6f5141a72cdf (127.16.104.129:46597)
I20250627 01:57:40.345079 17455 tablet_copy_client.cc:323] T 10b468f09d0749e8b6c723948379cb9b P b2454b3c7ee949e387890880e3129cc8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.104.129:46597
I20250627 01:57:40.346215 17007 tablet_copy_service.cc:140] P 51d9045707a34a03b7fc6f5141a72cdf: Received BeginTabletCopySession request for tablet 10b468f09d0749e8b6c723948379cb9b from peer b2454b3c7ee949e387890880e3129cc8 ({username='slave'} at 127.16.104.130:48963)
I20250627 01:57:40.346575 17007 tablet_copy_service.cc:161] P 51d9045707a34a03b7fc6f5141a72cdf: Beginning new tablet copy session on tablet 10b468f09d0749e8b6c723948379cb9b from peer b2454b3c7ee949e387890880e3129cc8 at {username='slave'} at 127.16.104.130:48963: session id = b2454b3c7ee949e387890880e3129cc8-10b468f09d0749e8b6c723948379cb9b
I20250627 01:57:40.350944 17007 tablet_copy_source_session.cc:215] T 10b468f09d0749e8b6c723948379cb9b P 51d9045707a34a03b7fc6f5141a72cdf: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 01:57:40.352854 17455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10b468f09d0749e8b6c723948379cb9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 01:57:40.360503 17455 tablet_copy_client.cc:806] T 10b468f09d0749e8b6c723948379cb9b P b2454b3c7ee949e387890880e3129cc8: tablet copy: Starting download of 0 data blocks...
I20250627 01:57:40.360875 17455 tablet_copy_client.cc:670] T 10b468f09d0749e8b6c723948379cb9b P b2454b3c7ee949e387890880e3129cc8: tablet copy: Starting download of 1 WAL segments...
I20250627 01:57:40.371438 17455 tablet_copy_client.cc:538] T 10b468f09d0749e8b6c723948379cb9b P b2454b3c7ee949e387890880e3129cc8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 01:57:40.376582 17455 tablet_bootstrap.cc:492] T 10b468f09d0749e8b6c723948379cb9b P b2454b3c7ee949e387890880e3129cc8: Bootstrap starting.
I20250627 01:57:41.426121 17455 tablet_bootstrap.cc:492] T 10b468f09d0749e8b6c723948379cb9b P b2454b3c7ee949e387890880e3129cc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=212 overwritten=0 applied=212 ignored=0} inserts{seen=2634 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 01:57:41.426908 17455 tablet_bootstrap.cc:492] T 10b468f09d0749e8b6c723948379cb9b P b2454b3c7ee949e387890880e3129cc8: Bootstrap complete.
I20250627 01:57:41.427420 17455 ts_tablet_manager.cc:1397] T 10b468f09d0749e8b6c723948379cb9b P b2454b3c7ee949e387890880e3129cc8: Time spent bootstrapping tablet: real 1.051s user 1.023s sys 0.028s
I20250627 01:57:41.429289 17455 raft_consensus.cc:357] T 10b468f09d0749e8b6c723948379cb9b P b2454b3c7ee949e387890880e3129cc8 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:41.429762 17455 raft_consensus.cc:738] T 10b468f09d0749e8b6c723948379cb9b P b2454b3c7ee949e387890880e3129cc8 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: b2454b3c7ee949e387890880e3129cc8, State: Initialized, Role: NON_PARTICIPANT
I20250627 01:57:41.430176 17455 consensus_queue.cc:260] T 10b468f09d0749e8b6c723948379cb9b P b2454b3c7ee949e387890880e3129cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 212, Last appended: 1.212, Last appended by leader: 212, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:41.432785 17455 ts_tablet_manager.cc:1428] T 10b468f09d0749e8b6c723948379cb9b P b2454b3c7ee949e387890880e3129cc8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250627 01:57:41.434085 17007 tablet_copy_service.cc:342] P 51d9045707a34a03b7fc6f5141a72cdf: Request end of tablet copy session b2454b3c7ee949e387890880e3129cc8-10b468f09d0749e8b6c723948379cb9b received from {username='slave'} at 127.16.104.130:48963
I20250627 01:57:41.434468 17007 tablet_copy_service.cc:434] P 51d9045707a34a03b7fc6f5141a72cdf: ending tablet copy session b2454b3c7ee949e387890880e3129cc8-10b468f09d0749e8b6c723948379cb9b on tablet 10b468f09d0749e8b6c723948379cb9b with peer b2454b3c7ee949e387890880e3129cc8
I20250627 01:57:41.437088 17455 ts_tablet_manager.cc:927] T 20c59250b06040f3968dcd6c31060df3 P b2454b3c7ee949e387890880e3129cc8: Initiating tablet copy from peer 51d9045707a34a03b7fc6f5141a72cdf (127.16.104.129:46597)
I20250627 01:57:41.438725 17455 tablet_copy_client.cc:323] T 20c59250b06040f3968dcd6c31060df3 P b2454b3c7ee949e387890880e3129cc8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.104.129:46597
I20250627 01:57:41.439853 17007 tablet_copy_service.cc:140] P 51d9045707a34a03b7fc6f5141a72cdf: Received BeginTabletCopySession request for tablet 20c59250b06040f3968dcd6c31060df3 from peer b2454b3c7ee949e387890880e3129cc8 ({username='slave'} at 127.16.104.130:48963)
I20250627 01:57:41.440279 17007 tablet_copy_service.cc:161] P 51d9045707a34a03b7fc6f5141a72cdf: Beginning new tablet copy session on tablet 20c59250b06040f3968dcd6c31060df3 from peer b2454b3c7ee949e387890880e3129cc8 at {username='slave'} at 127.16.104.130:48963: session id = b2454b3c7ee949e387890880e3129cc8-20c59250b06040f3968dcd6c31060df3
I20250627 01:57:41.444603 17007 tablet_copy_source_session.cc:215] T 20c59250b06040f3968dcd6c31060df3 P 51d9045707a34a03b7fc6f5141a72cdf: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 01:57:41.446625 17455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20c59250b06040f3968dcd6c31060df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 01:57:41.454501 17455 tablet_copy_client.cc:806] T 20c59250b06040f3968dcd6c31060df3 P b2454b3c7ee949e387890880e3129cc8: tablet copy: Starting download of 0 data blocks...
I20250627 01:57:41.454939 17455 tablet_copy_client.cc:670] T 20c59250b06040f3968dcd6c31060df3 P b2454b3c7ee949e387890880e3129cc8: tablet copy: Starting download of 1 WAL segments...
I20250627 01:57:41.465523 17455 tablet_copy_client.cc:538] T 20c59250b06040f3968dcd6c31060df3 P b2454b3c7ee949e387890880e3129cc8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 01:57:41.471479 17455 tablet_bootstrap.cc:492] T 20c59250b06040f3968dcd6c31060df3 P b2454b3c7ee949e387890880e3129cc8: Bootstrap starting.
I20250627 01:57:42.516778 17455 tablet_bootstrap.cc:492] T 20c59250b06040f3968dcd6c31060df3 P b2454b3c7ee949e387890880e3129cc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=212 overwritten=0 applied=212 ignored=0} inserts{seen=2584 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 01:57:42.517468 17455 tablet_bootstrap.cc:492] T 20c59250b06040f3968dcd6c31060df3 P b2454b3c7ee949e387890880e3129cc8: Bootstrap complete.
I20250627 01:57:42.517920 17455 ts_tablet_manager.cc:1397] T 20c59250b06040f3968dcd6c31060df3 P b2454b3c7ee949e387890880e3129cc8: Time spent bootstrapping tablet: real 1.047s user 1.030s sys 0.016s
I20250627 01:57:42.519428 17455 raft_consensus.cc:357] T 20c59250b06040f3968dcd6c31060df3 P b2454b3c7ee949e387890880e3129cc8 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:42.519774 17455 raft_consensus.cc:738] T 20c59250b06040f3968dcd6c31060df3 P b2454b3c7ee949e387890880e3129cc8 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: b2454b3c7ee949e387890880e3129cc8, State: Initialized, Role: NON_PARTICIPANT
I20250627 01:57:42.520155 17455 consensus_queue.cc:260] T 20c59250b06040f3968dcd6c31060df3 P b2454b3c7ee949e387890880e3129cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 212, Last appended: 1.212, Last appended by leader: 212, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:42.522676 17455 ts_tablet_manager.cc:1428] T 20c59250b06040f3968dcd6c31060df3 P b2454b3c7ee949e387890880e3129cc8: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20250627 01:57:42.524291 17007 tablet_copy_service.cc:342] P 51d9045707a34a03b7fc6f5141a72cdf: Request end of tablet copy session b2454b3c7ee949e387890880e3129cc8-20c59250b06040f3968dcd6c31060df3 received from {username='slave'} at 127.16.104.130:48963
I20250627 01:57:42.524658 17007 tablet_copy_service.cc:434] P 51d9045707a34a03b7fc6f5141a72cdf: ending tablet copy session b2454b3c7ee949e387890880e3129cc8-20c59250b06040f3968dcd6c31060df3 on tablet 20c59250b06040f3968dcd6c31060df3 with peer b2454b3c7ee949e387890880e3129cc8
W20250627 01:57:42.528137 17455 ts_tablet_manager.cc:726] T 10b468f09d0749e8b6c723948379cb9b P b2454b3c7ee949e387890880e3129cc8: Tablet Copy: Invalid argument: Leader has replica of tablet 10b468f09d0749e8b6c723948379cb9b with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
W20250627 01:57:42.533496 17455 ts_tablet_manager.cc:726] T 20c59250b06040f3968dcd6c31060df3 P b2454b3c7ee949e387890880e3129cc8: Tablet Copy: Invalid argument: Leader has replica of tablet 20c59250b06040f3968dcd6c31060df3 with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
I20250627 01:57:42.536703 17455 ts_tablet_manager.cc:927] T 5e77f449263c4c1e838ed55fb80982b7 P b2454b3c7ee949e387890880e3129cc8: Initiating tablet copy from peer 51d9045707a34a03b7fc6f5141a72cdf (127.16.104.129:46597)
I20250627 01:57:42.537688 17455 tablet_copy_client.cc:323] T 5e77f449263c4c1e838ed55fb80982b7 P b2454b3c7ee949e387890880e3129cc8: tablet copy: Beginning tablet copy session from remote peer at address 127.16.104.129:46597
I20250627 01:57:42.538852 17007 tablet_copy_service.cc:140] P 51d9045707a34a03b7fc6f5141a72cdf: Received BeginTabletCopySession request for tablet 5e77f449263c4c1e838ed55fb80982b7 from peer b2454b3c7ee949e387890880e3129cc8 ({username='slave'} at 127.16.104.130:48963)
I20250627 01:57:42.539242 17007 tablet_copy_service.cc:161] P 51d9045707a34a03b7fc6f5141a72cdf: Beginning new tablet copy session on tablet 5e77f449263c4c1e838ed55fb80982b7 from peer b2454b3c7ee949e387890880e3129cc8 at {username='slave'} at 127.16.104.130:48963: session id = b2454b3c7ee949e387890880e3129cc8-5e77f449263c4c1e838ed55fb80982b7
I20250627 01:57:42.543637 17007 tablet_copy_source_session.cc:215] T 5e77f449263c4c1e838ed55fb80982b7 P 51d9045707a34a03b7fc6f5141a72cdf: Tablet Copy: opened 0 blocks and 1 log segments
I20250627 01:57:42.545698 17455 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e77f449263c4c1e838ed55fb80982b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 01:57:42.553391 17455 tablet_copy_client.cc:806] T 5e77f449263c4c1e838ed55fb80982b7 P b2454b3c7ee949e387890880e3129cc8: tablet copy: Starting download of 0 data blocks...
I20250627 01:57:42.553812 17455 tablet_copy_client.cc:670] T 5e77f449263c4c1e838ed55fb80982b7 P b2454b3c7ee949e387890880e3129cc8: tablet copy: Starting download of 1 WAL segments...
I20250627 01:57:42.564688 17455 tablet_copy_client.cc:538] T 5e77f449263c4c1e838ed55fb80982b7 P b2454b3c7ee949e387890880e3129cc8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250627 01:57:42.569655 17455 tablet_bootstrap.cc:492] T 5e77f449263c4c1e838ed55fb80982b7 P b2454b3c7ee949e387890880e3129cc8: Bootstrap starting.
I20250627 01:57:43.574544 17455 tablet_bootstrap.cc:492] T 5e77f449263c4c1e838ed55fb80982b7 P b2454b3c7ee949e387890880e3129cc8: Bootstrap replayed 1/1 log segments. Stats: ops{read=212 overwritten=0 applied=212 ignored=0} inserts{seen=2656 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 01:57:43.575276 17455 tablet_bootstrap.cc:492] T 5e77f449263c4c1e838ed55fb80982b7 P b2454b3c7ee949e387890880e3129cc8: Bootstrap complete.
I20250627 01:57:43.575829 17455 ts_tablet_manager.cc:1397] T 5e77f449263c4c1e838ed55fb80982b7 P b2454b3c7ee949e387890880e3129cc8: Time spent bootstrapping tablet: real 1.006s user 0.995s sys 0.008s
I20250627 01:57:43.577452 17455 raft_consensus.cc:357] T 5e77f449263c4c1e838ed55fb80982b7 P b2454b3c7ee949e387890880e3129cc8 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:43.577924 17455 raft_consensus.cc:738] T 5e77f449263c4c1e838ed55fb80982b7 P b2454b3c7ee949e387890880e3129cc8 [term 1 NON_PARTICIPANT]: Becoming Follower/Learner. State: Replica: b2454b3c7ee949e387890880e3129cc8, State: Initialized, Role: NON_PARTICIPANT
I20250627 01:57:43.578384 17455 consensus_queue.cc:260] T 5e77f449263c4c1e838ed55fb80982b7 P b2454b3c7ee949e387890880e3129cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 212, Last appended: 1.212, Last appended by leader: 212, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d9045707a34a03b7fc6f5141a72cdf" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 46597 } }
I20250627 01:57:43.580431 17455 ts_tablet_manager.cc:1428] T 5e77f449263c4c1e838ed55fb80982b7 P b2454b3c7ee949e387890880e3129cc8: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20250627 01:57:43.581817 17007 tablet_copy_service.cc:342] P 51d9045707a34a03b7fc6f5141a72cdf: Request end of tablet copy session b2454b3c7ee949e387890880e3129cc8-5e77f449263c4c1e838ed55fb80982b7 received from {username='slave'} at 127.16.104.130:48963
I20250627 01:57:43.582165 17007 tablet_copy_service.cc:434] P 51d9045707a34a03b7fc6f5141a72cdf: ending tablet copy session b2454b3c7ee949e387890880e3129cc8-5e77f449263c4c1e838ed55fb80982b7 on tablet 5e77f449263c4c1e838ed55fb80982b7 with peer b2454b3c7ee949e387890880e3129cc8
W20250627 01:57:43.584554 17455 ts_tablet_manager.cc:726] T 5e77f449263c4c1e838ed55fb80982b7 P b2454b3c7ee949e387890880e3129cc8: Tablet Copy: Invalid argument: Leader has replica of tablet 5e77f449263c4c1e838ed55fb80982b7 with term 0, which is lower than last-logged term 1 on local replica. Rejecting tablet copy request
I20250627 01:57:43.588384 16802 tablet_copy-itest.cc:1252] Number of Service unavailable responses: 1294
I20250627 01:57:43.588747 16802 tablet_copy-itest.cc:1253] Number of in progress responses: 961
I20250627 01:57:43.591105 16802 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu with pid 16903
I20250627 01:57:43.642999 16802 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu with pid 17320
I20250627 01:57:43.672387 16802 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu with pid 17177
2025-06-27T01:57:43Z chronyd exiting
[ OK ] TabletCopyITest.TestTabletCopyThrottling (21777 ms)
[ RUN ] TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate
2025-06-27T01:57:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T01:57:43Z Disabled control of system clock
I20250627 01:57:43.764950 16802 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
/tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.104.190:38245
--webserver_interface=127.16.104.190
--webserver_port=0
--builtin_ntp_servers=127.16.104.148:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.104.190:38245 with env {}
W20250627 01:57:44.064394 17474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 01:57:44.065052 17474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 01:57:44.065594 17474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 01:57:44.097158 17474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 01:57:44.097505 17474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 01:57:44.097760 17474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 01:57:44.098004 17474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 01:57:44.133405 17474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.104.148:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.104.190:38245
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.104.190:38245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.104.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 27 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6789
TSAN enabled
I20250627 01:57:44.134721 17474 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 01:57:44.136395 17474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 01:57:44.151309 17483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 01:57:44.151386 17480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 01:57:44.151669 17481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:44.152984 17474 server_base.cc:1048] running on GCE node
I20250627 01:57:45.335248 17474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 01:57:45.338408 17474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 01:57:45.339856 17474 hybrid_clock.cc:648] HybridClock initialized: now 1750989465339814 us; error 60 us; skew 500 ppm
I20250627 01:57:45.340713 17474 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 01:57:45.351195 17474 webserver.cc:469] Webserver started at http://127.16.104.190:39727/ using document root <none> and password file <none>
I20250627 01:57:45.352212 17474 fs_manager.cc:362] Metadata directory not provided
I20250627 01:57:45.352519 17474 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 01:57:45.353140 17474 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 01:57:45.357807 17474 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/data/instance:
uuid: "25a6f89b637644dfb5ab21ece25925e0"
format_stamp: "Formatted at 2025-06-27 01:57:45 on dist-test-slave-dvrl"
I20250627 01:57:45.359131 17474 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/wal/instance:
uuid: "25a6f89b637644dfb5ab21ece25925e0"
format_stamp: "Formatted at 2025-06-27 01:57:45 on dist-test-slave-dvrl"
I20250627 01:57:45.366796 17474 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250627 01:57:45.372705 17490 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:45.373764 17474 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250627 01:57:45.374104 17474 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/data,/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/wal
uuid: "25a6f89b637644dfb5ab21ece25925e0"
format_stamp: "Formatted at 2025-06-27 01:57:45 on dist-test-slave-dvrl"
I20250627 01:57:45.374457 17474 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 01:57:45.424312 17474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 01:57:45.425841 17474 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 01:57:45.426280 17474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 01:57:45.496421 17474 rpc_server.cc:307] RPC server started. Bound to: 127.16.104.190:38245
I20250627 01:57:45.496502 17541 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.104.190:38245 every 8 connection(s)
I20250627 01:57:45.499178 17474 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/data/info.pb
I20250627 01:57:45.504356 17542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 01:57:45.507649 16802 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu as pid 17474
I20250627 01:57:45.508046 16802 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/master-0/wal/instance
I20250627 01:57:45.526484 17542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0: Bootstrap starting.
I20250627 01:57:45.532805 17542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0: Neither blocks nor log segments found. Creating new log.
I20250627 01:57:45.534569 17542 log.cc:826] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0: Log is configured to *not* fsync() on all Append() calls
I20250627 01:57:45.539197 17542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0: No bootstrap required, opened a new log
I20250627 01:57:45.556772 17542 raft_consensus.cc:357] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a6f89b637644dfb5ab21ece25925e0" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 38245 } }
I20250627 01:57:45.557500 17542 raft_consensus.cc:383] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 01:57:45.557752 17542 raft_consensus.cc:738] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25a6f89b637644dfb5ab21ece25925e0, State: Initialized, Role: FOLLOWER
I20250627 01:57:45.558590 17542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a6f89b637644dfb5ab21ece25925e0" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 38245 } }
I20250627 01:57:45.559231 17542 raft_consensus.cc:397] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 01:57:45.559509 17542 raft_consensus.cc:491] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 01:57:45.559801 17542 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [term 0 FOLLOWER]: Advancing to term 1
I20250627 01:57:45.563903 17542 raft_consensus.cc:513] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a6f89b637644dfb5ab21ece25925e0" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 38245 } }
I20250627 01:57:45.564656 17542 leader_election.cc:304] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25a6f89b637644dfb5ab21ece25925e0; no voters:
I20250627 01:57:45.566418 17542 leader_election.cc:290] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 01:57:45.567276 17547 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 01:57:45.569662 17547 raft_consensus.cc:695] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [term 1 LEADER]: Becoming Leader. State: Replica: 25a6f89b637644dfb5ab21ece25925e0, State: Running, Role: LEADER
I20250627 01:57:45.570427 17547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a6f89b637644dfb5ab21ece25925e0" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 38245 } }
I20250627 01:57:45.571399 17542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [sys.catalog]: configured and running, proceeding with master startup.
I20250627 01:57:45.580693 17549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25a6f89b637644dfb5ab21ece25925e0. Latest consensus state: current_term: 1 leader_uuid: "25a6f89b637644dfb5ab21ece25925e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a6f89b637644dfb5ab21ece25925e0" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 38245 } } }
I20250627 01:57:45.581460 17548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25a6f89b637644dfb5ab21ece25925e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25a6f89b637644dfb5ab21ece25925e0" member_type: VOTER last_known_addr { host: "127.16.104.190" port: 38245 } } }
I20250627 01:57:45.581861 17549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [sys.catalog]: This master's current role is: LEADER
I20250627 01:57:45.582330 17548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0 [sys.catalog]: This master's current role is: LEADER
I20250627 01:57:45.593567 17555 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 01:57:45.608019 17555 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 01:57:45.624859 17555 catalog_manager.cc:1349] Generated new cluster ID: 7946dfa052b74065a277aef5080736af
I20250627 01:57:45.625229 17555 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 01:57:45.654245 17555 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 01:57:45.655704 17555 catalog_manager.cc:1506] Loading token signing keys...
I20250627 01:57:45.669917 17555 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 25a6f89b637644dfb5ab21ece25925e0: Generated new TSK 0
I20250627 01:57:45.670785 17555 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 01:57:45.680387 16802 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
/tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.104.129:0
--local_ip_for_outbound_sockets=127.16.104.129
--webserver_interface=127.16.104.129
--webserver_port=0
--tserver_master_addrs=127.16.104.190:38245
--builtin_ntp_servers=127.16.104.148:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
W20250627 01:57:46.032620 17566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 01:57:46.033120 17566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 01:57:46.033535 17566 flags.cc:425] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20250627 01:57:46.033730 17566 flags.cc:425] Enabled unsafe flag: --enable_flush_memrowset=false
W20250627 01:57:46.034049 17566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 01:57:46.064917 17566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 01:57:46.065919 17566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.104.129
I20250627 01:57:46.101225 17566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.104.148:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.104.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.104.129
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.16.104.190:38245
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.104.129
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 27 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6789
TSAN enabled
I20250627 01:57:46.102579 17566 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 01:57:46.104183 17566 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 01:57:46.121970 17573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 01:57:46.124624 17572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 01:57:46.125415 17575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:46.123940 17566 server_base.cc:1048] running on GCE node
I20250627 01:57:47.302912 17566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 01:57:47.305696 17566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 01:57:47.307150 17566 hybrid_clock.cc:648] HybridClock initialized: now 1750989467307085 us; error 80 us; skew 500 ppm
I20250627 01:57:47.307971 17566 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 01:57:47.315526 17566 webserver.cc:469] Webserver started at http://127.16.104.129:40155/ using document root <none> and password file <none>
I20250627 01:57:47.316500 17566 fs_manager.cc:362] Metadata directory not provided
I20250627 01:57:47.316725 17566 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 01:57:47.317173 17566 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 01:57:47.321683 17566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/data/instance:
uuid: "f8b2efac2d2d473182aee7941c2e3fe8"
format_stamp: "Formatted at 2025-06-27 01:57:47 on dist-test-slave-dvrl"
I20250627 01:57:47.322815 17566 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/wal/instance:
uuid: "f8b2efac2d2d473182aee7941c2e3fe8"
format_stamp: "Formatted at 2025-06-27 01:57:47 on dist-test-slave-dvrl"
I20250627 01:57:47.330402 17566 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250627 01:57:47.337225 17582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:47.338359 17566 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250627 01:57:47.338685 17566 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/wal
uuid: "f8b2efac2d2d473182aee7941c2e3fe8"
format_stamp: "Formatted at 2025-06-27 01:57:47 on dist-test-slave-dvrl"
I20250627 01:57:47.338999 17566 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 01:57:47.389710 17566 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 01:57:47.391413 17566 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 01:57:47.392045 17566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 01:57:47.394731 17566 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 01:57:47.398759 17566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 01:57:47.398959 17566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:47.399163 17566 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 01:57:47.399293 17566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:47.538605 17566 rpc_server.cc:307] RPC server started. Bound to: 127.16.104.129:34665
I20250627 01:57:47.538714 17694 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.104.129:34665 every 8 connection(s)
I20250627 01:57:47.541388 17566 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/data/info.pb
I20250627 01:57:47.551525 16802 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu as pid 17566
I20250627 01:57:47.552033 16802 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-0/wal/instance
I20250627 01:57:47.558898 16802 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
/tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.104.130:0
--local_ip_for_outbound_sockets=127.16.104.130
--webserver_interface=127.16.104.130
--webserver_port=0
--tserver_master_addrs=127.16.104.190:38245
--builtin_ntp_servers=127.16.104.148:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
I20250627 01:57:47.564754 17695 heartbeater.cc:344] Connected to a master server at 127.16.104.190:38245
I20250627 01:57:47.565171 17695 heartbeater.cc:461] Registering TS with master...
I20250627 01:57:47.566416 17695 heartbeater.cc:507] Master 127.16.104.190:38245 requested a full tablet report, sending...
I20250627 01:57:47.569499 17507 ts_manager.cc:194] Registered new tserver with Master: f8b2efac2d2d473182aee7941c2e3fe8 (127.16.104.129:34665)
I20250627 01:57:47.571362 17507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.104.129:39445
W20250627 01:57:47.856701 17699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 01:57:47.857262 17699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 01:57:47.857692 17699 flags.cc:425] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20250627 01:57:47.857873 17699 flags.cc:425] Enabled unsafe flag: --enable_flush_memrowset=false
W20250627 01:57:47.858217 17699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 01:57:47.890040 17699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 01:57:47.891036 17699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.104.130
I20250627 01:57:47.926390 17699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.104.148:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.104.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.104.130
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.16.104.190:38245
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.104.130
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 27 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6789
TSAN enabled
I20250627 01:57:47.927693 17699 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 01:57:47.929363 17699 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 01:57:47.947263 17705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:48.575456 17695 heartbeater.cc:499] Master 127.16.104.190:38245 was elected leader, sending a full tablet report...
W20250627 01:57:47.947346 17706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:47.949609 17699 server_base.cc:1048] running on GCE node
W20250627 01:57:47.947362 17708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:49.112337 17699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 01:57:49.114620 17699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 01:57:49.116006 17699 hybrid_clock.cc:648] HybridClock initialized: now 1750989469115955 us; error 57 us; skew 500 ppm
I20250627 01:57:49.116820 17699 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 01:57:49.123112 17699 webserver.cc:469] Webserver started at http://127.16.104.130:41787/ using document root <none> and password file <none>
I20250627 01:57:49.124053 17699 fs_manager.cc:362] Metadata directory not provided
I20250627 01:57:49.124289 17699 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 01:57:49.124747 17699 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 01:57:49.129314 17699 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/data/instance:
uuid: "ad4b390fedd44692b464a4f84bddeba2"
format_stamp: "Formatted at 2025-06-27 01:57:49 on dist-test-slave-dvrl"
I20250627 01:57:49.130437 17699 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/wal/instance:
uuid: "ad4b390fedd44692b464a4f84bddeba2"
format_stamp: "Formatted at 2025-06-27 01:57:49 on dist-test-slave-dvrl"
I20250627 01:57:49.137584 17699 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250627 01:57:49.143141 17715 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:49.144230 17699 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250627 01:57:49.144584 17699 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/wal
uuid: "ad4b390fedd44692b464a4f84bddeba2"
format_stamp: "Formatted at 2025-06-27 01:57:49 on dist-test-slave-dvrl"
I20250627 01:57:49.144918 17699 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 01:57:49.204226 17699 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 01:57:49.205744 17699 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 01:57:49.206180 17699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 01:57:49.208683 17699 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 01:57:49.212942 17699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 01:57:49.213177 17699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:49.213495 17699 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 01:57:49.213662 17699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:49.359406 17699 rpc_server.cc:307] RPC server started. Bound to: 127.16.104.130:41091
I20250627 01:57:49.359442 17827 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.104.130:41091 every 8 connection(s)
I20250627 01:57:49.363214 17699 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/data/info.pb
I20250627 01:57:49.365358 16802 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu as pid 17699
I20250627 01:57:49.365878 16802 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-1/wal/instance
I20250627 01:57:49.373122 16802 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
/tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.104.131:0
--local_ip_for_outbound_sockets=127.16.104.131
--webserver_interface=127.16.104.131
--webserver_port=0
--tserver_master_addrs=127.16.104.190:38245
--builtin_ntp_servers=127.16.104.148:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_flush_memrowset=false
--enable_flush_deltamemstores=false
--tablet_copy_download_threads_nums_per_session=4
--log_segment_size_mb=1 with env {}
I20250627 01:57:49.386211 17828 heartbeater.cc:344] Connected to a master server at 127.16.104.190:38245
I20250627 01:57:49.386667 17828 heartbeater.cc:461] Registering TS with master...
I20250627 01:57:49.387741 17828 heartbeater.cc:507] Master 127.16.104.190:38245 requested a full tablet report, sending...
I20250627 01:57:49.390009 17507 ts_manager.cc:194] Registered new tserver with Master: ad4b390fedd44692b464a4f84bddeba2 (127.16.104.130:41091)
I20250627 01:57:49.391249 17507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.104.130:49241
W20250627 01:57:49.674371 17832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 01:57:49.674889 17832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 01:57:49.675328 17832 flags.cc:425] Enabled unsafe flag: --enable_flush_deltamemstores=false
W20250627 01:57:49.675549 17832 flags.cc:425] Enabled unsafe flag: --enable_flush_memrowset=false
W20250627 01:57:49.675879 17832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 01:57:49.707362 17832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 01:57:49.708344 17832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.104.131
I20250627 01:57:49.743114 17832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.104.148:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.104.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.104.131
--webserver_port=0
--enable_flush_deltamemstores=false
--enable_flush_memrowset=false
--tserver_master_addrs=127.16.104.190:38245
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.104.131
--log_dir=/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f7c956859e2f49c4cf1caffa969c1777a7a5d81c
build type FASTDEBUG
built by None at 27 Jun 2025 01:43:20 UTC on 5fd53c4cbb9d
build id 6789
TSAN enabled
I20250627 01:57:49.744463 17832 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 01:57:49.746104 17832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 01:57:49.763078 17838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 01:57:50.394323 17828 heartbeater.cc:499] Master 127.16.104.190:38245 was elected leader, sending a full tablet report...
W20250627 01:57:49.763885 17841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 01:57:49.763377 17839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 01:57:50.950263 17840 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250627 01:57:50.950438 17832 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250627 01:57:50.955236 17832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 01:57:50.957958 17832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 01:57:50.959419 17832 hybrid_clock.cc:648] HybridClock initialized: now 1750989470959380 us; error 57 us; skew 500 ppm
I20250627 01:57:50.960207 17832 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 01:57:50.971837 17832 webserver.cc:469] Webserver started at http://127.16.104.131:44037/ using document root <none> and password file <none>
I20250627 01:57:50.972891 17832 fs_manager.cc:362] Metadata directory not provided
I20250627 01:57:50.973114 17832 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 01:57:50.973596 17832 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 01:57:50.979621 17832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/data/instance:
uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8"
format_stamp: "Formatted at 2025-06-27 01:57:50 on dist-test-slave-dvrl"
I20250627 01:57:50.980790 17832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/wal/instance:
uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8"
format_stamp: "Formatted at 2025-06-27 01:57:50 on dist-test-slave-dvrl"
I20250627 01:57:50.988096 17832 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250627 01:57:50.993928 17848 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:50.994982 17832 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250627 01:57:50.995298 17832 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/wal
uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8"
format_stamp: "Formatted at 2025-06-27 01:57:50 on dist-test-slave-dvrl"
I20250627 01:57:50.995667 17832 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 01:57:51.042271 17832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 01:57:51.043798 17832 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 01:57:51.044234 17832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 01:57:51.046715 17832 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 01:57:51.050737 17832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 01:57:51.050953 17832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:51.051198 17832 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 01:57:51.051367 17832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 01:57:51.185925 17832 rpc_server.cc:307] RPC server started. Bound to: 127.16.104.131:46015
I20250627 01:57:51.186023 17960 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.16.104.131:46015 every 8 connection(s)
I20250627 01:57:51.188395 17832 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/data/info.pb
I20250627 01:57:51.197157 16802 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu as pid 17832
I20250627 01:57:51.197592 16802 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0/minicluster-data/ts-2/wal/instance
I20250627 01:57:51.209077 17961 heartbeater.cc:344] Connected to a master server at 127.16.104.190:38245
I20250627 01:57:51.209596 17961 heartbeater.cc:461] Registering TS with master...
I20250627 01:57:51.210597 17961 heartbeater.cc:507] Master 127.16.104.190:38245 requested a full tablet report, sending...
I20250627 01:57:51.212654 17507 ts_manager.cc:194] Registered new tserver with Master: c1fb7a5c939b43d1a9af547a9b6f18b8 (127.16.104.131:46015)
I20250627 01:57:51.213992 17507 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.104.131:36045
I20250627 01:57:51.217453 16802 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250627 01:57:51.246135 16802 test_util.cc:276] Using random seed: 24143649
I20250627 01:57:51.286556 17506 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46922:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250627 01:57:51.289043 17506 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250627 01:57:51.339754 17763 tablet_service.cc:1468] Processing CreateTablet for tablet f7da78342ae749e3b105660117c68e80 (DEFAULT_TABLE table=test-workload [id=05bfb18d3a3f4737b11915783e0e097e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 01:57:51.341861 17763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7da78342ae749e3b105660117c68e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 01:57:51.350477 17630 tablet_service.cc:1468] Processing CreateTablet for tablet f7da78342ae749e3b105660117c68e80 (DEFAULT_TABLE table=test-workload [id=05bfb18d3a3f4737b11915783e0e097e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 01:57:51.350571 17896 tablet_service.cc:1468] Processing CreateTablet for tablet f7da78342ae749e3b105660117c68e80 (DEFAULT_TABLE table=test-workload [id=05bfb18d3a3f4737b11915783e0e097e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 01:57:51.352480 17896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7da78342ae749e3b105660117c68e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 01:57:51.352542 17630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7da78342ae749e3b105660117c68e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 01:57:51.379086 17985 tablet_bootstrap.cc:492] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2: Bootstrap starting.
I20250627 01:57:51.385175 17987 tablet_bootstrap.cc:492] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8: Bootstrap starting.
I20250627 01:57:51.388760 17986 tablet_bootstrap.cc:492] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8: Bootstrap starting.
I20250627 01:57:51.390777 17985 tablet_bootstrap.cc:654] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2: Neither blocks nor log segments found. Creating new log.
I20250627 01:57:51.393922 17985 log.cc:826] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2: Log is configured to *not* fsync() on all Append() calls
I20250627 01:57:51.397119 17987 tablet_bootstrap.cc:654] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8: Neither blocks nor log segments found. Creating new log.
I20250627 01:57:51.398721 17986 tablet_bootstrap.cc:654] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8: Neither blocks nor log segments found. Creating new log.
I20250627 01:57:51.401077 17986 log.cc:826] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8: Log is configured to *not* fsync() on all Append() calls
I20250627 01:57:51.403084 17987 log.cc:826] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8: Log is configured to *not* fsync() on all Append() calls
I20250627 01:57:51.407603 17985 tablet_bootstrap.cc:492] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2: No bootstrap required, opened a new log
I20250627 01:57:51.408142 17985 ts_tablet_manager.cc:1397] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.007s
I20250627 01:57:51.409056 17986 tablet_bootstrap.cc:492] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8: No bootstrap required, opened a new log
I20250627 01:57:51.409053 17987 tablet_bootstrap.cc:492] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8: No bootstrap required, opened a new log
I20250627 01:57:51.409507 17986 ts_tablet_manager.cc:1397] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20250627 01:57:51.409672 17987 ts_tablet_manager.cc:1397] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.006s
I20250627 01:57:51.427608 17986 raft_consensus.cc:357] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 } } peers { permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 } } peers { permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 } }
I20250627 01:57:51.428367 17986 raft_consensus.cc:383] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 01:57:51.428646 17986 raft_consensus.cc:738] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1fb7a5c939b43d1a9af547a9b6f18b8, State: Initialized, Role: FOLLOWER
I20250627 01:57:51.429603 17986 consensus_queue.cc:260] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 } } peers { permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 } } peers { permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 } }
I20250627 01:57:51.434144 17961 heartbeater.cc:499] Master 127.16.104.190:38245 was elected leader, sending a full tablet report...
I20250627 01:57:51.435492 17986 ts_tablet_manager.cc:1428] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8: Time spent starting tablet: real 0.026s user 0.021s sys 0.003s
I20250627 01:57:51.436273 17985 raft_consensus.cc:357] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 } } peers { permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 } } peers { permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 } }
I20250627 01:57:51.437170 17985 raft_consensus.cc:383] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 01:57:51.437520 17985 raft_consensus.cc:738] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4b390fedd44692b464a4f84bddeba2, State: Initialized, Role: FOLLOWER
I20250627 01:57:51.437448 17987 raft_consensus.cc:357] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 } } peers { permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 } } peers { permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 } }
I20250627 01:57:51.438365 17987 raft_consensus.cc:383] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 01:57:51.438688 17987 raft_consensus.cc:738] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b2efac2d2d473182aee7941c2e3fe8, State: Initialized, Role: FOLLOWER
I20250627 01:57:51.438388 17985 consensus_queue.cc:260] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 } } peers { permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 } } peers { permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 } }
I20250627 01:57:51.439579 17987 consensus_queue.cc:260] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 } } peers { permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 } } peers { permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 } }
W20250627 01:57:51.443077 17962 tablet.cc:2378] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 01:57:51.449987 17987 ts_tablet_manager.cc:1428] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8: Time spent starting tablet: real 0.040s user 0.029s sys 0.007s
I20250627 01:57:51.453323 17985 ts_tablet_manager.cc:1428] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2: Time spent starting tablet: real 0.045s user 0.035s sys 0.012s
I20250627 01:57:51.454727 17992 raft_consensus.cc:491] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 01:57:51.455322 17992 raft_consensus.cc:513] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 } } peers { permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 } } peers { permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 } }
I20250627 01:57:51.469014 17992 leader_election.cc:290] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8b2efac2d2d473182aee7941c2e3fe8 (127.16.104.129:34665), c1fb7a5c939b43d1a9af547a9b6f18b8 (127.16.104.131:46015)
I20250627 01:57:51.474061 17650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7da78342ae749e3b105660117c68e80" candidate_uuid: "ad4b390fedd44692b464a4f84bddeba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" is_pre_election: true
I20250627 01:57:51.474776 17650 raft_consensus.cc:2466] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad4b390fedd44692b464a4f84bddeba2 in term 0.
I20250627 01:57:51.475880 17718 leader_election.cc:304] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad4b390fedd44692b464a4f84bddeba2, f8b2efac2d2d473182aee7941c2e3fe8; no voters:
I20250627 01:57:51.476714 17993 raft_consensus.cc:2802] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 01:57:51.477133 17993 raft_consensus.cc:491] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 01:57:51.477484 17993 raft_consensus.cc:3058] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 0 FOLLOWER]: Advancing to term 1
I20250627 01:57:51.480374 17916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7da78342ae749e3b105660117c68e80" candidate_uuid: "ad4b390fedd44692b464a4f84bddeba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" is_pre_election: true
I20250627 01:57:51.481045 17916 raft_consensus.cc:2466] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad4b390fedd44692b464a4f84bddeba2 in term 0.
I20250627 01:57:51.483024 17993 raft_consensus.cc:513] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 } } peers { permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 } } peers { permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 } }
I20250627 01:57:51.484994 17993 leader_election.cc:290] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [CANDIDATE]: Term 1 election: Requested vote from peers f8b2efac2d2d473182aee7941c2e3fe8 (127.16.104.129:34665), c1fb7a5c939b43d1a9af547a9b6f18b8 (127.16.104.131:46015)
I20250627 01:57:51.485313 17650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7da78342ae749e3b105660117c68e80" candidate_uuid: "ad4b390fedd44692b464a4f84bddeba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8b2efac2d2d473182aee7941c2e3fe8"
I20250627 01:57:51.485797 17916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7da78342ae749e3b105660117c68e80" candidate_uuid: "ad4b390fedd44692b464a4f84bddeba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8"
I20250627 01:57:51.485862 17650 raft_consensus.cc:3058] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [term 0 FOLLOWER]: Advancing to term 1
I20250627 01:57:51.486199 17916 raft_consensus.cc:3058] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8 [term 0 FOLLOWER]: Advancing to term 1
I20250627 01:57:51.490723 17650 raft_consensus.cc:2466] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad4b390fedd44692b464a4f84bddeba2 in term 1.
I20250627 01:57:51.490792 17916 raft_consensus.cc:2466] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad4b390fedd44692b464a4f84bddeba2 in term 1.
I20250627 01:57:51.491770 17716 leader_election.cc:304] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad4b390fedd44692b464a4f84bddeba2, c1fb7a5c939b43d1a9af547a9b6f18b8; no voters:
I20250627 01:57:51.492439 17993 raft_consensus.cc:2802] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 01:57:51.492798 17993 raft_consensus.cc:695] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 1 LEADER]: Becoming Leader. State: Replica: ad4b390fedd44692b464a4f84bddeba2, State: Running, Role: LEADER
I20250627 01:57:51.493631 17993 consensus_queue.cc:237] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 } } peers { permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 } } peers { permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 } }
I20250627 01:57:51.506237 17504 catalog_manager.cc:5582] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad4b390fedd44692b464a4f84bddeba2 (127.16.104.130). New cstate: current_term: 1 leader_uuid: "ad4b390fedd44692b464a4f84bddeba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 } health_report { overall_health: UNKNOWN } } }
I20250627 01:57:51.550295 17650 raft_consensus.cc:1273] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [term 1 FOLLOWER]: Refusing update from remote peer ad4b390fedd44692b464a4f84bddeba2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 01:57:51.550379 17916 raft_consensus.cc:1273] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8 [term 1 FOLLOWER]: Refusing update from remote peer ad4b390fedd44692b464a4f84bddeba2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250627 01:57:51.550812 17696 tablet.cc:2378] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 01:57:51.551945 17992 consensus_queue.cc:1035] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 01:57:51.552564 17993 consensus_queue.cc:1035] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 01:57:51.576782 17999 mvcc.cc:204] Tried to move back new op lower bound from 7172052875452993536 to 7172052875246645248. Current Snapshot: MvccSnapshot[applied={T|T < 7172052875452993536}]
I20250627 01:57:51.579733 18000 mvcc.cc:204] Tried to move back new op lower bound from 7172052875452993536 to 7172052875246645248. Current Snapshot: MvccSnapshot[applied={T|T < 7172052875452993536}]
I20250627 01:57:51.641009 17650 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f7da78342ae749e3b105660117c68e80"
dest_uuid: "f8b2efac2d2d473182aee7941c2e3fe8"
from {username='slave'} at 127.0.0.1:52340
I20250627 01:57:51.641605 17650 raft_consensus.cc:491] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 01:57:51.641881 17650 raft_consensus.cc:3058] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [term 1 FOLLOWER]: Advancing to term 2
I20250627 01:57:51.646121 17650 raft_consensus.cc:513] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 } } peers { permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 } } peers { permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 } }
I20250627 01:57:51.648379 17650 leader_election.cc:290] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [CANDIDATE]: Term 2 election: Requested vote from peers ad4b390fedd44692b464a4f84bddeba2 (127.16.104.130:41091), c1fb7a5c939b43d1a9af547a9b6f18b8 (127.16.104.131:46015)
I20250627 01:57:51.663934 17916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7da78342ae749e3b105660117c68e80" candidate_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8"
I20250627 01:57:51.663966 17783 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f7da78342ae749e3b105660117c68e80" candidate_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ad4b390fedd44692b464a4f84bddeba2"
I20250627 01:57:51.664525 17916 raft_consensus.cc:3058] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8 [term 1 FOLLOWER]: Advancing to term 2
I20250627 01:57:51.664759 17783 raft_consensus.cc:3053] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 01:57:51.665105 17783 raft_consensus.cc:738] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ad4b390fedd44692b464a4f84bddeba2, State: Running, Role: LEADER
I20250627 01:57:51.665905 17783 consensus_queue.cc:260] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 } } peers { permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 } } peers { permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 } }
I20250627 01:57:51.667181 17783 raft_consensus.cc:3058] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 1 FOLLOWER]: Advancing to term 2
I20250627 01:57:51.671671 17916 raft_consensus.cc:2466] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8b2efac2d2d473182aee7941c2e3fe8 in term 2.
I20250627 01:57:51.673401 17583 leader_election.cc:304] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1fb7a5c939b43d1a9af547a9b6f18b8, f8b2efac2d2d473182aee7941c2e3fe8; no voters:
I20250627 01:57:51.674422 17994 raft_consensus.cc:2802] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [term 2 FOLLOWER]: Leader election won for term 2
I20250627 01:57:51.676591 17994 raft_consensus.cc:695] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [term 2 LEADER]: Becoming Leader. State: Replica: f8b2efac2d2d473182aee7941c2e3fe8, State: Running, Role: LEADER
I20250627 01:57:51.676599 17783 raft_consensus.cc:2466] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8b2efac2d2d473182aee7941c2e3fe8 in term 2.
I20250627 01:57:51.678326 17994 consensus_queue.cc:237] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 } } peers { permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 } } peers { permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 } }
I20250627 01:57:51.688257 17506 catalog_manager.cc:5582] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 reported cstate change: term changed from 1 to 2, leader changed from ad4b390fedd44692b464a4f84bddeba2 (127.16.104.130) to f8b2efac2d2d473182aee7941c2e3fe8 (127.16.104.129). New cstate: current_term: 2 leader_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8b2efac2d2d473182aee7941c2e3fe8" member_type: VOTER last_known_addr { host: "127.16.104.129" port: 34665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 } health_report { overall_health: UNKNOWN } } }
W20250627 01:57:51.755923 17743 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34614: Illegal state: replica ad4b390fedd44692b464a4f84bddeba2 is not leader of this config: current role FOLLOWER
W20250627 01:57:51.760882 17742 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34614: Illegal state: replica ad4b390fedd44692b464a4f84bddeba2 is not leader of this config: current role FOLLOWER
W20250627 01:57:51.775069 17742 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34614: Illegal state: replica ad4b390fedd44692b464a4f84bddeba2 is not leader of this config: current role FOLLOWER
I20250627 01:57:51.789016 17916 raft_consensus.cc:1273] T f7da78342ae749e3b105660117c68e80 P c1fb7a5c939b43d1a9af547a9b6f18b8 [term 2 FOLLOWER]: Refusing update from remote peer f8b2efac2d2d473182aee7941c2e3fe8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250627 01:57:51.789935 17783 raft_consensus.cc:1273] T f7da78342ae749e3b105660117c68e80 P ad4b390fedd44692b464a4f84bddeba2 [term 2 FOLLOWER]: Refusing update from remote peer f8b2efac2d2d473182aee7941c2e3fe8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250627 01:57:51.790349 17994 consensus_queue.cc:1035] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1fb7a5c939b43d1a9af547a9b6f18b8" member_type: VOTER last_known_addr { host: "127.16.104.131" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250627 01:57:51.791587 18014 consensus_queue.cc:1035] T f7da78342ae749e3b105660117c68e80 P f8b2efac2d2d473182aee7941c2e3fe8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad4b390fedd44692b464a4f84bddeba2" member_type: VOTER last_known_addr { host: "127.16.104.130" port: 41091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250627 01:57:51.978243 18001 mvcc.cc:204] Tried to move back new op lower bound from 7172052877128105984 to 7172052876007645184. Current Snapshot: MvccSnapshot[applied={T|T < 7172052877128105984 or (T in {7172052877128105984})}]
W20250627 01:58:01.400323 17957 debug-util.cc:398] Leaking SignalData structure 0x7b08000c7620 after lost signal to thread 17833
W20250627 01:58:01.401489 17957 debug-util.cc:398] Leaking SignalData structure 0x7b08000acb80 after lost signal to thread 17960
W20250627 01:58:01.678586 17824 debug-util.cc:398] Leaking SignalData structure 0x7b08000cdd60 after lost signal to thread 17700
W20250627 01:58:01.679790 17824 debug-util.cc:398] Leaking SignalData structure 0x7b08000ca380 after lost signal to thread 17827
W20250627 01:58:07.605485 18017 meta_cache.cc:1261] Time spent looking up entry by key: real 0.064s user 0.004s sys 0.000s
W20250627 01:58:09.108574 17957 debug-util.cc:398] Leaking SignalData structure 0x7b08000c8fa0 after lost signal to thread 17833
W20250627 01:58:09.109633 17957 debug-util.cc:398] Leaking SignalData structure 0x7b08000cb400 after lost signal to thread 17960
W20250627 01:58:14.212666 17538 debug-util.cc:398] Leaking SignalData structure 0x7b080006f1a0 after lost signal to thread 17475
W20250627 01:58:14.213421 17538 debug-util.cc:398] Leaking SignalData structure 0x7b080009f3a0 after lost signal to thread 17541
W20250627 01:58:21.071203 17824 debug-util.cc:398] Leaking SignalData structure 0x7b08000c70a0 after lost signal to thread 17700
W20250627 01:58:21.072276 17824 debug-util.cc:398] Leaking SignalData structure 0x7b0800081fa0 after lost signal to thread 17827
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:2151: Failure
Failed
Bad status: Timed out: Timed out waiting for number of WAL segments on tablet f7da78342ae749e3b105660117c68e80 on TS 0 to be 6. Found 5
I20250627 01:58:51.847016 16802 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20250627 01:58:51.847455 16802 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID f8b2efac2d2d473182aee7941c2e3fe8 and pid 17566
************************ BEGIN STACKS **************************
[New LWP 17567]
[New LWP 17568]
[New LWP 17569]
[New LWP 17570]
[New LWP 17571]
[New LWP 17578]
[New LWP 17579]
[New LWP 17580]
[New LWP 17583]
[New LWP 17584]
[New LWP 17585]
[New LWP 17586]
[New LWP 17587]
[New LWP 17588]
[New LWP 17589]
[New LWP 17590]
[New LWP 17591]
[New LWP 17592]
[New LWP 17593]
[New LWP 17594]
[New LWP 17595]
[New LWP 17596]
[New LWP 17597]
[New LWP 17598]
[New LWP 17599]
[New LWP 17600]
[New LWP 17601]
[New LWP 17602]
[New LWP 17603]
[New LWP 17604]
[New LWP 17605]
[New LWP 17606]
[New LWP 17607]
[New LWP 17608]
[New LWP 17609]
[New LWP 17610]
[New LWP 17611]
[New LWP 17612]
[New LWP 17613]
[New LWP 17614]
[New LWP 17615]
[New LWP 17616]
[New LWP 17617]
[New LWP 17618]
[New LWP 17619]
[New LWP 17620]
[New LWP 17621]
[New LWP 17622]
[New LWP 17623]
[New LWP 17624]
[New LWP 17625]
[New LWP 17626]
[New LWP 17627]
[New LWP 17628]
[New LWP 17629]
[New LWP 17630]
[New LWP 17631]
[New LWP 17632]
[New LWP 17633]
[New LWP 17634]
[New LWP 17635]
[New LWP 17636]
[New LWP 17637]
[New LWP 17638]
[New LWP 17639]
[New LWP 17640]
[New LWP 17641]
[New LWP 17642]
[New LWP 17643]
[New LWP 17644]
[New LWP 17645]
[New LWP 17646]
[New LWP 17647]
[New LWP 17648]
[New LWP 17649]
[New LWP 17650]
[New LWP 17651]
[New LWP 17652]
[New LWP 17653]
[New LWP 17654]
[New LWP 17655]
[New LWP 17656]
[New LWP 17657]
[New LWP 17658]
[New LWP 17659]
[New LWP 17660]
[New LWP 17661]
[New LWP 17662]
[New LWP 17663]
[New LWP 17664]
[New LWP 17665]
[New LWP 17666]
[New LWP 17667]
[New LWP 17668]
[New LWP 17669]
[New LWP 17670]
[New LWP 17671]
[New LWP 17672]
[New LWP 17673]
[New LWP 17674]
[New LWP 17675]
[New LWP 17676]
[New LWP 17677]
[New LWP 17678]
[New LWP 17679]
[New LWP 17680]
[New LWP 17681]
[New LWP 17682]
[New LWP 17683]
[New LWP 17684]
[New LWP 17685]
[New LWP 17686]
[New LWP 17687]
[New LWP 17688]
[New LWP 17689]
[New LWP 17690]
[New LWP 17691]
[New LWP 17692]
[New LWP 17693]
[New LWP 17694]
[New LWP 17695]
[New LWP 17696]
[New LWP 18003]
[New LWP 18177]
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
0x00007f496c196d50 in ?? ()
Id Target Id Frame
* 1 LWP 17566 "kudu" 0x00007f496c196d50 in ?? ()
2 LWP 17567 "kudu" 0x00007f496755a7a0 in ?? ()
3 LWP 17568 "kudu" 0x00007f496c192fb9 in ?? ()
4 LWP 17569 "kudu" 0x00007f496c192fb9 in ?? ()
5 LWP 17570 "kudu" 0x00007f496c192fb9 in ?? ()
6 LWP 17571 "kernel-watcher-" 0x00007f496c192fb9 in ?? ()
7 LWP 17578 "ntp client-1757" 0x00007f496c1969e2 in ?? ()
8 LWP 17579 "file cache-evic" 0x00007f496c192fb9 in ?? ()
9 LWP 17580 "sq_acceptor" 0x00007f496758acb9 in ?? ()
10 LWP 17583 "rpc reactor-175" 0x00007f4967597a47 in ?? ()
11 LWP 17584 "rpc reactor-175" 0x00007f4967597a47 in ?? ()
12 LWP 17585 "rpc reactor-175" 0x00007f4967597a47 in ?? ()
13 LWP 17586 "rpc reactor-175" 0x00007f4967597a47 in ?? ()
14 LWP 17587 "MaintenanceMgr " 0x00007f496c192ad3 in ?? ()
15 LWP 17588 "txn-status-mana" 0x00007f496c192fb9 in ?? ()
16 LWP 17589 "collect_and_rem" 0x00007f496c192fb9 in ?? ()
17 LWP 17590 "tc-session-exp-" 0x00007f496c192fb9 in ?? ()
18 LWP 17591 "rpc worker-1759" 0x00007f496c192ad3 in ?? ()
19 LWP 17592 "rpc worker-1759" 0x00007f496c192ad3 in ?? ()
20 LWP 17593 "rpc worker-1759" 0x00007f496c192ad3 in ?? ()
21 LWP 17594 "rpc worker-1759" 0x00007f496c192ad3 in ?? ()
22 LWP 17595 "rpc worker-1759" 0x00007f496c192ad3 in ?? ()
23 LWP 17596 "rpc worker-1759" 0x00007f496c192ad3 in ?? ()
24 LWP 17597 "rpc worker-1759" 0x00007f496c192ad3 in ?? ()
25 LWP 17598 "rpc worker-1759" 0x00007f496c192ad3 in ?? ()
26 LWP 17599 "rpc worker-1759" 0x00007f496c192ad3 in ?? ()
27 LWP 17600 "rpc worker-1760" 0x00007f496c192ad3 in ?? ()
28 LWP 17601 "rpc worker-1760" 0x00007f496c192ad3 in ?? ()
29 LWP 17602 "rpc worker-1760" 0x00007f496c192ad3 in ?? ()
30 LWP 17603 "rpc worker-1760" 0x00007f496c192ad3 in ?? ()
31 LWP 17604 "rpc worker-1760" 0x00007f496c192ad3 in ?? ()
32 LWP 17605 "rpc worker-1760" 0x00007f496c192ad3 in ?? ()
33 LWP 17606 "rpc worker-1760" 0x00007f496c192ad3 in ?? ()
34 LWP 17607 "rpc worker-1760" 0x00007f496c192ad3 in ?? ()
35 LWP 17608 "rpc worker-1760" 0x00007f496c192ad3 in ?? ()
36 LWP 17609 "rpc worker-1760" 0x00007f496c192ad3 in ?? ()
37 LWP 17610 "rpc worker-1761" 0x00007f496c192ad3 in ?? ()
38 LWP 17611 "rpc worker-1761" 0x00007f496c192ad3 in ?? ()
39 LWP 17612 "rpc worker-1761" 0x00007f496c192ad3 in ?? ()
40 LWP 17613 "rpc worker-1761" 0x00007f496c192ad3 in ?? ()
41 LWP 17614 "rpc worker-1761" 0x00007f496c192ad3 in ?? ()
42 LWP 17615 "rpc worker-1761" 0x00007f496c192ad3 in ?? ()
43 LWP 17616 "rpc worker-1761" 0x00007f496c192ad3 in ?? ()
44 LWP 17617 "rpc worker-1761" 0x00007f496c192ad3 in ?? ()
45 LWP 17618 "rpc worker-1761" 0x00007f496c192ad3 in ?? ()
46 LWP 17619 "rpc worker-1761" 0x00007f496c192ad3 in ?? ()
47 LWP 17620 "rpc worker-1762" 0x00007f496c192ad3 in ?? ()
48 LWP 17621 "rpc worker-1762" 0x00007f496c192ad3 in ?? ()
49 LWP 17622 "rpc worker-1762" 0x00007f496c192ad3 in ?? ()
50 LWP 17623 "rpc worker-1762" 0x00007f496c192ad3 in ?? ()
51 LWP 17624 "rpc worker-1762" 0x00007f496c192ad3 in ?? ()
52 LWP 17625 "rpc worker-1762" 0x00007f496c192ad3 in ?? ()
53 LWP 17626 "rpc worker-1762" 0x00007f496c192ad3 in ?? ()
54 LWP 17627 "rpc worker-1762" 0x00007f496c192ad3 in ?? ()
55 LWP 17628 "rpc worker-1762" 0x00007f496c192ad3 in ?? ()
56 LWP 17629 "rpc worker-1762" 0x00007f496c192ad3 in ?? ()
57 LWP 17630 "rpc worker-1763" 0x00007f496c192ad3 in ?? ()
58 LWP 17631 "rpc worker-1763" 0x00007f496c192ad3 in ?? ()
59 LWP 17632 "rpc worker-1763" 0x00007f496c192ad3 in ?? ()
60 LWP 17633 "rpc worker-1763" 0x00007f496c192ad3 in ?? ()
61 LWP 17634 "rpc worker-1763" 0x00007f496c192ad3 in ?? ()
62 LWP 17635 "rpc worker-1763" 0x00007f496c192ad3 in ?? ()
63 LWP 17636 "rpc worker-1763" 0x00007f496c192ad3 in ?? ()
64 LWP 17637 "rpc worker-1763" 0x00007f496c192ad3 in ?? ()
65 LWP 17638 "rpc worker-1763" 0x00007f496c192ad3 in ?? ()
66 LWP 17639 "rpc worker-1763" 0x00007f496c192ad3 in ?? ()
67 LWP 17640 "rpc worker-1764" 0x00007f496c192ad3 in ?? ()
68 LWP 17641 "rpc worker-1764" 0x00007f496c192ad3 in ?? ()
69 LWP 17642 "rpc worker-1764" 0x00007f496c192ad3 in ?? ()
70 LWP 17643 "rpc worker-1764" 0x00007f496c192ad3 in ?? ()
71 LWP 17644 "rpc worker-1764" 0x00007f496c192ad3 in ?? ()
72 LWP 17645 "rpc worker-1764" 0x00007f496c192ad3 in ?? ()
73 LWP 17646 "rpc worker-1764" 0x00007f496c192ad3 in ?? ()
74 LWP 17647 "rpc worker-1764" 0x00007f496c192ad3 in ?? ()
75 LWP 17648 "rpc worker-1764" 0x00007f496c192ad3 in ?? ()
76 LWP 17649 "rpc worker-1764" 0x00007f496c192ad3 in ?? ()
77 LWP 17650 "rpc worker-1765" 0x00007f496c192ad3 in ?? ()
78 LWP 17651 "rpc worker-1765" 0x00007f496c192ad3 in ?? ()
79 LWP 17652 "rpc worker-1765" 0x00007f496c192ad3 in ?? ()
80 LWP 17653 "rpc worker-1765" 0x00007f496c192ad3 in ?? ()
81 LWP 17654 "rpc worker-1765" 0x00007f496c192ad3 in ?? ()
82 LWP 17655 "rpc worker-1765" 0x00007f496c192ad3 in ?? ()
83 LWP 17656 "rpc worker-1765" 0x00007f496c192ad3 in ?? ()
84 LWP 17657 "rpc worker-1765" 0x00007f496c192ad3 in ?? ()
85 LWP 17658 "rpc worker-1765" 0x00007f496c192ad3 in ?? ()
86 LWP 17659 "rpc worker-1765" 0x00007f496c192ad3 in ?? ()
87 LWP 17660 "rpc worker-1766" 0x00007f496c192ad3 in ?? ()
88 LWP 17661 "rpc worker-1766" 0x00007f496c192ad3 in ?? ()
89 LWP 17662 "rpc worker-1766" 0x00007f496c192ad3 in ?? ()
90 LWP 17663 "rpc worker-1766" 0x00007f496c192ad3 in ?? ()
91 LWP 17664 "rpc worker-1766" 0x00007f496c192ad3 in ?? ()
92 LWP 17665 "rpc worker-1766" 0x00007f496c192ad3 in ?? ()
93 LWP 17666 "rpc worker-1766" 0x00007f496c192ad3 in ?? ()
94 LWP 17667 "rpc worker-1766" 0x00007f496c192ad3 in ?? ()
95 LWP 17668 "rpc worker-1766" 0x00007f496c192ad3 in ?? ()
96 LWP 17669 "rpc worker-1766" 0x00007f496c192ad3 in ?? ()
97 LWP 17670 "rpc worker-1767" 0x00007f496c192ad3 in ?? ()
98 LWP 17671 "rpc worker-1767" 0x00007f496c192ad3 in ?? ()
99 LWP 17672 "rpc worker-1767" 0x00007f496c192ad3 in ?? ()
100 LWP 17673 "rpc worker-1767" 0x00007f496c192ad3 in ?? ()
101 LWP 17674 "rpc worker-1767" 0x00007f496c192ad3 in ?? ()
102 LWP 17675 "rpc worker-1767" 0x00007f496c192ad3 in ?? ()
103 LWP 17676 "rpc worker-1767" 0x00007f496c192ad3 in ?? ()
104 LWP 17677 "rpc worker-1767" 0x00007f496c192ad3 in ?? ()
105 LWP 17678 "rpc worker-1767" 0x00007f496c192ad3 in ?? ()
106 LWP 17679 "rpc worker-1767" 0x00007f496c192ad3 in ?? ()
107 LWP 17680 "rpc worker-1768" 0x00007f496c192ad3 in ?? ()
108 LWP 17681 "rpc worker-1768" 0x00007f496c192ad3 in ?? ()
109 LWP 17682 "rpc worker-1768" 0x00007f496c192ad3 in ?? ()
110 LWP 17683 "rpc worker-1768" 0x00007f496c192ad3 in ?? ()
111 LWP 17684 "rpc worker-1768" 0x00007f496c192ad3 in ?? ()
112 LWP 17685 "rpc worker-1768" 0x00007f496c192ad3 in ?? ()
113 LWP 17686 "rpc worker-1768" 0x00007f496c192ad3 in ?? ()
114 LWP 17687 "rpc worker-1768" 0x00007f496c192ad3 in ?? ()
115 LWP 17688 "rpc worker-1768" 0x00007f496c192ad3 in ?? ()
116 LWP 17689 "rpc worker-1768" 0x00007f496c192ad3 in ?? ()
117 LWP 17690 "rpc worker-1769" 0x00007f496c192ad3 in ?? ()
118 LWP 17691 "diag-logger-176" 0x00007f496c192fb9 in ?? ()
119 LWP 17692 "result-tracker-" 0x00007f496c192fb9 in ?? ()
120 LWP 17693 "excess-log-dele" 0x00007f496c192fb9 in ?? ()
121 LWP 17694 "acceptor-17694" 0x00007f49675990c7 in ?? ()
122 LWP 17695 "heartbeat-17695" 0x00007f496c192fb9 in ?? ()
123 LWP 17696 "maintenance_sch" 0x00007f496c192fb9 in ?? ()
124 LWP 18003 "wal-append [wor" 0x00007f496c192fb9 in ?? ()
125 LWP 18177 "raft [worker]-1" 0x00007f496c192fb9 in ?? ()
Thread 125 (LWP 18177):
#0 0x00007f496c192fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 124 (LWP 18003):
#0 0x00007f496c192fb9 in ?? ()
#1 0x00007b10000583f0 in ?? ()
#2 0x00000000000012c0 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b6400060018 in ?? ()
#5 0x00007f491d9bd440 in ?? ()
#6 0x0000000000002580 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 123 (LWP 17696):
#0 0x00007f496c192fb9 in ?? ()
#1 0x00007b0100000000 in ?? ()
#2 0x0000000000000103 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007f49203b96c0 in ?? ()
#6 0x0000000000000206 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 17695):
#0 0x00007f496c192fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 17694):
#0 0x00007f49675990c7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 17693):
#0 0x00007f496c192fb9 in ?? ()
#1 0x00007f4921bbc940 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffcc4905590 in ?? ()
#5 0x00007f4921bbc7b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 17692):
#0 0x00007f496c192fb9 in ?? ()
#1 0x0000000085352fb8 in ?? ()
#2 0x0000000000000041 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001008 in ?? ()
#5 0x00007f49223bd800 in ?? ()
#6 0x0000000000000082 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 17691):
#0 0x00007f496c192fb9 in ?? ()
#1 0x00007f496560e008 in ?? ()
#2 0x0000000000000041 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000c90 in ?? ()
#5 0x00007f4922bbe750 in ?? ()
#6 0x0000000000000082 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 17690):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 17689):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 17688):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 17687):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 17686):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 17685):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 17684):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 17683):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 17682):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 17681):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 17680):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 17679):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 17678):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 17677):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 17676):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 17675):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 17674):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 17673):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 17672):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 17671):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 17670):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 17669):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 17668):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 17667):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 17666):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 17665):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 17664):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 17663):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 17662):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 17661):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 17660):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 17659):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 17658):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 17657):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 17656):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 17655):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 17654):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 17653):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 17652):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 17651):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 17650):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000006 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b24001147c8 in ?? ()
#4 0x00007f4937dba710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4937dba730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 76 (LWP 17649):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 75 (LWP 17648):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 17647):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 17646):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 17645):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 17644):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 17643):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 17642):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 17641):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 17640):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 17639):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 17638):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 17637):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 17636):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 17635):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 17634):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 17633):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 17632):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 17631):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 17630):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000b902c in ?? ()
#4 0x00007f49421bc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f49421bc730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x007f0400000026c8 in ?? ()
#9 0x00007f496c192770 in ?? ()
#10 0x00007f49421bc730 in ?? ()
#11 0x0002008300000dfe in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 17629):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 17628):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 17627):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 17626):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 17625):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 17624):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 17623):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 17622):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 17621):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 17620):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 17619):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 17618):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 17617):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 17616):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 17615):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 17614):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 17613):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 17612):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 17611):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 17610):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000480 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240005ffe8 in ?? ()
#4 0x00007f494c5be710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f494c5be730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 36 (LWP 17609):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240005d7fc in ?? ()
#4 0x00007f494cfb6710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f494cfb6730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f496c192770 in ?? ()
#10 0x00007f494cfb6730 in ?? ()
#11 0x00007f4964597c60 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 17608):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000410 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b2400058ff8 in ?? ()
#4 0x00007f494d7b7710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f494d7b7730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 34 (LWP 17607):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000547fc in ?? ()
#4 0x00007f494dfb8710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f494dfb8730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f496c192770 in ?? ()
#10 0x00007f494dfb8730 in ?? ()
#11 0x00007f4964587c60 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 33 (LWP 17606):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240004fffc in ?? ()
#4 0x00007f494e7b9710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f494e7b9730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f496c192770 in ?? ()
#10 0x00007f494e7b9730 in ?? ()
#11 0x00007f4964affc60 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 32 (LWP 17605):
#0 0x00007f496c192ad3 in ?? ()
#1 0x00000000000000fd in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240004900c in ?? ()
#4 0x00007f494efba710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f494efba730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f496c192770 in ?? ()
#10 0x00007f494efba730 in ?? ()
#11 0x00007f491a4422a0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 31 (LWP 17604):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 17603):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 17602):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 17601):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 17600):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 17599):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 17598):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 17597):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 17596):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 17595):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 17594):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 17593):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 17592):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 17591):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 17590):
#0 0x00007f496c192fb9 in ?? ()
#1 0x0000000017a335f0 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800003a00 in ?? ()
#5 0x00007f4956b92700 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 17589):
#0 0x00007f496c192fb9 in ?? ()
#1 0x00007f49573939a8 in ?? ()
#2 0x000000000000000d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400037198 in ?? ()
#5 0x00007f4957393840 in ?? ()
#6 0x000000000000001a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 17588):
#0 0x00007f496c192fb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007f4957b94410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 17587):
#0 0x00007f496c192ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 17586):
#0 0x00007f4967597a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 17585):
#0 0x00007f4967597a47 in ?? ()
#1 0x00007b2800030028 in ?? ()
#2 0x0044e00003220066 in ?? ()
#3 0x00007f4959397500 in ?? ()
#4 0x00007f4959398b80 in ?? ()
#5 0x00007f4959397500 in ?? ()
#6 0x0000000000000011 in ?? ()
#7 0x00007b5800001800 in ?? ()
#8 0x0000000000488695 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f4964e22000 in ?? ()
#10 0x0000000000488599 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f4959398b80 in ?? ()
#12 0x00007f4969ff0069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f496f7711a0 in ?? ()
#15 0x00007b4c00002f90 in ?? ()
#16 0x00007b4c00002f98 in ?? ()
#17 0x00007f49593977a0 in ?? ()
#18 0x00007b4400033d00 in ?? ()
#19 0x00007f4959397cd0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 11 (LWP 17584):
#0 0x00007f4967597a47 in ?? ()
#1 0x00007b2800035028 in ?? ()
#2 0x0040e000000a7870 in ?? ()
#3 0x00007f4959b98500 in ?? ()
#4 0x00007f4959b99b80 in ?? ()
#5 0x00007f4959b98500 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x00007b5800001200 in ?? ()
#8 0x0000000000488695 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f4964e34000 in ?? ()
#10 0x0000000000488599 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f4959b99b80 in ?? ()
#12 0x00007f4969ff0069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f496f7711a0 in ?? ()
#15 0x00007b4c000026d0 in ?? ()
#16 0x00007b4c000026d8 in ?? ()
#17 0x00007f4959b987a0 in ?? ()
#18 0x00007b44000363c0 in ?? ()
#19 0x00007f4959b98cd0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 10 (LWP 17583):
#0 0x00007f4967597a47 in ?? ()
#1 0x00007b5800010408 in ?? ()
#2 0x003ce00002279783 in ?? ()
#3 0x00007f495cbbe500 in ?? ()
#4 0x00007f495cbbfb80 in ?? ()
#5 0x00007f495cbbe500 in ?? ()
#6 0x000000000000000d in ?? ()
#7 0x00007b5800000f00 in ?? ()
#8 0x0000000000488695 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f4964e46000 in ?? ()
#10 0x0000000000488599 in __sanitizer::internal_alloc_placeholder ()
#11 0x00007f495cbbfb80 in ?? ()
#12 0x00007f4969ff0069 in ?? ()
#13 0x00007b4c00000000 in ?? ()
#14 0x00007f496f7711a0 in ?? ()
#15 0x00007b4c00002c10 in ?? ()
#16 0x00007b4c00002c18 in ?? ()
#17 0x00007f495cbbe7a0 in ?? ()
#18 0x00007b4400036a00 in ?? ()
#19 0x00007f495cbbecd0 in ?? ()
#20 0x0000000000000000 in ?? ()
Thread 9 (LWP 17580):
#0 0x00007f496758acb9 in ?? ()
#1 0x00007f49603bcc10 in ?? ()
#2 0x00007b0400009510 in ?? ()
#3 0x00007f49603bdb80 in ?? ()
#4 0x00007f49603bcc10 in ?? ()
#5 0x00007b0400009510 in ?? ()
#6 0x00000000004888a3 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007f4965000000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007f49603bdb80 in ?? ()
#10 0x00007f4970f6fb28 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 8 (LWP 17579):
#0 0x00007f496c192fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 7 (LWP 17578):
#0 0x00007f496c1969e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 17571):
#0 0x00007f496c192fb9 in ?? ()
#1 0x00007f49613bea40 in ?? ()
#2 0x000000000000014a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000361d8 in ?? ()
#5 0x00007f49613be5d0 in ?? ()
#6 0x0000000000000294 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 17570):
#0 0x00007f496c192fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 17569):
#0 0x00007f496c192fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 17568):
#0 0x00007f496c192fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 17567):
#0 0x00007f496755a7a0 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 17566):
#0 0x00007f496c196d50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x0000000000467b2b in __sanitizer::internal_alloc_placeholder ()
#3 0x00007f49667b8cc0 in ?? ()
#4 0x00007f49667b8cc0 in ?? ()
#5 0x00007ffcc49053a0 in ?? ()
#6 0x000000000048aef4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000a940da in ?? ()
#9 0x00007f49667b8cc0 in ?? ()
#10 0x00007f496a6bef0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 01:58:52.831281 16802 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID ad4b390fedd44692b464a4f84bddeba2 and pid 17699
************************ BEGIN STACKS **************************
[New LWP 17700]
[New LWP 17701]
[New LWP 17702]
[New LWP 17703]
[New LWP 17704]
[New LWP 17711]
[New LWP 17712]
[New LWP 17713]
[New LWP 17716]
[New LWP 17717]
[New LWP 17718]
[New LWP 17719]
[New LWP 17720]
[New LWP 17721]
[New LWP 17722]
[New LWP 17723]
[New LWP 17724]
[New LWP 17725]
[New LWP 17726]
[New LWP 17727]
[New LWP 17728]
[New LWP 17729]
[New LWP 17730]
[New LWP 17731]
[New LWP 17732]
[New LWP 17733]
[New LWP 17734]
[New LWP 17735]
[New LWP 17736]
[New LWP 17737]
[New LWP 17738]
[New LWP 17739]
[New LWP 17740]
[New LWP 17741]
[New LWP 17742]
[New LWP 17743]
[New LWP 17744]
[New LWP 17745]
[New LWP 17746]
[New LWP 17747]
[New LWP 17748]
[New LWP 17749]
[New LWP 17750]
[New LWP 17751]
[New LWP 17752]
[New LWP 17753]
[New LWP 17754]
[New LWP 17755]
[New LWP 17756]
[New LWP 17757]
[New LWP 17758]
[New LWP 17759]
[New LWP 17760]
[New LWP 17761]
[New LWP 17762]
[New LWP 17763]
[New LWP 17764]
[New LWP 17765]
[New LWP 17766]
[New LWP 17767]
[New LWP 17768]
[New LWP 17769]
[New LWP 17770]
[New LWP 17771]
[New LWP 17772]
[New LWP 17773]
[New LWP 17774]
[New LWP 17775]
[New LWP 17776]
[New LWP 17777]
[New LWP 17778]
[New LWP 17779]
[New LWP 17780]
[New LWP 17781]
[New LWP 17782]
[New LWP 17783]
[New LWP 17784]
[New LWP 17785]
[New LWP 17786]
[New LWP 17787]
[New LWP 17788]
[New LWP 17789]
[New LWP 17790]
[New LWP 17791]
[New LWP 17792]
[New LWP 17793]
[New LWP 17794]
[New LWP 17795]
[New LWP 17796]
[New LWP 17797]
[New LWP 17798]
[New LWP 17799]
[New LWP 17800]
[New LWP 17801]
[New LWP 17802]
[New LWP 17803]
[New LWP 17804]
[New LWP 17805]
[New LWP 17806]
[New LWP 17807]
[New LWP 17808]
[New LWP 17809]
[New LWP 17810]
[New LWP 17811]
[New LWP 17812]
[New LWP 17813]
[New LWP 17814]
[New LWP 17815]
[New LWP 17816]
[New LWP 17817]
[New LWP 17818]
[New LWP 17819]
[New LWP 17820]
[New LWP 17821]
[New LWP 17822]
[New LWP 17823]
[New LWP 17824]
[New LWP 17825]
[New LWP 17826]
[New LWP 17827]
[New LWP 17828]
[New LWP 17829]
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
0x00007f5ee2495d50 in ?? ()
Id Target Id Frame
* 1 LWP 17699 "kudu" 0x00007f5ee2495d50 in ?? ()
2 LWP 17700 "kudu" 0x00007f5edd8597a0 in ?? ()
3 LWP 17701 "kudu" 0x00007f5ee2491fb9 in ?? ()
4 LWP 17702 "kudu" 0x00007f5ee2491fb9 in ?? ()
5 LWP 17703 "kudu" 0x00007f5ee2491fb9 in ?? ()
6 LWP 17704 "kernel-watcher-" 0x00007f5ee2491fb9 in ?? ()
7 LWP 17711 "ntp client-1771" 0x00007f5ee24959e2 in ?? ()
8 LWP 17712 "file cache-evic" 0x00007f5ee2491fb9 in ?? ()
9 LWP 17713 "sq_acceptor" 0x00007f5edd889cb9 in ?? ()
10 LWP 17716 "rpc reactor-177" 0x00007f5edd896a47 in ?? ()
11 LWP 17717 "rpc reactor-177" 0x00007f5edd896a47 in ?? ()
12 LWP 17718 "rpc reactor-177" 0x00007f5edd896a47 in ?? ()
13 LWP 17719 "rpc reactor-177" 0x00007f5edd896a47 in ?? ()
14 LWP 17720 "MaintenanceMgr " 0x00007f5ee2491ad3 in ?? ()
15 LWP 17721 "txn-status-mana" 0x00007f5ee2491fb9 in ?? ()
16 LWP 17722 "collect_and_rem" 0x00007f5ee2491fb9 in ?? ()
17 LWP 17723 "tc-session-exp-" 0x00007f5ee2491fb9 in ?? ()
18 LWP 17724 "rpc worker-1772" 0x00007f5ee2491ad3 in ?? ()
19 LWP 17725 "rpc worker-1772" 0x00007f5ee2491ad3 in ?? ()
20 LWP 17726 "rpc worker-1772" 0x00007f5ee2491ad3 in ?? ()
21 LWP 17727 "rpc worker-1772" 0x00007f5ee2491ad3 in ?? ()
22 LWP 17728 "rpc worker-1772" 0x00007f5ee2491ad3 in ?? ()
23 LWP 17729 "rpc worker-1772" 0x00007f5ee2491ad3 in ?? ()
24 LWP 17730 "rpc worker-1773" 0x00007f5ee2491ad3 in ?? ()
25 LWP 17731 "rpc worker-1773" 0x00007f5ee2491ad3 in ?? ()
26 LWP 17732 "rpc worker-1773" 0x00007f5ee2491ad3 in ?? ()
27 LWP 17733 "rpc worker-1773" 0x00007f5ee2491ad3 in ?? ()
28 LWP 17734 "rpc worker-1773" 0x00007f5ee2491ad3 in ?? ()
29 LWP 17735 "rpc worker-1773" 0x00007f5ee2491ad3 in ?? ()
30 LWP 17736 "rpc worker-1773" 0x00007f5ee2491ad3 in ?? ()
31 LWP 17737 "rpc worker-1773" 0x00007f5ee2491ad3 in ?? ()
32 LWP 17738 "rpc worker-1773" 0x00007f5ee2491ad3 in ?? ()
33 LWP 17739 "rpc worker-1773" 0x00007f5ee2491ad3 in ?? ()
34 LWP 17740 "rpc worker-1774" 0x00007f5ee2491ad3 in ?? ()
35 LWP 17741 "rpc worker-1774" 0x00007f5ee2491ad3 in ?? ()
36 LWP 17742 "rpc worker-1774" 0x00007f5ee2491ad3 in ?? ()
37 LWP 17743 "rpc worker-1774" 0x00007f5ee2491ad3 in ?? ()
38 LWP 17744 "rpc worker-1774" 0x00007f5ee2491ad3 in ?? ()
39 LWP 17745 "rpc worker-1774" 0x00007f5ee2491ad3 in ?? ()
40 LWP 17746 "rpc worker-1774" 0x00007f5ee2491ad3 in ?? ()
41 LWP 17747 "rpc worker-1774" 0x00007f5ee2491ad3 in ?? ()
42 LWP 17748 "rpc worker-1774" 0x00007f5ee2491ad3 in ?? ()
43 LWP 17749 "rpc worker-1774" 0x00007f5ee2491ad3 in ?? ()
44 LWP 17750 "rpc worker-1775" 0x00007f5ee2491ad3 in ?? ()
45 LWP 17751 "rpc worker-1775" 0x00007f5ee2491ad3 in ?? ()
46 LWP 17752 "rpc worker-1775" 0x00007f5ee2491ad3 in ?? ()
47 LWP 17753 "rpc worker-1775" 0x00007f5ee2491ad3 in ?? ()
48 LWP 17754 "rpc worker-1775" 0x00007f5ee2491ad3 in ?? ()
49 LWP 17755 "rpc worker-1775" 0x00007f5ee2491ad3 in ?? ()
50 LWP 17756 "rpc worker-1775" 0x00007f5ee2491ad3 in ?? ()
51 LWP 17757 "rpc worker-1775" 0x00007f5ee2491ad3 in ?? ()
52 LWP 17758 "rpc worker-1775" 0x00007f5ee2491ad3 in ?? ()
53 LWP 17759 "rpc worker-1775" 0x00007f5ee2491ad3 in ?? ()
54 LWP 17760 "rpc worker-1776" 0x00007f5ee2491ad3 in ?? ()
55 LWP 17761 "rpc worker-1776" 0x00007f5ee2491ad3 in ?? ()
56 LWP 17762 "rpc worker-1776" 0x00007f5ee2491ad3 in ?? ()
57 LWP 17763 "rpc worker-1776" 0x00007f5ee2491ad3 in ?? ()
58 LWP 17764 "rpc worker-1776" 0x00007f5ee2491ad3 in ?? ()
59 LWP 17765 "rpc worker-1776" 0x00007f5ee2491ad3 in ?? ()
60 LWP 17766 "rpc worker-1776" 0x00007f5ee2491ad3 in ?? ()
61 LWP 17767 "rpc worker-1776" 0x00007f5ee2491ad3 in ?? ()
62 LWP 17768 "rpc worker-1776" 0x00007f5ee2491ad3 in ?? ()
63 LWP 17769 "rpc worker-1776" 0x00007f5ee2491ad3 in ?? ()
64 LWP 17770 "rpc worker-1777" 0x00007f5ee2491ad3 in ?? ()
65 LWP 17771 "rpc worker-1777" 0x00007f5ee2491ad3 in ?? ()
66 LWP 17772 "rpc worker-1777" 0x00007f5ee2491ad3 in ?? ()
67 LWP 17773 "rpc worker-1777" 0x00007f5ee2491ad3 in ?? ()
68 LWP 17774 "rpc worker-1777" 0x00007f5ee2491ad3 in ?? ()
69 LWP 17775 "rpc worker-1777" 0x00007f5ee2491ad3 in ?? ()
70 LWP 17776 "rpc worker-1777" 0x00007f5ee2491ad3 in ?? ()
71 LWP 17777 "rpc worker-1777" 0x00007f5ee2491ad3 in ?? ()
72 LWP 17778 "rpc worker-1777" 0x00007f5ee2491ad3 in ?? ()
73 LWP 17779 "rpc worker-1777" 0x00007f5ee2491ad3 in ?? ()
74 LWP 17780 "rpc worker-1778" 0x00007f5ee2491ad3 in ?? ()
75 LWP 17781 "rpc worker-1778" 0x00007f5ee2491ad3 in ?? ()
76 LWP 17782 "rpc worker-1778" 0x00007f5ee2491ad3 in ?? ()
77 LWP 17783 "rpc worker-1778" 0x00007f5ee2491ad3 in ?? ()
78 LWP 17784 "rpc worker-1778" 0x00007f5ee2491ad3 in ?? ()
79 LWP 17785 "rpc worker-1778" 0x00007f5ee2491ad3 in ?? ()
80 LWP 17786 "rpc worker-1778" 0x00007f5ee2491ad3 in ?? ()
81 LWP 17787 "rpc worker-1778" 0x00007f5ee2491ad3 in ?? ()
82 LWP 17788 "rpc worker-1778" 0x00007f5ee2491ad3 in ?? ()
83 LWP 17789 "rpc worker-1778" 0x00007f5ee2491ad3 in ?? ()
84 LWP 17790 "rpc worker-1779" 0x00007f5ee2491ad3 in ?? ()
85 LWP 17791 "rpc worker-1779" 0x00007f5ee2491ad3 in ?? ()
86 LWP 17792 "rpc worker-1779" 0x00007f5ee2491ad3 in ?? ()
87 LWP 17793 "rpc worker-1779" 0x00007f5ee2491ad3 in ?? ()
88 LWP 17794 "rpc worker-1779" 0x00007f5ee2491ad3 in ?? ()
89 LWP 17795 "rpc worker-1779" 0x00007f5ee2491ad3 in ?? ()
90 LWP 17796 "rpc worker-1779" 0x00007f5ee2491ad3 in ?? ()
91 LWP 17797 "rpc worker-1779" 0x00007f5ee2491ad3 in ?? ()
92 LWP 17798 "rpc worker-1779" 0x00007f5ee2491ad3 in ?? ()
93 LWP 17799 "rpc worker-1779" 0x00007f5ee2491ad3 in ?? ()
94 LWP 17800 "rpc worker-1780" 0x00007f5ee2491ad3 in ?? ()
95 LWP 17801 "rpc worker-1780" 0x00007f5ee2491ad3 in ?? ()
96 LWP 17802 "rpc worker-1780" 0x00007f5ee2491ad3 in ?? ()
97 LWP 17803 "rpc worker-1780" 0x00007f5ee2491ad3 in ?? ()
98 LWP 17804 "rpc worker-1780" 0x00007f5ee2491ad3 in ?? ()
99 LWP 17805 "rpc worker-1780" 0x00007f5ee2491ad3 in ?? ()
100 LWP 17806 "rpc worker-1780" 0x00007f5ee2491ad3 in ?? ()
101 LWP 17807 "rpc worker-1780" 0x00007f5ee2491ad3 in ?? ()
102 LWP 17808 "rpc worker-1780" 0x00007f5ee2491ad3 in ?? ()
103 LWP 17809 "rpc worker-1780" 0x00007f5ee2491ad3 in ?? ()
104 LWP 17810 "rpc worker-1781" 0x00007f5ee2491ad3 in ?? ()
105 LWP 17811 "rpc worker-1781" 0x00007f5ee2491ad3 in ?? ()
106 LWP 17812 "rpc worker-1781" 0x00007f5ee2491ad3 in ?? ()
107 LWP 17813 "rpc worker-1781" 0x00007f5ee2491ad3 in ?? ()
108 LWP 17814 "rpc worker-1781" 0x00007f5ee2491ad3 in ?? ()
109 LWP 17815 "rpc worker-1781" 0x00007f5ee2491ad3 in ?? ()
110 LWP 17816 "rpc worker-1781" 0x00007f5ee2491ad3 in ?? ()
111 LWP 17817 "rpc worker-1781" 0x00007f5ee2491ad3 in ?? ()
112 LWP 17818 "rpc worker-1781" 0x00007f5ee2491ad3 in ?? ()
113 LWP 17819 "rpc worker-1781" 0x00007f5ee2491ad3 in ?? ()
114 LWP 17820 "rpc worker-1782" 0x00007f5ee2491ad3 in ?? ()
115 LWP 17821 "rpc worker-1782" 0x00007f5ee2491ad3 in ?? ()
116 LWP 17822 "rpc worker-1782" 0x00007f5ee2491ad3 in ?? ()
117 LWP 17823 "rpc worker-1782" 0x00007f5ee2491ad3 in ?? ()
118 LWP 17824 "diag-logger-178" 0x00007f5ee2491fb9 in ?? ()
119 LWP 17825 "result-tracker-" 0x00007f5ee2491fb9 in ?? ()
120 LWP 17826 "excess-log-dele" 0x00007f5ee2491fb9 in ?? ()
121 LWP 17827 "acceptor-17827" 0x00007f5edd8980c7 in ?? ()
122 LWP 17828 "heartbeat-17828" 0x00007f5ee2491fb9 in ?? ()
123 LWP 17829 "maintenance_sch" 0x00007f5ee2491fb9 in ?? ()
Thread 123 (LWP 17829):
#0 0x00007f5ee2491fb9 in ?? ()
#1 0x00007b0100000000 in ?? ()
#2 0x0000000000000100 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007f5e966b96c0 in ?? ()
#6 0x0000000000000200 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 17828):
#0 0x00007f5ee2491fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 17827):
#0 0x00007f5edd8980c7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 17826):
#0 0x00007f5ee2491fb9 in ?? ()
#1 0x00007f5e97ebc940 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffcdce7b4c0 in ?? ()
#5 0x00007f5e97ebc7b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 17825):
#0 0x00007f5ee2491fb9 in ?? ()
#1 0x0000000085352fb8 in ?? ()
#2 0x0000000000000040 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001008 in ?? ()
#5 0x00007f5e986bd800 in ?? ()
#6 0x0000000000000080 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 17824):
#0 0x00007f5ee2491fb9 in ?? ()
#1 0x00007f5edb90e008 in ?? ()
#2 0x000000000000003b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000c90 in ?? ()
#5 0x00007f5e98ebe750 in ?? ()
#6 0x0000000000000076 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 17823):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 17822):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 17821):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 17820):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 17819):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 17818):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 17817):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 17816):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 17815):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 17814):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 17813):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 17812):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 17811):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 17810):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 17809):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 17808):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 17807):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 17806):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 17805):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 17804):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 17803):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 17802):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 17801):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 17800):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 17799):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 17798):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 17797):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 17796):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 17795):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 17794):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 17793):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 17792):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 17791):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 17790):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 17789):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 17788):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 17787):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 17786):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 17785):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 17784):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 17783):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x00000000000001ce in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b24001147c8 in ?? ()
#4 0x00007f5eae0ba710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5eae0ba730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 76 (LWP 17782):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000919 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240010ffcc in ?? ()
#4 0x00007f5eae8bb710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5eae8bb730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f5ee2491770 in ?? ()
#10 0x00007f5eae8bb730 in ?? ()
#11 0x00007f5e990f0678 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 17781):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x000000000000073a in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240010d7d8 in ?? ()
#4 0x00007f5eaf0bc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5eaf0bc730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 17780):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 17779):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 17778):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 17777):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 17776):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 17775):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 17774):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 17773):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 17772):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 17771):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 17770):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 17769):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 17768):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 17767):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 17766):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 17765):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 17764):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 17763):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000b902c in ?? ()
#4 0x00007f5eb84bc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5eb84bc730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x007f0400000026c8 in ?? ()
#9 0x00007f5ee2491770 in ?? ()
#10 0x00007f5eb84bc730 in ?? ()
#11 0x0002008300000dfe in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 17762):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 17761):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 17760):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 17759):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 17758):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 17757):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 17756):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 17755):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 17754):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 17753):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 17752):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 17751):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 17750):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 17749):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 17748):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 17747):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 17746):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 17745):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 17744):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 17743):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240005ffec in ?? ()
#4 0x00007f5ec28be710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5ec28be730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x0000000000000000 in ?? ()
Thread 36 (LWP 17742):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240005d7f8 in ?? ()
#4 0x00007f5ec32b6710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5ec32b6730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 17741):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 17740):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 17739):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 17738):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 17737):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 17736):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 17735):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 17734):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 17733):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 17732):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 17731):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 17730):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 17729):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 17728):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 17727):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 17726):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 17725):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 17724):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 17723):
#0 0x00007f5ee2491fb9 in ?? ()
#1 0x0000000017a335f0 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800003a00 in ?? ()
#5 0x00007f5ecce92700 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 17722):
#0 0x00007f5ee2491fb9 in ?? ()
#1 0x00007f5ecd6939a8 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400037198 in ?? ()
#5 0x00007f5ecd693840 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 17721):
#0 0x00007f5ee2491fb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007f5ecde94410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 17720):
#0 0x00007f5ee2491ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 17719):
#0 0x00007f5edd896a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 17718):
#0 0x00007f5edd896a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 11 (LWP 17717):
#0 0x00007f5edd896a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 10 (LWP 17716):
#0 0x00007f5edd896a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 9 (LWP 17713):
#0 0x00007f5edd889cb9 in ?? ()
#1 0x00007f5ed66bcc10 in ?? ()
#2 0x00007b040000a850 in ?? ()
#3 0x00007f5ed66bdb80 in ?? ()
#4 0x00007f5ed66bcc10 in ?? ()
#5 0x00007b040000a850 in ?? ()
#6 0x00000000004888a3 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007f5edb302000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007f5ed66bdb80 in ?? ()
#10 0x00007f5ee726eb28 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 8 (LWP 17712):
#0 0x00007f5ee2491fb9 in ?? ()
#1 0x0000600000000000 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400034018 in ?? ()
#5 0x00007f5ed5ebb7f0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 17711):
#0 0x00007f5ee24959e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 17704):
#0 0x00007f5ee2491fb9 in ?? ()
#1 0x00007f5ed76bea40 in ?? ()
#2 0x0000000000000146 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000361d8 in ?? ()
#5 0x00007f5ed76be5d0 in ?? ()
#6 0x000000000000028c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 17703):
#0 0x00007f5ee2491fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 17702):
#0 0x00007f5ee2491fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 17701):
#0 0x00007f5ee2491fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 17700):
#0 0x00007f5edd8597a0 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 17699):
#0 0x00007f5ee2495d50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x0000000000467b2b in __sanitizer::internal_alloc_placeholder ()
#3 0x00007f5edcab7cc0 in ?? ()
#4 0x00007f5edcab7cc0 in ?? ()
#5 0x00007ffcdce7b2d0 in ?? ()
#6 0x000000000048aef4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000a9dd48 in ?? ()
#9 0x00007f5edcab7cc0 in ?? ()
#10 0x00007f5ee09bdf0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 01:58:53.779870 16802 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID c1fb7a5c939b43d1a9af547a9b6f18b8 and pid 17832
************************ BEGIN STACKS **************************
[New LWP 17833]
[New LWP 17834]
[New LWP 17835]
[New LWP 17836]
[New LWP 17837]
[New LWP 17844]
[New LWP 17845]
[New LWP 17846]
[New LWP 17849]
[New LWP 17850]
[New LWP 17851]
[New LWP 17852]
[New LWP 17853]
[New LWP 17854]
[New LWP 17855]
[New LWP 17856]
[New LWP 17857]
[New LWP 17858]
[New LWP 17859]
[New LWP 17860]
[New LWP 17861]
[New LWP 17862]
[New LWP 17863]
[New LWP 17864]
[New LWP 17865]
[New LWP 17866]
[New LWP 17867]
[New LWP 17868]
[New LWP 17869]
[New LWP 17870]
[New LWP 17871]
[New LWP 17872]
[New LWP 17873]
[New LWP 17874]
[New LWP 17875]
[New LWP 17876]
[New LWP 17877]
[New LWP 17878]
[New LWP 17879]
[New LWP 17880]
[New LWP 17881]
[New LWP 17882]
[New LWP 17883]
[New LWP 17884]
[New LWP 17885]
[New LWP 17886]
[New LWP 17887]
[New LWP 17888]
[New LWP 17889]
[New LWP 17890]
[New LWP 17891]
[New LWP 17892]
[New LWP 17893]
[New LWP 17894]
[New LWP 17895]
[New LWP 17896]
[New LWP 17897]
[New LWP 17898]
[New LWP 17899]
[New LWP 17900]
[New LWP 17901]
[New LWP 17902]
[New LWP 17903]
[New LWP 17904]
[New LWP 17905]
[New LWP 17906]
[New LWP 17907]
[New LWP 17908]
[New LWP 17909]
[New LWP 17910]
[New LWP 17911]
[New LWP 17912]
[New LWP 17913]
[New LWP 17914]
[New LWP 17915]
[New LWP 17916]
[New LWP 17917]
[New LWP 17918]
[New LWP 17919]
[New LWP 17920]
[New LWP 17921]
[New LWP 17922]
[New LWP 17923]
[New LWP 17924]
[New LWP 17925]
[New LWP 17926]
[New LWP 17927]
[New LWP 17928]
[New LWP 17929]
[New LWP 17930]
[New LWP 17931]
[New LWP 17932]
[New LWP 17933]
[New LWP 17934]
[New LWP 17935]
[New LWP 17936]
[New LWP 17937]
[New LWP 17938]
[New LWP 17939]
[New LWP 17940]
[New LWP 17941]
[New LWP 17942]
[New LWP 17943]
[New LWP 17944]
[New LWP 17945]
[New LWP 17946]
[New LWP 17947]
[New LWP 17948]
[New LWP 17949]
[New LWP 17950]
[New LWP 17951]
[New LWP 17952]
[New LWP 17953]
[New LWP 17954]
[New LWP 17955]
[New LWP 17956]
[New LWP 17957]
[New LWP 17958]
[New LWP 17959]
[New LWP 17960]
[New LWP 17961]
[New LWP 17962]
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c48020396
Cannot access memory at address 0x4108070c4802038e
0x00007f07805c1d50 in ?? ()
Id Target Id Frame
* 1 LWP 17832 "kudu" 0x00007f07805c1d50 in ?? ()
2 LWP 17833 "kudu" 0x00007f077b9857a0 in ?? ()
3 LWP 17834 "kudu" 0x00007f07805bdfb9 in ?? ()
4 LWP 17835 "kudu" 0x00007f07805bdfb9 in ?? ()
5 LWP 17836 "kudu" 0x00007f07805bdfb9 in ?? ()
6 LWP 17837 "kernel-watcher-" 0x00007f07805bdfb9 in ?? ()
7 LWP 17844 "ntp client-1784" 0x00007f07805c19e2 in ?? ()
8 LWP 17845 "file cache-evic" 0x00007f07805bdfb9 in ?? ()
9 LWP 17846 "sq_acceptor" 0x00007f077b9b5cb9 in ?? ()
10 LWP 17849 "rpc reactor-178" 0x00007f077b9c2a47 in ?? ()
11 LWP 17850 "rpc reactor-178" 0x00007f077b9c2a47 in ?? ()
12 LWP 17851 "rpc reactor-178" 0x00007f077b9c2a47 in ?? ()
13 LWP 17852 "rpc reactor-178" 0x00007f077b9c2a47 in ?? ()
14 LWP 17853 "MaintenanceMgr " 0x00007f07805bdad3 in ?? ()
15 LWP 17854 "txn-status-mana" 0x00007f07805bdfb9 in ?? ()
16 LWP 17855 "collect_and_rem" 0x00007f07805bdfb9 in ?? ()
17 LWP 17856 "tc-session-exp-" 0x00007f07805bdfb9 in ?? ()
18 LWP 17857 "rpc worker-1785" 0x00007f07805bdad3 in ?? ()
19 LWP 17858 "rpc worker-1785" 0x00007f07805bdad3 in ?? ()
20 LWP 17859 "rpc worker-1785" 0x00007f07805bdad3 in ?? ()
21 LWP 17860 "rpc worker-1786" 0x00007f07805bdad3 in ?? ()
22 LWP 17861 "rpc worker-1786" 0x00007f07805bdad3 in ?? ()
23 LWP 17862 "rpc worker-1786" 0x00007f07805bdad3 in ?? ()
24 LWP 17863 "rpc worker-1786" 0x00007f07805bdad3 in ?? ()
25 LWP 17864 "rpc worker-1786" 0x00007f07805bdad3 in ?? ()
26 LWP 17865 "rpc worker-1786" 0x00007f07805bdad3 in ?? ()
27 LWP 17866 "rpc worker-1786" 0x00007f07805bdad3 in ?? ()
28 LWP 17867 "rpc worker-1786" 0x00007f07805bdad3 in ?? ()
29 LWP 17868 "rpc worker-1786" 0x00007f07805bdad3 in ?? ()
30 LWP 17869 "rpc worker-1786" 0x00007f07805bdad3 in ?? ()
31 LWP 17870 "rpc worker-1787" 0x00007f07805bdad3 in ?? ()
32 LWP 17871 "rpc worker-1787" 0x00007f07805bdad3 in ?? ()
33 LWP 17872 "rpc worker-1787" 0x00007f07805bdad3 in ?? ()
34 LWP 17873 "rpc worker-1787" 0x00007f07805bdad3 in ?? ()
35 LWP 17874 "rpc worker-1787" 0x00007f07805bdad3 in ?? ()
36 LWP 17875 "rpc worker-1787" 0x00007f07805bdad3 in ?? ()
37 LWP 17876 "rpc worker-1787" 0x00007f07805bdad3 in ?? ()
38 LWP 17877 "rpc worker-1787" 0x00007f07805bdad3 in ?? ()
39 LWP 17878 "rpc worker-1787" 0x00007f07805bdad3 in ?? ()
40 LWP 17879 "rpc worker-1787" 0x00007f07805bdad3 in ?? ()
41 LWP 17880 "rpc worker-1788" 0x00007f07805bdad3 in ?? ()
42 LWP 17881 "rpc worker-1788" 0x00007f07805bdad3 in ?? ()
43 LWP 17882 "rpc worker-1788" 0x00007f07805bdad3 in ?? ()
44 LWP 17883 "rpc worker-1788" 0x00007f07805bdad3 in ?? ()
45 LWP 17884 "rpc worker-1788" 0x00007f07805bdad3 in ?? ()
46 LWP 17885 "rpc worker-1788" 0x00007f07805bdad3 in ?? ()
47 LWP 17886 "rpc worker-1788" 0x00007f07805bdad3 in ?? ()
48 LWP 17887 "rpc worker-1788" 0x00007f07805bdad3 in ?? ()
49 LWP 17888 "rpc worker-1788" 0x00007f07805bdad3 in ?? ()
50 LWP 17889 "rpc worker-1788" 0x00007f07805bdad3 in ?? ()
51 LWP 17890 "rpc worker-1789" 0x00007f07805bdad3 in ?? ()
52 LWP 17891 "rpc worker-1789" 0x00007f07805bdad3 in ?? ()
53 LWP 17892 "rpc worker-1789" 0x00007f07805bdad3 in ?? ()
54 LWP 17893 "rpc worker-1789" 0x00007f07805bdad3 in ?? ()
55 LWP 17894 "rpc worker-1789" 0x00007f07805bdad3 in ?? ()
56 LWP 17895 "rpc worker-1789" 0x00007f07805bdad3 in ?? ()
57 LWP 17896 "rpc worker-1789" 0x00007f07805bdad3 in ?? ()
58 LWP 17897 "rpc worker-1789" 0x00007f07805bdad3 in ?? ()
59 LWP 17898 "rpc worker-1789" 0x00007f07805bdad3 in ?? ()
60 LWP 17899 "rpc worker-1789" 0x00007f07805bdad3 in ?? ()
61 LWP 17900 "rpc worker-1790" 0x00007f07805bdad3 in ?? ()
62 LWP 17901 "rpc worker-1790" 0x00007f07805bdad3 in ?? ()
63 LWP 17902 "rpc worker-1790" 0x00007f07805bdad3 in ?? ()
64 LWP 17903 "rpc worker-1790" 0x00007f07805bdad3 in ?? ()
65 LWP 17904 "rpc worker-1790" 0x00007f07805bdad3 in ?? ()
66 LWP 17905 "rpc worker-1790" 0x00007f07805bdad3 in ?? ()
67 LWP 17906 "rpc worker-1790" 0x00007f07805bdad3 in ?? ()
68 LWP 17907 "rpc worker-1790" 0x00007f07805bdad3 in ?? ()
69 LWP 17908 "rpc worker-1790" 0x00007f07805bdad3 in ?? ()
70 LWP 17909 "rpc worker-1790" 0x00007f07805bdad3 in ?? ()
71 LWP 17910 "rpc worker-1791" 0x00007f07805bdad3 in ?? ()
72 LWP 17911 "rpc worker-1791" 0x00007f07805bdad3 in ?? ()
73 LWP 17912 "rpc worker-1791" 0x00007f07805bdad3 in ?? ()
74 LWP 17913 "rpc worker-1791" 0x00007f07805bdad3 in ?? ()
75 LWP 17914 "rpc worker-1791" 0x00007f07805bdad3 in ?? ()
76 LWP 17915 "rpc worker-1791" 0x00007f07805bdad3 in ?? ()
77 LWP 17916 "rpc worker-1791" 0x00007f07805bdad3 in ?? ()
78 LWP 17917 "rpc worker-1791" 0x00007f07805bdad3 in ?? ()
79 LWP 17918 "rpc worker-1791" 0x00007f07805bdad3 in ?? ()
80 LWP 17919 "rpc worker-1791" 0x00007f07805bdad3 in ?? ()
81 LWP 17920 "rpc worker-1792" 0x00007f07805bdad3 in ?? ()
82 LWP 17921 "rpc worker-1792" 0x00007f07805bdad3 in ?? ()
83 LWP 17922 "rpc worker-1792" 0x00007f07805bdad3 in ?? ()
84 LWP 17923 "rpc worker-1792" 0x00007f07805bdad3 in ?? ()
85 LWP 17924 "rpc worker-1792" 0x00007f07805bdad3 in ?? ()
86 LWP 17925 "rpc worker-1792" 0x00007f07805bdad3 in ?? ()
87 LWP 17926 "rpc worker-1792" 0x00007f07805bdad3 in ?? ()
88 LWP 17927 "rpc worker-1792" 0x00007f07805bdad3 in ?? ()
89 LWP 17928 "rpc worker-1792" 0x00007f07805bdad3 in ?? ()
90 LWP 17929 "rpc worker-1792" 0x00007f07805bdad3 in ?? ()
91 LWP 17930 "rpc worker-1793" 0x00007f07805bdad3 in ?? ()
92 LWP 17931 "rpc worker-1793" 0x00007f07805bdad3 in ?? ()
93 LWP 17932 "rpc worker-1793" 0x00007f07805bdad3 in ?? ()
94 LWP 17933 "rpc worker-1793" 0x00007f07805bdad3 in ?? ()
95 LWP 17934 "rpc worker-1793" 0x00007f07805bdad3 in ?? ()
96 LWP 17935 "rpc worker-1793" 0x00007f07805bdad3 in ?? ()
97 LWP 17936 "rpc worker-1793" 0x00007f07805bdad3 in ?? ()
98 LWP 17937 "rpc worker-1793" 0x00007f07805bdad3 in ?? ()
99 LWP 17938 "rpc worker-1793" 0x00007f07805bdad3 in ?? ()
100 LWP 17939 "rpc worker-1793" 0x00007f07805bdad3 in ?? ()
101 LWP 17940 "rpc worker-1794" 0x00007f07805bdad3 in ?? ()
102 LWP 17941 "rpc worker-1794" 0x00007f07805bdad3 in ?? ()
103 LWP 17942 "rpc worker-1794" 0x00007f07805bdad3 in ?? ()
104 LWP 17943 "rpc worker-1794" 0x00007f07805bdad3 in ?? ()
105 LWP 17944 "rpc worker-1794" 0x00007f07805bdad3 in ?? ()
106 LWP 17945 "rpc worker-1794" 0x00007f07805bdad3 in ?? ()
107 LWP 17946 "rpc worker-1794" 0x00007f07805bdad3 in ?? ()
108 LWP 17947 "rpc worker-1794" 0x00007f07805bdad3 in ?? ()
109 LWP 17948 "rpc worker-1794" 0x00007f07805bdad3 in ?? ()
110 LWP 17949 "rpc worker-1794" 0x00007f07805bdad3 in ?? ()
111 LWP 17950 "rpc worker-1795" 0x00007f07805bdad3 in ?? ()
112 LWP 17951 "rpc worker-1795" 0x00007f07805bdad3 in ?? ()
113 LWP 17952 "rpc worker-1795" 0x00007f07805bdad3 in ?? ()
114 LWP 17953 "rpc worker-1795" 0x00007f07805bdad3 in ?? ()
115 LWP 17954 "rpc worker-1795" 0x00007f07805bdad3 in ?? ()
116 LWP 17955 "rpc worker-1795" 0x00007f07805bdad3 in ?? ()
117 LWP 17956 "rpc worker-1795" 0x00007f07805bdad3 in ?? ()
118 LWP 17957 "diag-logger-179" 0x00007f07805bdfb9 in ?? ()
119 LWP 17958 "result-tracker-" 0x00007f07805bdfb9 in ?? ()
120 LWP 17959 "excess-log-dele" 0x00007f07805bdfb9 in ?? ()
121 LWP 17960 "acceptor-17960" 0x00007f077b9c40c7 in ?? ()
122 LWP 17961 "heartbeat-17961" 0x00007f07805bdfb9 in ?? ()
123 LWP 17962 "maintenance_sch" 0x00007f07805bdfb9 in ?? ()
Thread 123 (LWP 17962):
#0 0x00007f07805bdfb9 in ?? ()
#1 0x00007b0100000000 in ?? ()
#2 0x00000000000000fc in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b54000028f0 in ?? ()
#5 0x00007f07347b96c0 in ?? ()
#6 0x00000000000001f8 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 17961):
#0 0x00007f07805bdfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 121 (LWP 17960):
#0 0x00007f077b9c40c7 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 17959):
#0 0x00007f07805bdfb9 in ?? ()
#1 0x00007f0735fbc940 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffc9bfdf730 in ?? ()
#5 0x00007f0735fbc7b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 119 (LWP 17958):
#0 0x00007f07805bdfb9 in ?? ()
#1 0x0000000085352fb8 in ?? ()
#2 0x000000000000003f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b3400001008 in ?? ()
#5 0x00007f07367bd800 in ?? ()
#6 0x000000000000007e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 17957):
#0 0x00007f07805bdfb9 in ?? ()
#1 0x00007f0779a36008 in ?? ()
#2 0x0000000000000039 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4000000c90 in ?? ()
#5 0x00007f0736fbe750 in ?? ()
#6 0x0000000000000072 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 17956):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 17955):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 17954):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 17953):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 17952):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 17951):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 17950):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 17949):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 17948):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 17947):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 17946):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 17945):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 17944):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 17943):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 17942):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 17941):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 17940):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 17939):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 17938):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 17937):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 17936):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 17935):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 17934):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 17933):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 17932):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 17931):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 17930):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 17929):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 17928):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 17927):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 17926):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 17925):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 17924):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 17923):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 17922):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 17921):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 17920):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 17919):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 17918):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 17917):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 17916):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000974 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b24001147c8 in ?? ()
#4 0x00007f074c1ba710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f074c1ba730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 76 (LWP 17915):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000958 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007b240010ffc8 in ?? ()
#4 0x00007f074c9bb710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f074c9bb730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 17914):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 17913):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 17912):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 17911):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 17910):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 17909):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 17908):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 17907):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 17906):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 17905):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 17904):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 17903):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 17902):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 17901):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 17900):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 17899):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 17898):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 17897):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 17896):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b24000b902c in ?? ()
#4 0x00007f07565bc710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f07565bc730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x007f0400000026c8 in ?? ()
#9 0x00007f07805bd770 in ?? ()
#10 0x00007f07565bc730 in ?? ()
#11 0x0002008300000dfe in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 56 (LWP 17895):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 17894):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 17893):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 17892):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 17891):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 17890):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 17889):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 17888):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 17887):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 17886):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 17885):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 17884):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 17883):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 17882):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 17881):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 17880):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 17879):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 17878):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 17877):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 17876):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007b240005ffec in ?? ()
#4 0x00007f07609be710 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f07609be730 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000000000045e4c9 in __sanitizer::internal_alloc_placeholder ()
#9 0x00007f07805bd770 in ?? ()
#10 0x00007f07609be730 in ?? ()
#11 0x00007f0778b36c48 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 36 (LWP 17875):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 17874):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 17873):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 17872):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 17871):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 17870):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 17869):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 17868):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 17867):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 17866):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 17865):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 17864):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 17863):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 17862):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 17861):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 17860):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 17859):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 17858):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 17857):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 17856):
#0 0x00007f07805bdfb9 in ?? ()
#1 0x0000000017a335f0 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4800003a00 in ?? ()
#5 0x00007f076af92700 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 16 (LWP 17855):
#0 0x00007f07805bdfb9 in ?? ()
#1 0x00007f076b7939a8 in ?? ()
#2 0x000000000000000c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400037198 in ?? ()
#5 0x00007f076b793840 in ?? ()
#6 0x0000000000000018 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 17854):
#0 0x00007f07805bdfb9 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b5800000118 in ?? ()
#5 0x00007f076bf94410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 17853):
#0 0x00007f07805bdad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 13 (LWP 17852):
#0 0x00007f077b9c2a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 17851):
#0 0x00007f077b9c2a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 11 (LWP 17850):
#0 0x00007f077b9c2a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 10 (LWP 17849):
#0 0x00007f077b9c2a47 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 9 (LWP 17846):
#0 0x00007f077b9b5cb9 in ?? ()
#1 0x00007f07747bcc10 in ?? ()
#2 0x00007b040000a050 in ?? ()
#3 0x00007f07747bdb80 in ?? ()
#4 0x00007f07747bcc10 in ?? ()
#5 0x00007b040000a050 in ?? ()
#6 0x00000000004888a3 in __sanitizer::internal_alloc_placeholder ()
#7 0x00007f0779432000 in ?? ()
#8 0x0100000000000001 in ?? ()
#9 0x00007f07747bdb80 in ?? ()
#10 0x00007f078539ab28 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 8 (LWP 17845):
#0 0x00007f07805bdfb9 in ?? ()
#1 0x0000600000000000 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b4400034018 in ?? ()
#5 0x00007f0773fbb7f0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 17844):
#0 0x00007f07805c19e2 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 6 (LWP 17837):
#0 0x00007f07805bdfb9 in ?? ()
#1 0x00007f07757bea40 in ?? ()
#2 0x0000000000000142 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007b44000361d8 in ?? ()
#5 0x00007f07757be5d0 in ?? ()
#6 0x0000000000000284 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 5 (LWP 17836):
#0 0x00007f07805bdfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 4 (LWP 17835):
#0 0x00007f07805bdfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 17834):
#0 0x00007f07805bdfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 17833):
#0 0x00007f077b9857a0 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 1 (LWP 17832):
#0 0x00007f07805c1d50 in ?? ()
#1 0x0000600001000078 in ?? ()
#2 0x0000000000467b2b in __sanitizer::internal_alloc_placeholder ()
#3 0x00007f077abe3cc0 in ?? ()
#4 0x00007f077abe3cc0 in ?? ()
#5 0x00007ffc9bfdf540 in ?? ()
#6 0x000000000048aef4 in __sanitizer::internal_alloc_placeholder ()
#7 0x0000600001000078 in ?? ()
#8 0x0000e00000a9b99b in ?? ()
#9 0x00007f077abe3cc0 in ?? ()
#10 0x00007f077eae9f0b in ?? ()
#11 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 01:58:54.718225 16802 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu with pid 17566
I20250627 01:58:54.777978 16802 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu with pid 17699
I20250627 01:58:54.827416 16802 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu with pid 17832
I20250627 01:58:54.874636 16802 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskVaBo6p/build/tsan/bin/kudu with pid 17474
2025-06-27T01:58:54Z chronyd exiting
I20250627 01:58:54.928464 16802 test_util.cc:183] -----------------------------------------------
I20250627 01:58:54.928681 16802 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskVaBo6p/test-tmp/tablet_copy-itest.0.TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate.1750989441767164-16802-0
[ FAILED ] TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate (71202 ms)
[----------] 4 tests from TabletCopyITest (92996 ms total)
[----------] 1 test from FaultFlags/BadTabletCopyITest
[ RUN ] FaultFlags/BadTabletCopyITest.TestBadCopy/1
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/tablet_copy-itest.cc:1510: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] FaultFlags/BadTabletCopyITest.TestBadCopy/1 (7 ms)
[----------] 1 test from FaultFlags/BadTabletCopyITest (7 ms total)
[----------] Global test environment tear-down
[==========] 5 tests from 2 test suites ran. (93004 ms total)
[ PASSED ] 1 test.
[ SKIPPED ] 3 tests, listed below:
[ SKIPPED ] TabletCopyITest.TestRejectRogueLeader
[ SKIPPED ] TabletCopyITest.TestDeleteLeaderDuringTabletCopyStressTest
[ SKIPPED ] FaultFlags/BadTabletCopyITest.TestBadCopy/1
[ FAILED ] 1 test, listed below:
[ FAILED ] TabletCopyITest.TestDownloadWalInParallelWithHeavyUpdate
1 FAILED TEST