Note: This is test shard 8 of 8.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RollingRestartArgs/RollingRestartITest
[ RUN ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3
2025-06-27T13:54:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-27T13:54:17Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250627 13:54:17.513695 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.140.190:35061
--webserver_interface=127.1.140.190
--webserver_port=0
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.140.190:35061
--location_mapping_cmd=/tmp/dist-test-taskbElLHZ/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/location-assignment.state --map /L0:3 --map /L1:4 --map /L2:4
--master_client_location_assignment_enabled=false with env {}
W20250627 13:54:17.795264 1594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:17.795575 1594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:17.795710 1594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:17.802809 1594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250627 13:54:17.802884 1594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:17.802932 1594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250627 13:54:17.802975 1594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250627 13:54:17.811106 1594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/wal
--location_mapping_cmd=/tmp/dist-test-taskbElLHZ/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/location-assignment.state --map /L0:3 --map /L1:4 --map /L2:4
--ipki_ca_key_size=768
--master_addresses=127.1.140.190:35061
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.140.190:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.140.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:17.812383 1594 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:17.814054 1594 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:17.823472 1602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:17.823468 1599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:17.823503 1600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:17.824383 1594 server_base.cc:1048] running on GCE node
I20250627 13:54:17.824960 1594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:17.826153 1594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:17.827334 1594 hybrid_clock.cc:648] HybridClock initialized: now 1751032457827263 us; error 63 us; skew 500 ppm
I20250627 13:54:17.827628 1594 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:17.836257 1594 webserver.cc:469] Webserver started at http://127.1.140.190:36463/ using document root <none> and password file <none>
I20250627 13:54:17.836913 1594 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:17.836997 1594 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:17.837344 1594 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:54:17.839498 1594 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/data/instance:
uuid: "c49acafbbcb74268bd768406057f1bce"
format_stamp: "Formatted at 2025-06-27 13:54:17 on dist-test-slave-lj78"
I20250627 13:54:17.840169 1594 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/wal/instance:
uuid: "c49acafbbcb74268bd768406057f1bce"
format_stamp: "Formatted at 2025-06-27 13:54:17 on dist-test-slave-lj78"
I20250627 13:54:17.844635 1594 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250627 13:54:17.847661 1608 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:17.848846 1594 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250627 13:54:17.848995 1594 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/wal
uuid: "c49acafbbcb74268bd768406057f1bce"
format_stamp: "Formatted at 2025-06-27 13:54:17 on dist-test-slave-lj78"
I20250627 13:54:17.849174 1594 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:17.862066 1594 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:17.862736 1594 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:17.862977 1594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:17.881260 1594 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.190:35061
I20250627 13:54:17.881285 1659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.190:35061 every 8 connection(s)
I20250627 13:54:17.882468 1594 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/data/info.pb
I20250627 13:54:17.885649 1660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:17.887151 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 1594
I20250627 13:54:17.887423 1586 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/wal/instance
I20250627 13:54:17.896493 1660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce: Bootstrap starting.
I20250627 13:54:17.899111 1660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce: Neither blocks nor log segments found. Creating new log.
I20250627 13:54:17.899993 1660 log.cc:826] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce: Log is configured to *not* fsync() on all Append() calls
I20250627 13:54:17.902192 1660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce: No bootstrap required, opened a new log
I20250627 13:54:17.906286 1660 raft_consensus.cc:357] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49acafbbcb74268bd768406057f1bce" member_type: VOTER last_known_addr { host: "127.1.140.190" port: 35061 } }
I20250627 13:54:17.906600 1660 raft_consensus.cc:383] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:54:17.906715 1660 raft_consensus.cc:738] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c49acafbbcb74268bd768406057f1bce, State: Initialized, Role: FOLLOWER
I20250627 13:54:17.907167 1660 consensus_queue.cc:260] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49acafbbcb74268bd768406057f1bce" member_type: VOTER last_known_addr { host: "127.1.140.190" port: 35061 } }
I20250627 13:54:17.907364 1660 raft_consensus.cc:397] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250627 13:54:17.907460 1660 raft_consensus.cc:491] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250627 13:54:17.907636 1660 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:54:17.909991 1660 raft_consensus.cc:513] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49acafbbcb74268bd768406057f1bce" member_type: VOTER last_known_addr { host: "127.1.140.190" port: 35061 } }
I20250627 13:54:17.910466 1660 leader_election.cc:304] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c49acafbbcb74268bd768406057f1bce; no voters:
I20250627 13:54:17.911000 1660 leader_election.cc:290] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [CANDIDATE]: Term 1 election: Requested vote from peers
I20250627 13:54:17.911129 1665 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:54:17.912636 1665 raft_consensus.cc:695] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [term 1 LEADER]: Becoming Leader. State: Replica: c49acafbbcb74268bd768406057f1bce, State: Running, Role: LEADER
I20250627 13:54:17.913054 1665 consensus_queue.cc:237] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49acafbbcb74268bd768406057f1bce" member_type: VOTER last_known_addr { host: "127.1.140.190" port: 35061 } }
I20250627 13:54:17.913192 1660 sys_catalog.cc:564] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [sys.catalog]: configured and running, proceeding with master startup.
I20250627 13:54:17.915947 1666 sys_catalog.cc:455] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c49acafbbcb74268bd768406057f1bce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49acafbbcb74268bd768406057f1bce" member_type: VOTER last_known_addr { host: "127.1.140.190" port: 35061 } } }
I20250627 13:54:17.916002 1667 sys_catalog.cc:455] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [sys.catalog]: SysCatalogTable state changed. Reason: New leader c49acafbbcb74268bd768406057f1bce. Latest consensus state: current_term: 1 leader_uuid: "c49acafbbcb74268bd768406057f1bce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49acafbbcb74268bd768406057f1bce" member_type: VOTER last_known_addr { host: "127.1.140.190" port: 35061 } } }
I20250627 13:54:17.916265 1667 sys_catalog.cc:458] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [sys.catalog]: This master's current role is: LEADER
I20250627 13:54:17.916265 1666 sys_catalog.cc:458] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce [sys.catalog]: This master's current role is: LEADER
I20250627 13:54:17.917587 1674 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250627 13:54:17.921202 1674 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250627 13:54:17.939060 1674 catalog_manager.cc:1349] Generated new cluster ID: f6e7ed4c58734bde9253bbd2abfa6eed
I20250627 13:54:17.939172 1674 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250627 13:54:17.962057 1674 catalog_manager.cc:1372] Generated new certificate authority record
I20250627 13:54:17.963021 1674 catalog_manager.cc:1506] Loading token signing keys...
I20250627 13:54:17.973477 1674 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c49acafbbcb74268bd768406057f1bce: Generated new TSK 0
I20250627 13:54:17.974152 1674 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250627 13:54:17.982410 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.129:0
--local_ip_for_outbound_sockets=127.1.140.129
--webserver_interface=127.1.140.129
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:54:18.272238 1684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:18.272503 1684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:18.272653 1684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:18.280001 1684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:18.280186 1684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.129
I20250627 13:54:18.288517 1684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.140.129
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.129
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:18.289907 1684 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:18.291502 1684 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:18.306525 1689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:18.306485 1692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:18.306448 1690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:18.307487 1684 server_base.cc:1048] running on GCE node
I20250627 13:54:18.308041 1684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:18.308867 1684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:18.310053 1684 hybrid_clock.cc:648] HybridClock initialized: now 1751032458310014 us; error 36 us; skew 500 ppm
I20250627 13:54:18.310309 1684 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:18.312219 1684 webserver.cc:469] Webserver started at http://127.1.140.129:42277/ using document root <none> and password file <none>
I20250627 13:54:18.312850 1684 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:18.312944 1684 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:18.313292 1684 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:54:18.315462 1684 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data/instance:
uuid: "978726a17dd44a5891b685d3b2ae2b99"
format_stamp: "Formatted at 2025-06-27 13:54:18 on dist-test-slave-lj78"
I20250627 13:54:18.316223 1684 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/wal/instance:
uuid: "978726a17dd44a5891b685d3b2ae2b99"
format_stamp: "Formatted at 2025-06-27 13:54:18 on dist-test-slave-lj78"
I20250627 13:54:18.321048 1684 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20250627 13:54:18.323932 1698 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:18.325032 1684 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250627 13:54:18.325169 1684 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/wal
uuid: "978726a17dd44a5891b685d3b2ae2b99"
format_stamp: "Formatted at 2025-06-27 13:54:18 on dist-test-slave-lj78"
I20250627 13:54:18.325373 1684 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:18.340977 1684 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:18.341634 1684 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:18.341852 1684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:18.342602 1684 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:18.343852 1684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:18.343937 1684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:18.344039 1684 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:18.344103 1684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:18.374135 1684 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.129:35871
I20250627 13:54:18.374202 1810 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.129:35871 every 8 connection(s)
I20250627 13:54:18.375284 1684 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data/info.pb
I20250627 13:54:18.376839 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 1684
I20250627 13:54:18.377023 1586 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/wal/instance
I20250627 13:54:18.380381 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.130:0
--local_ip_for_outbound_sockets=127.1.140.130
--webserver_interface=127.1.140.130
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:54:18.388586 1811 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:54:18.388950 1811 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:18.389992 1811 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:18.444653 1625 ts_manager.cc:194] Registered new tserver with Master: 978726a17dd44a5891b685d3b2ae2b99 (127.1.140.129:35871)
I20250627 13:54:18.446295 1625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.129:60641
W20250627 13:54:18.683892 1814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:18.684144 1814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:18.684262 1814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:18.691465 1814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:18.691596 1814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.130
I20250627 13:54:18.700237 1814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.140.130
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.130
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:18.701602 1814 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:18.703233 1814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:18.713697 1821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:18.713816 1823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:18.713668 1820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:18.714666 1814 server_base.cc:1048] running on GCE node
I20250627 13:54:18.715147 1814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:18.715987 1814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:18.717181 1814 hybrid_clock.cc:648] HybridClock initialized: now 1751032458717127 us; error 43 us; skew 500 ppm
I20250627 13:54:18.717448 1814 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:18.719401 1814 webserver.cc:469] Webserver started at http://127.1.140.130:46771/ using document root <none> and password file <none>
I20250627 13:54:18.720044 1814 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:18.720119 1814 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:18.720381 1814 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:54:18.723035 1814 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data/instance:
uuid: "0204b303a14a41b096bfddffc4d04290"
format_stamp: "Formatted at 2025-06-27 13:54:18 on dist-test-slave-lj78"
I20250627 13:54:18.723696 1814 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal/instance:
uuid: "0204b303a14a41b096bfddffc4d04290"
format_stamp: "Formatted at 2025-06-27 13:54:18 on dist-test-slave-lj78"
I20250627 13:54:18.728183 1814 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250627 13:54:18.731135 1829 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:18.732208 1814 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250627 13:54:18.732321 1814 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal
uuid: "0204b303a14a41b096bfddffc4d04290"
format_stamp: "Formatted at 2025-06-27 13:54:18 on dist-test-slave-lj78"
I20250627 13:54:18.732470 1814 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:18.754590 1814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:18.755209 1814 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:18.755414 1814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:18.756188 1814 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:18.757534 1814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:18.757622 1814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:18.757690 1814 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:18.757740 1814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:18.789214 1814 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.130:42207
I20250627 13:54:18.789321 1941 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.130:42207 every 8 connection(s)
I20250627 13:54:18.790472 1814 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data/info.pb
I20250627 13:54:18.796834 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 1814
I20250627 13:54:18.796976 1586 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal/instance
I20250627 13:54:18.799872 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.131:0
--local_ip_for_outbound_sockets=127.1.140.131
--webserver_interface=127.1.140.131
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:54:18.802831 1942 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:54:18.803128 1942 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:18.803797 1942 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:18.842859 1625 ts_manager.cc:194] Registered new tserver with Master: 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207)
I20250627 13:54:18.843663 1625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.130:52883
W20250627 13:54:19.095386 1945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:19.095741 1945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:19.095880 1945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:19.102905 1945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:19.103075 1945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.131
I20250627 13:54:19.111649 1945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.140.131
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.131
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:19.113014 1945 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:19.114706 1945 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:19.126291 1952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:19.126349 1951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:19.126328 1954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:19.127333 1945 server_base.cc:1048] running on GCE node
I20250627 13:54:19.127823 1945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:19.128618 1945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:19.129812 1945 hybrid_clock.cc:648] HybridClock initialized: now 1751032459129765 us; error 41 us; skew 500 ppm
I20250627 13:54:19.130059 1945 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:19.131944 1945 webserver.cc:469] Webserver started at http://127.1.140.131:41547/ using document root <none> and password file <none>
I20250627 13:54:19.132599 1945 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:19.132695 1945 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:19.132998 1945 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:54:19.135167 1945 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data/instance:
uuid: "e879d1cb3a7c4226884531badda84252"
format_stamp: "Formatted at 2025-06-27 13:54:19 on dist-test-slave-lj78"
I20250627 13:54:19.135869 1945 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/wal/instance:
uuid: "e879d1cb3a7c4226884531badda84252"
format_stamp: "Formatted at 2025-06-27 13:54:19 on dist-test-slave-lj78"
I20250627 13:54:19.140439 1945 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250627 13:54:19.143604 1960 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:19.144652 1945 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250627 13:54:19.144781 1945 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/wal
uuid: "e879d1cb3a7c4226884531badda84252"
format_stamp: "Formatted at 2025-06-27 13:54:19 on dist-test-slave-lj78"
I20250627 13:54:19.144938 1945 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:19.167593 1945 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:19.168237 1945 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:19.168488 1945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:19.169814 1945 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:19.171145 1945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:19.171233 1945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:19.171330 1945 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:19.171394 1945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:19.204718 1945 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.131:33589
I20250627 13:54:19.204782 2072 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.131:33589 every 8 connection(s)
I20250627 13:54:19.206027 1945 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data/info.pb
I20250627 13:54:19.214607 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 1945
I20250627 13:54:19.214761 1586 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/wal/instance
I20250627 13:54:19.218065 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.132:0
--local_ip_for_outbound_sockets=127.1.140.132
--webserver_interface=127.1.140.132
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:54:19.218161 2073 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:54:19.218503 2073 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:19.219477 2073 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:19.259434 1625 ts_manager.cc:194] Registered new tserver with Master: e879d1cb3a7c4226884531badda84252 (127.1.140.131:33589)
I20250627 13:54:19.260349 1625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.131:60495
I20250627 13:54:19.449743 1811 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
W20250627 13:54:19.515240 2076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:19.515534 2076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:19.515697 2076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:19.522984 2076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:19.523152 2076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.132
I20250627 13:54:19.532090 2076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.1.140.132
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.132
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:19.533483 2076 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:19.535076 2076 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:19.546181 2085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:19.546165 2083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:19.546324 2082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:19.546885 2076 server_base.cc:1048] running on GCE node
I20250627 13:54:19.547338 2076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:19.548067 2076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:19.549273 2076 hybrid_clock.cc:648] HybridClock initialized: now 1751032459549202 us; error 41 us; skew 500 ppm
I20250627 13:54:19.549563 2076 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:19.551677 2076 webserver.cc:469] Webserver started at http://127.1.140.132:45491/ using document root <none> and password file <none>
I20250627 13:54:19.552281 2076 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:19.552373 2076 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:19.552682 2076 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:54:19.554898 2076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data/instance:
uuid: "99bf3fe9a41d4ce1b61e440b31778689"
format_stamp: "Formatted at 2025-06-27 13:54:19 on dist-test-slave-lj78"
I20250627 13:54:19.555615 2076 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/wal/instance:
uuid: "99bf3fe9a41d4ce1b61e440b31778689"
format_stamp: "Formatted at 2025-06-27 13:54:19 on dist-test-slave-lj78"
I20250627 13:54:19.560132 2076 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250627 13:54:19.563004 2091 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:19.564086 2076 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250627 13:54:19.564672 2076 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/wal
uuid: "99bf3fe9a41d4ce1b61e440b31778689"
format_stamp: "Formatted at 2025-06-27 13:54:19 on dist-test-slave-lj78"
I20250627 13:54:19.564842 2076 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:19.590703 2076 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:19.591851 2076 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:19.592073 2076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:19.592860 2076 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:19.594466 2076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:19.594568 2076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:19.594666 2076 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:19.594743 2076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:19.627909 2076 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.132:44007
I20250627 13:54:19.627978 2203 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.132:44007 every 8 connection(s)
I20250627 13:54:19.629113 2076 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data/info.pb
I20250627 13:54:19.635587 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 2076
I20250627 13:54:19.635756 1586 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/wal/instance
I20250627 13:54:19.638552 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.133:0
--local_ip_for_outbound_sockets=127.1.140.133
--webserver_interface=127.1.140.133
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:54:19.641992 2204 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:54:19.642313 2204 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:19.643085 2204 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:19.682780 1625 ts_manager.cc:194] Registered new tserver with Master: 99bf3fe9a41d4ce1b61e440b31778689 (127.1.140.132:44007)
I20250627 13:54:19.684082 1625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.132:44427
I20250627 13:54:19.846174 1942 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
W20250627 13:54:19.934113 2207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:19.934345 2207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:19.934423 2207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:19.941159 2207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:19.941368 2207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.133
I20250627 13:54:19.949939 2207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data/info.pb
--webserver_interface=127.1.140.133
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.133
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:19.951123 2207 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:19.952616 2207 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:19.963008 2216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:19.962934 2214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:19.962970 2213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:19.963913 2207 server_base.cc:1048] running on GCE node
I20250627 13:54:19.964325 2207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:19.965075 2207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:19.966244 2207 hybrid_clock.cc:648] HybridClock initialized: now 1751032459966203 us; error 40 us; skew 500 ppm
I20250627 13:54:19.966476 2207 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:19.968278 2207 webserver.cc:469] Webserver started at http://127.1.140.133:43649/ using document root <none> and password file <none>
I20250627 13:54:19.968979 2207 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:19.969070 2207 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:19.969430 2207 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:54:19.971441 2207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data/instance:
uuid: "ffa5b64156aa4e0dbef89b3fb8111e44"
format_stamp: "Formatted at 2025-06-27 13:54:19 on dist-test-slave-lj78"
I20250627 13:54:19.972107 2207 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/wal/instance:
uuid: "ffa5b64156aa4e0dbef89b3fb8111e44"
format_stamp: "Formatted at 2025-06-27 13:54:19 on dist-test-slave-lj78"
I20250627 13:54:19.976445 2207 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250627 13:54:19.979327 2222 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:19.980463 2207 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250627 13:54:19.980602 2207 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/wal
uuid: "ffa5b64156aa4e0dbef89b3fb8111e44"
format_stamp: "Formatted at 2025-06-27 13:54:19 on dist-test-slave-lj78"
I20250627 13:54:19.980767 2207 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:20.021369 2207 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:20.022038 2207 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:20.022254 2207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:20.023039 2207 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:20.024348 2207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:20.024436 2207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:20.024539 2207 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:20.024610 2207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:20.055989 2207 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.133:44765
I20250627 13:54:20.056044 2334 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.133:44765 every 8 connection(s)
I20250627 13:54:20.057256 2207 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data/info.pb
I20250627 13:54:20.065445 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 2207
I20250627 13:54:20.065649 1586 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/wal/instance
I20250627 13:54:20.069011 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.134:0
--local_ip_for_outbound_sockets=127.1.140.134
--webserver_interface=127.1.140.134
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:54:20.069865 2335 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:54:20.070181 2335 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:20.071159 2335 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:20.111474 1625 ts_manager.cc:194] Registered new tserver with Master: ffa5b64156aa4e0dbef89b3fb8111e44 (127.1.140.133:44765)
I20250627 13:54:20.112978 1625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.133:39145
I20250627 13:54:20.262838 2073 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
W20250627 13:54:20.367655 2338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:20.367928 2338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:20.368005 2338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:20.376201 2338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:20.376433 2338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.134
I20250627 13:54:20.385761 2338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data/info.pb
--webserver_interface=127.1.140.134
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.134
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:20.387111 2338 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:20.388759 2338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:20.399840 2347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:20.399818 2345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:20.399771 2344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:20.400439 2338 server_base.cc:1048] running on GCE node
I20250627 13:54:20.400943 2338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:20.401775 2338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:20.402951 2338 hybrid_clock.cc:648] HybridClock initialized: now 1751032460402915 us; error 28 us; skew 500 ppm
I20250627 13:54:20.403218 2338 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:20.405156 2338 webserver.cc:469] Webserver started at http://127.1.140.134:37575/ using document root <none> and password file <none>
I20250627 13:54:20.405827 2338 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:20.405915 2338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:20.406229 2338 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:54:20.408272 2338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data/instance:
uuid: "aa90f1d263644537b32ad1a4595aef92"
format_stamp: "Formatted at 2025-06-27 13:54:20 on dist-test-slave-lj78"
I20250627 13:54:20.409475 2338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/wal/instance:
uuid: "aa90f1d263644537b32ad1a4595aef92"
format_stamp: "Formatted at 2025-06-27 13:54:20 on dist-test-slave-lj78"
I20250627 13:54:20.414045 2338 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20250627 13:54:20.416996 2353 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:20.418170 2338 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250627 13:54:20.418303 2338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/wal
uuid: "aa90f1d263644537b32ad1a4595aef92"
format_stamp: "Formatted at 2025-06-27 13:54:20 on dist-test-slave-lj78"
I20250627 13:54:20.418493 2338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:20.440788 2338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:20.441450 2338 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:20.441664 2338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:20.442415 2338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:20.443727 2338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:20.443805 2338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:20.443913 2338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:20.443979 2338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:20.475768 2338 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.134:36295
I20250627 13:54:20.475834 2465 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.134:36295 every 8 connection(s)
I20250627 13:54:20.476992 2338 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data/info.pb
I20250627 13:54:20.486127 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 2338
I20250627 13:54:20.486256 1586 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/wal/instance
I20250627 13:54:20.489315 2466 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:54:20.489593 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.135:0
--local_ip_for_outbound_sockets=127.1.140.135
--webserver_interface=127.1.140.135
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:54:20.489636 2466 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:20.490376 2466 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:20.529603 1625 ts_manager.cc:194] Registered new tserver with Master: aa90f1d263644537b32ad1a4595aef92 (127.1.140.134:36295)
I20250627 13:54:20.530562 1625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.134:44527
I20250627 13:54:20.686729 2204 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
W20250627 13:54:20.786167 2469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:20.786408 2469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:20.786536 2469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:20.794756 2469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:20.794932 2469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.135
I20250627 13:54:20.803378 2469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data/info.pb
--webserver_interface=127.1.140.135
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.135
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:20.804734 2469 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:20.806396 2469 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:20.817045 2475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:20.817119 2476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:20.817153 2478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:20.817711 2469 server_base.cc:1048] running on GCE node
I20250627 13:54:20.818265 2469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:20.819041 2469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:20.820242 2469 hybrid_clock.cc:648] HybridClock initialized: now 1751032460820209 us; error 30 us; skew 500 ppm
I20250627 13:54:20.820470 2469 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:20.822474 2469 webserver.cc:469] Webserver started at http://127.1.140.135:45365/ using document root <none> and password file <none>
I20250627 13:54:20.823092 2469 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:20.823165 2469 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:20.823414 2469 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:54:20.825414 2469 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data/instance:
uuid: "373a9aec2682487a8ff2a4f0d1ee87ab"
format_stamp: "Formatted at 2025-06-27 13:54:20 on dist-test-slave-lj78"
I20250627 13:54:20.826067 2469 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/wal/instance:
uuid: "373a9aec2682487a8ff2a4f0d1ee87ab"
format_stamp: "Formatted at 2025-06-27 13:54:20 on dist-test-slave-lj78"
I20250627 13:54:20.830773 2469 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250627 13:54:20.833832 2484 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:20.834908 2469 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250627 13:54:20.835018 2469 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/wal
uuid: "373a9aec2682487a8ff2a4f0d1ee87ab"
format_stamp: "Formatted at 2025-06-27 13:54:20 on dist-test-slave-lj78"
I20250627 13:54:20.835160 2469 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:20.850922 2469 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:20.851545 2469 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:20.851733 2469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:20.852464 2469 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:20.853806 2469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:20.853885 2469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:20.853952 2469 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:20.854001 2469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:20.886109 2469 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.135:34009
I20250627 13:54:20.886147 2596 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.135:34009 every 8 connection(s)
I20250627 13:54:20.887225 2469 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data/info.pb
I20250627 13:54:20.895660 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 2469
I20250627 13:54:20.895824 1586 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/wal/instance
I20250627 13:54:20.898777 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.136:0
--local_ip_for_outbound_sockets=127.1.140.136
--webserver_interface=127.1.140.136
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:54:20.899194 2597 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:54:20.899498 2597 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:20.900209 2597 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:20.966905 1625 ts_manager.cc:194] Registered new tserver with Master: 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009)
I20250627 13:54:20.968097 1625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.135:52699
I20250627 13:54:21.115753 2335 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
W20250627 13:54:21.204355 2600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:21.204640 2600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:21.204741 2600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:21.212116 2600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:21.212248 2600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.136
I20250627 13:54:21.221089 2600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data/info.pb
--webserver_interface=127.1.140.136
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.136
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:21.222470 2600 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:21.224032 2600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:21.235760 2609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:21.235721 2606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:21.235734 2607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:21.237339 2600 server_base.cc:1048] running on GCE node
I20250627 13:54:21.237769 2600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:21.238545 2600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:21.239745 2600 hybrid_clock.cc:648] HybridClock initialized: now 1751032461239699 us; error 45 us; skew 500 ppm
I20250627 13:54:21.239997 2600 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:21.241956 2600 webserver.cc:469] Webserver started at http://127.1.140.136:33675/ using document root <none> and password file <none>
I20250627 13:54:21.242576 2600 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:21.242666 2600 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:21.242961 2600 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:54:21.244954 2600 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data/instance:
uuid: "f9d895c7b77646baaff4f9a677e8c69d"
format_stamp: "Formatted at 2025-06-27 13:54:21 on dist-test-slave-lj78"
I20250627 13:54:21.245615 2600 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/wal/instance:
uuid: "f9d895c7b77646baaff4f9a677e8c69d"
format_stamp: "Formatted at 2025-06-27 13:54:21 on dist-test-slave-lj78"
I20250627 13:54:21.250336 2600 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250627 13:54:21.253620 2615 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:21.254846 2600 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250627 13:54:21.254976 2600 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/wal
uuid: "f9d895c7b77646baaff4f9a677e8c69d"
format_stamp: "Formatted at 2025-06-27 13:54:21 on dist-test-slave-lj78"
I20250627 13:54:21.255123 2600 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:21.278293 2600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:21.279017 2600 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:21.279247 2600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:21.280536 2600 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:21.281971 2600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:21.282055 2600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:21.282150 2600 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:21.282212 2600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:21.315485 2600 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.136:39253
I20250627 13:54:21.315565 2727 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.136:39253 every 8 connection(s)
I20250627 13:54:21.316852 2600 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data/info.pb
I20250627 13:54:21.323810 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 2600
I20250627 13:54:21.323976 1586 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/wal/instance
I20250627 13:54:21.327050 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.137:0
--local_ip_for_outbound_sockets=127.1.140.137
--webserver_interface=127.1.140.137
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:54:21.328586 2728 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:54:21.329034 2728 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:21.329999 2728 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:21.380779 1625 ts_manager.cc:194] Registered new tserver with Master: f9d895c7b77646baaff4f9a677e8c69d (127.1.140.136:39253)
I20250627 13:54:21.381757 1625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.136:33547
I20250627 13:54:21.533145 2466 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
W20250627 13:54:21.637246 2731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:21.637503 2731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:21.637632 2731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:21.644678 2731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:21.644847 2731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.137
I20250627 13:54:21.652556 2731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.137:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data/info.pb
--webserver_interface=127.1.140.137
--webserver_port=0
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.137
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:21.653875 2731 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:21.655444 2731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:21.666097 2738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:21.666158 2737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:21.666131 2740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:21.666707 2731 server_base.cc:1048] running on GCE node
I20250627 13:54:21.667204 2731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:21.667946 2731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:21.669160 2731 hybrid_clock.cc:648] HybridClock initialized: now 1751032461669107 us; error 44 us; skew 500 ppm
I20250627 13:54:21.669437 2731 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:21.671345 2731 webserver.cc:469] Webserver started at http://127.1.140.137:34499/ using document root <none> and password file <none>
I20250627 13:54:21.672012 2731 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:21.672109 2731 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:21.672438 2731 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250627 13:54:21.674538 2731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data/instance:
uuid: "5f1fab906378491e919a4a3364723428"
format_stamp: "Formatted at 2025-06-27 13:54:21 on dist-test-slave-lj78"
I20250627 13:54:21.675205 2731 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/wal/instance:
uuid: "5f1fab906378491e919a4a3364723428"
format_stamp: "Formatted at 2025-06-27 13:54:21 on dist-test-slave-lj78"
I20250627 13:54:21.679828 2731 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250627 13:54:21.682513 2746 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:21.683561 2731 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250627 13:54:21.683681 2731 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/wal
uuid: "5f1fab906378491e919a4a3364723428"
format_stamp: "Formatted at 2025-06-27 13:54:21 on dist-test-slave-lj78"
I20250627 13:54:21.683827 2731 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:21.709617 2731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:21.710299 2731 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:21.710505 2731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:21.711292 2731 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:21.712599 2731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:21.712687 2731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:21.712788 2731 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:21.713342 2731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:21.745987 2731 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.137:40611
I20250627 13:54:21.746052 2858 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.137:40611 every 8 connection(s)
I20250627 13:54:21.747279 2731 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data/info.pb
I20250627 13:54:21.754057 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 2731
I20250627 13:54:21.754223 1586 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/wal/instance
I20250627 13:54:21.759481 2859 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:54:21.759796 2859 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:21.760506 2859 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:21.802711 1625 ts_manager.cc:194] Registered new tserver with Master: 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611)
I20250627 13:54:21.803529 1625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.137:44979
I20250627 13:54:21.808632 1586 external_mini_cluster.cc:934] 9 TS(s) registered with all masters
I20250627 13:54:21.826692 1586 test_util.cc:276] Using random seed: 65051238
I20250627 13:54:21.843819 1625 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43524:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250627 13:54:21.892650 1877 tablet_service.cc:1468] Processing CreateTablet for tablet 8dca13501f184a25a621c700a1939986 (DEFAULT_TABLE table=test-workload [id=34125e5afeac4f5db4cb16ab0f7494f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:54:21.893673 2794 tablet_service.cc:1468] Processing CreateTablet for tablet 8dca13501f184a25a621c700a1939986 (DEFAULT_TABLE table=test-workload [id=34125e5afeac4f5db4cb16ab0f7494f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:54:21.894112 1877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dca13501f184a25a621c700a1939986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:21.894778 2794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dca13501f184a25a621c700a1939986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:21.894697 2532 tablet_service.cc:1468] Processing CreateTablet for tablet 8dca13501f184a25a621c700a1939986 (DEFAULT_TABLE table=test-workload [id=34125e5afeac4f5db4cb16ab0f7494f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250627 13:54:21.895831 2532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dca13501f184a25a621c700a1939986. 1 dirs total, 0 dirs full, 0 dirs failed
I20250627 13:54:21.906383 2883 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Bootstrap starting.
I20250627 13:54:21.908747 2884 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap starting.
I20250627 13:54:21.910271 2883 tablet_bootstrap.cc:654] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Neither blocks nor log segments found. Creating new log.
I20250627 13:54:21.911283 2883 log.cc:826] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Log is configured to *not* fsync() on all Append() calls
I20250627 13:54:21.911953 2884 tablet_bootstrap.cc:654] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Neither blocks nor log segments found. Creating new log.
I20250627 13:54:21.912027 2885 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Bootstrap starting.
I20250627 13:54:21.913069 2884 log.cc:826] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Log is configured to *not* fsync() on all Append() calls
I20250627 13:54:21.914923 2885 tablet_bootstrap.cc:654] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Neither blocks nor log segments found. Creating new log.
I20250627 13:54:21.914952 2883 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: No bootstrap required, opened a new log
I20250627 13:54:21.915200 2883 ts_tablet_manager.cc:1397] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20250627 13:54:21.915719 2884 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: No bootstrap required, opened a new log
I20250627 13:54:21.915927 2884 ts_tablet_manager.cc:1397] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20250627 13:54:21.916064 2885 log.cc:826] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Log is configured to *not* fsync() on all Append() calls
I20250627 13:54:21.918777 2885 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: No bootstrap required, opened a new log
I20250627 13:54:21.919023 2885 ts_tablet_manager.cc:1397] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20250627 13:54:21.921927 2883 raft_consensus.cc:357] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:54:21.922225 2883 raft_consensus.cc:383] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:54:21.922343 2883 raft_consensus.cc:738] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0204b303a14a41b096bfddffc4d04290, State: Initialized, Role: FOLLOWER
I20250627 13:54:21.922410 2884 raft_consensus.cc:357] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:54:21.922686 2884 raft_consensus.cc:383] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:54:21.922828 2884 raft_consensus.cc:738] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1fab906378491e919a4a3364723428, State: Initialized, Role: FOLLOWER
I20250627 13:54:21.923235 2883 consensus_queue.cc:260] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:54:21.923389 2884 consensus_queue.cc:260] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:54:21.923735 2889 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250627 13:54:21.924072 2889 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:54:21.925488 2885 raft_consensus.cc:357] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:54:21.925925 2885 raft_consensus.cc:383] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250627 13:54:21.926090 2885 raft_consensus.cc:738] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 373a9aec2682487a8ff2a4f0d1ee87ab, State: Initialized, Role: FOLLOWER
I20250627 13:54:21.927304 2885 consensus_queue.cc:260] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:54:21.930907 2885 ts_tablet_manager.cc:1428] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20250627 13:54:21.931291 2597 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
I20250627 13:54:21.933925 2859 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
I20250627 13:54:21.934386 2884 ts_tablet_manager.cc:1428] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Time spent starting tablet: real 0.018s user 0.007s sys 0.011s
I20250627 13:54:21.935556 2889 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611), 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009)
I20250627 13:54:21.940213 2883 ts_tablet_manager.cc:1428] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Time spent starting tablet: real 0.025s user 0.011s sys 0.009s
I20250627 13:54:21.945880 2814 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "0204b303a14a41b096bfddffc4d04290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1fab906378491e919a4a3364723428" is_pre_election: true
I20250627 13:54:21.946305 2814 raft_consensus.cc:2466] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0204b303a14a41b096bfddffc4d04290 in term 0.
I20250627 13:54:21.946820 1831 leader_election.cc:304] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0204b303a14a41b096bfddffc4d04290, 5f1fab906378491e919a4a3364723428; no voters:
I20250627 13:54:21.947096 2889 raft_consensus.cc:2802] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250627 13:54:21.947045 2552 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "0204b303a14a41b096bfddffc4d04290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" is_pre_election: true
I20250627 13:54:21.947222 2889 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250627 13:54:21.947320 2889 raft_consensus.cc:3058] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:54:21.947443 2552 raft_consensus.cc:2466] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0204b303a14a41b096bfddffc4d04290 in term 0.
I20250627 13:54:21.949519 2889 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:54:21.949996 2889 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [CANDIDATE]: Term 1 election: Requested vote from peers 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611), 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009)
I20250627 13:54:21.950290 2814 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "0204b303a14a41b096bfddffc4d04290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1fab906378491e919a4a3364723428"
I20250627 13:54:21.950371 2552 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "0204b303a14a41b096bfddffc4d04290" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab"
I20250627 13:54:21.950425 2814 raft_consensus.cc:3058] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:54:21.950512 2552 raft_consensus.cc:3058] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 0 FOLLOWER]: Advancing to term 1
I20250627 13:54:21.952173 2814 raft_consensus.cc:2466] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0204b303a14a41b096bfddffc4d04290 in term 1.
I20250627 13:54:21.952450 1831 leader_election.cc:304] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0204b303a14a41b096bfddffc4d04290, 5f1fab906378491e919a4a3364723428; no voters:
I20250627 13:54:21.952711 2889 raft_consensus.cc:2802] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 1 FOLLOWER]: Leader election won for term 1
I20250627 13:54:21.952883 2889 raft_consensus.cc:695] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 1 LEADER]: Becoming Leader. State: Replica: 0204b303a14a41b096bfddffc4d04290, State: Running, Role: LEADER
I20250627 13:54:21.953011 2552 raft_consensus.cc:2466] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0204b303a14a41b096bfddffc4d04290 in term 1.
I20250627 13:54:21.953262 2889 consensus_queue.cc:237] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:54:21.956980 1624 catalog_manager.cc:5582] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0204b303a14a41b096bfddffc4d04290 (127.1.140.130). New cstate: current_term: 1 leader_uuid: "0204b303a14a41b096bfddffc4d04290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } health_report { overall_health: HEALTHY } } }
I20250627 13:54:21.971105 1586 maintenance_mode-itest.cc:745] Restarting batch of 3 tservers: aa90f1d263644537b32ad1a4595aef92,ffa5b64156aa4e0dbef89b3fb8111e44,99bf3fe9a41d4ce1b61e440b31778689
W20250627 13:54:21.998776 2860 tablet.cc:2378] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250627 13:54:22.044149 1943 tablet.cc:2378] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250627 13:54:22.071496 2889 consensus_queue.cc:1035] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250627 13:54:22.078922 2814 raft_consensus.cc:1273] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 1 FOLLOWER]: Refusing update from remote peer 0204b303a14a41b096bfddffc4d04290: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250627 13:54:22.080447 2889 consensus_queue.cc:1035] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250627 13:54:22.129628 2909 mvcc.cc:204] Tried to move back new op lower bound from 7172228964710531072 to 7172228964164022272. Current Snapshot: MvccSnapshot[applied={T|T < 7172228964649586688}]
W20250627 13:54:22.139808 2774 tablet_service.cc:2912] Rejecting scan request for tablet 8dca13501f184a25a621c700a1939986: Uninitialized: clean time has not yet been initialized
W20250627 13:54:22.141109 2773 tablet_service.cc:2912] Rejecting scan request for tablet 8dca13501f184a25a621c700a1939986: Uninitialized: clean time has not yet been initialized
W20250627 13:54:22.141873 2772 tablet_service.cc:2912] Rejecting scan request for tablet 8dca13501f184a25a621c700a1939986: Uninitialized: clean time has not yet been initialized
I20250627 13:54:22.384135 2728 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
I20250627 13:54:23.390035 1618 ts_manager.cc:295] Set tserver state for aa90f1d263644537b32ad1a4595aef92 to MAINTENANCE_MODE
I20250627 13:54:23.443733 1617 ts_manager.cc:295] Set tserver state for 99bf3fe9a41d4ce1b61e440b31778689 to MAINTENANCE_MODE
I20250627 13:54:23.570830 1617 ts_manager.cc:295] Set tserver state for ffa5b64156aa4e0dbef89b3fb8111e44 to MAINTENANCE_MODE
I20250627 13:54:26.632622 2401 tablet_service.cc:1423] Tablet server aa90f1d263644537b32ad1a4595aef92 set to quiescing
I20250627 13:54:26.632901 2401 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:26.866462 2270 tablet_service.cc:1423] Tablet server ffa5b64156aa4e0dbef89b3fb8111e44 set to quiescing
I20250627 13:54:26.866753 2270 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:27.325053 2139 tablet_service.cc:1423] Tablet server 99bf3fe9a41d4ce1b61e440b31778689 set to quiescing
I20250627 13:54:27.325297 2139 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:28.934479 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 2338
I20250627 13:54:28.960255 2461 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:54:29.133175 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 2338
I20250627 13:54:29.159054 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.134:36295
--local_ip_for_outbound_sockets=127.1.140.134
--tserver_master_addrs=127.1.140.190:35061
--webserver_port=37575
--webserver_interface=127.1.140.134
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:54:29.681782 3029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:29.682261 3029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:29.682464 3029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:29.694373 3029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:29.694648 3029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.134
I20250627 13:54:29.709134 3029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.134:36295
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data/info.pb
--webserver_interface=127.1.140.134
--webserver_port=37575
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.134
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:29.711498 3029 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:29.713932 3029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:29.775489 3038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:29.784724 3040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:29.793601 3037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:29.806905 3029 server_base.cc:1048] running on GCE node
I20250627 13:54:29.807646 3029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:29.808681 3029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:29.811976 3029 hybrid_clock.cc:648] HybridClock initialized: now 1751032469811908 us; error 61 us; skew 500 ppm
I20250627 13:54:29.812330 3029 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:29.875936 3029 webserver.cc:469] Webserver started at http://127.1.140.134:37575/ using document root <none> and password file <none>
I20250627 13:54:29.877260 3029 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:29.877471 3029 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:29.964253 3029 fs_manager.cc:714] Time spent opening directory manager: real 0.083s user 0.019s sys 0.020s
I20250627 13:54:30.022081 3047 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:30.024377 3029 fs_manager.cc:730] Time spent opening block manager: real 0.057s user 0.008s sys 0.029s
I20250627 13:54:30.024678 3029 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/wal
uuid: "aa90f1d263644537b32ad1a4595aef92"
format_stamp: "Formatted at 2025-06-27 13:54:20 on dist-test-slave-lj78"
I20250627 13:54:30.026046 3029 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:30.091912 3029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:30.094329 3029 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:30.094736 3029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:30.108417 3029 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:30.123312 3029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:30.123469 3029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250627 13:54:30.128044 3029 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:30.128175 3029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:31.176352 3029 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.134:36295
I20250627 13:54:31.176399 3162 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.134:36295 every 8 connection(s)
I20250627 13:54:31.178196 3029 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data/info.pb
I20250627 13:54:31.188841 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 3029
I20250627 13:54:31.189074 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 2207
I20250627 13:54:31.254441 2330 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:54:31.269349 3163 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:54:31.269851 3163 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:31.271137 3163 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:31.273249 1617 ts_manager.cc:194] Re-registered known tserver with Master: aa90f1d263644537b32ad1a4595aef92 (127.1.140.134:36295)
I20250627 13:54:31.274333 1617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.134:60317
I20250627 13:54:31.515381 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 2207
I20250627 13:54:31.559859 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.133:44765
--local_ip_for_outbound_sockets=127.1.140.133
--tserver_master_addrs=127.1.140.190:35061
--webserver_port=43649
--webserver_interface=127.1.140.133
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:54:32.277590 3163 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
W20250627 13:54:32.310600 3171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:32.311110 3171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:32.311348 3171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:32.327075 3171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:32.327363 3171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.133
I20250627 13:54:32.342808 3171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.133:44765
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data/info.pb
--webserver_interface=127.1.140.133
--webserver_port=43649
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.133
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:32.345039 3171 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:32.347235 3171 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:32.391877 3181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:32.392498 3180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:32.409360 3184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:32.409808 3171 server_base.cc:1048] running on GCE node
I20250627 13:54:32.410903 3171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:32.412921 3171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:32.417863 3171 hybrid_clock.cc:648] HybridClock initialized: now 1751032472416694 us; error 1145 us; skew 500 ppm
I20250627 13:54:32.418591 3171 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:32.433894 3171 webserver.cc:469] Webserver started at http://127.1.140.133:43649/ using document root <none> and password file <none>
I20250627 13:54:32.435031 3171 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:32.435210 3171 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:32.443055 3171 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20250627 13:54:32.450228 3189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:32.458319 3171 fs_manager.cc:730] Time spent opening block manager: real 0.014s user 0.005s sys 0.000s
I20250627 13:54:32.458528 3171 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/wal
uuid: "ffa5b64156aa4e0dbef89b3fb8111e44"
format_stamp: "Formatted at 2025-06-27 13:54:19 on dist-test-slave-lj78"
I20250627 13:54:32.461163 3171 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:32.539613 3171 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:32.540751 3171 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:32.541191 3171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:32.545174 3171 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:32.547487 3171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:32.547581 3171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:32.547703 3171 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:32.547755 3171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:33.569747 3171 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.133:44765
I20250627 13:54:33.569823 3305 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.133:44765 every 8 connection(s)
I20250627 13:54:33.571812 3171 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data/info.pb
I20250627 13:54:33.583336 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 3171
I20250627 13:54:33.583541 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 2076
I20250627 13:54:33.609546 3306 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:54:33.609954 3306 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:33.611014 3306 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:33.615448 1617 ts_manager.cc:194] Re-registered known tserver with Master: ffa5b64156aa4e0dbef89b3fb8111e44 (127.1.140.133:44765)
I20250627 13:54:33.616453 1617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.133:33195
I20250627 13:54:33.644659 2199 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:54:33.857177 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 2076
I20250627 13:54:33.890882 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.132:44007
--local_ip_for_outbound_sockets=127.1.140.132
--tserver_master_addrs=127.1.140.190:35061
--webserver_port=45491
--webserver_interface=127.1.140.132
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:54:34.462445 3314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:34.462816 3314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:34.462939 3314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:34.474740 3314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:34.475013 3314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.132
I20250627 13:54:34.489378 3314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.132:44007
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.1.140.132
--webserver_port=45491
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.132
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:34.491654 3314 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:34.494032 3314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:34.546361 3320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:34.549564 3322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:34.560698 3314 server_base.cc:1048] running on GCE node
W20250627 13:54:34.571622 3325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:34.572384 3314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:34.576457 3314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:34.594193 3314 hybrid_clock.cc:648] HybridClock initialized: now 1751032474594118 us; error 64 us; skew 500 ppm
I20250627 13:54:34.594573 3314 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:34.619763 3306 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
I20250627 13:54:34.639034 3314 webserver.cc:469] Webserver started at http://127.1.140.132:45491/ using document root <none> and password file <none>
I20250627 13:54:34.640734 3314 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:34.641289 3314 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:34.657025 3314 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.007s sys 0.003s
I20250627 13:54:34.688267 3330 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:34.690594 3314 fs_manager.cc:730] Time spent opening block manager: real 0.031s user 0.023s sys 0.000s
I20250627 13:54:34.690874 3314 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/wal
uuid: "99bf3fe9a41d4ce1b61e440b31778689"
format_stamp: "Formatted at 2025-06-27 13:54:19 on dist-test-slave-lj78"
I20250627 13:54:34.691638 3314 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:34.925449 3314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:34.927246 3314 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:34.927549 3314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:34.949977 3314 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:34.952193 3314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:34.952277 3314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:34.952389 3314 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:34.952440 3314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:36.060636 3446 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.132:44007 every 8 connection(s)
I20250627 13:54:36.060976 3314 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.132:44007
I20250627 13:54:36.063118 3314 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data/info.pb
I20250627 13:54:36.063724 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 3314
I20250627 13:54:36.130954 3447 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:54:36.131379 3447 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:36.132433 3447 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:36.133983 1617 ts_manager.cc:194] Re-registered known tserver with Master: 99bf3fe9a41d4ce1b61e440b31778689 (127.1.140.132:44007)
I20250627 13:54:36.135025 1617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.132:51535
I20250627 13:54:37.138231 3447 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
I20250627 13:54:37.659224 2532 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:37.661726 1877 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:37.721511 2794 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:54:37.729859 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:37.756376 2008 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:37.807624 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:37.820591 1746 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:37.884881 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:37.898033 2663 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:39.194963 1617 ts_manager.cc:284] Unset tserver state for aa90f1d263644537b32ad1a4595aef92 from MAINTENANCE_MODE
I20250627 13:54:39.242568 1942 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:39.294113 3163 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:39.305508 2073 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:39.338488 2859 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:39.470643 2728 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:39.503021 1811 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:39.586417 1617 ts_manager.cc:284] Unset tserver state for ffa5b64156aa4e0dbef89b3fb8111e44 from MAINTENANCE_MODE
I20250627 13:54:39.634316 3306 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:39.993566 1617 ts_manager.cc:284] Unset tserver state for 99bf3fe9a41d4ce1b61e440b31778689 from MAINTENANCE_MODE
I20250627 13:54:40.144917 3447 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:40.147763 2597 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:40.247867 1942 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:40.297346 3163 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:40.311501 2073 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:40.344669 2859 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:40.474220 2728 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:40.506685 1811 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:40.637542 3306 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:42.269449 1586 maintenance_mode-itest.cc:745] Restarting batch of 3 tservers: 0204b303a14a41b096bfddffc4d04290,e879d1cb3a7c4226884531badda84252,978726a17dd44a5891b685d3b2ae2b99
I20250627 13:54:43.260571 1617 ts_manager.cc:295] Set tserver state for e879d1cb3a7c4226884531badda84252 to MAINTENANCE_MODE
I20250627 13:54:43.640393 1617 ts_manager.cc:295] Set tserver state for 0204b303a14a41b096bfddffc4d04290 to MAINTENANCE_MODE
I20250627 13:54:43.796666 1617 ts_manager.cc:295] Set tserver state for 978726a17dd44a5891b685d3b2ae2b99 to MAINTENANCE_MODE
I20250627 13:54:47.753014 2008 tablet_service.cc:1423] Tablet server e879d1cb3a7c4226884531badda84252 set to quiescing
I20250627 13:54:47.753379 2008 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:47.783303 1877 tablet_service.cc:1423] Tablet server 0204b303a14a41b096bfddffc4d04290 set to quiescing
I20250627 13:54:47.783520 1877 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:54:47.789534 3569 raft_consensus.cc:991] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: : Instructing follower 373a9aec2682487a8ff2a4f0d1ee87ab to start an election
I20250627 13:54:47.789680 3569 raft_consensus.cc:1079] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 1 LEADER]: Signalling peer 373a9aec2682487a8ff2a4f0d1ee87ab to start an election
I20250627 13:54:47.809052 2952 raft_consensus.cc:991] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: : Instructing follower 373a9aec2682487a8ff2a4f0d1ee87ab to start an election
I20250627 13:54:47.809181 2952 raft_consensus.cc:1079] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 1 LEADER]: Signalling peer 373a9aec2682487a8ff2a4f0d1ee87ab to start an election
I20250627 13:54:47.809432 2952 raft_consensus.cc:991] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: : Instructing follower 5f1fab906378491e919a4a3364723428 to start an election
I20250627 13:54:47.809531 2952 raft_consensus.cc:1079] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 1 LEADER]: Signalling peer 5f1fab906378491e919a4a3364723428 to start an election
I20250627 13:54:47.810887 2551 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8dca13501f184a25a621c700a1939986"
dest_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab"
from {username='slave'} at 127.1.140.130:33317
I20250627 13:54:47.811127 2551 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:47.811312 2551 raft_consensus.cc:3058] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:47.813584 2814 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8dca13501f184a25a621c700a1939986"
dest_uuid: "5f1fab906378491e919a4a3364723428"
from {username='slave'} at 127.1.140.130:41103
I20250627 13:54:47.813825 2814 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:47.813958 2814 raft_consensus.cc:3058] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 1 FOLLOWER]: Advancing to term 2
I20250627 13:54:47.816882 2814 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:54:47.817556 2552 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8dca13501f184a25a621c700a1939986"
dest_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab"
from {username='slave'} at 127.1.140.130:33317
I20250627 13:54:47.818833 2551 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:54:47.819501 2552 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:54:47.819594 2552 raft_consensus.cc:3058] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 2 FOLLOWER]: Advancing to term 3
I20250627 13:54:47.822152 2552 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:54:47.825066 2551 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [CANDIDATE]: Term 2 election: Requested vote from peers 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611), 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207)
I20250627 13:54:47.827311 2814 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [CANDIDATE]: Term 2 election: Requested vote from peers 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009), 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207)
I20250627 13:54:47.835664 2552 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [CANDIDATE]: Term 3 election: Requested vote from peers 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611), 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207)
I20250627 13:54:47.850914 2552 raft_consensus.cc:1238] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 3 FOLLOWER]: Rejecting Update request from peer 0204b303a14a41b096bfddffc4d04290 for earlier term 1. Current term is 3. Ops: [1.1942-1.1942]
I20250627 13:54:47.851635 3569 consensus_queue.cc:1046] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 }, Status: INVALID_TERM, Last received: 1.1941, Next index: 1942, Last known committed idx: 1941, Time since last communication: 0.000s
I20250627 13:54:47.852159 3556 raft_consensus.cc:3053] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 1 LEADER]: Stepping down as leader of term 1
I20250627 13:54:47.852258 3556 raft_consensus.cc:738] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0204b303a14a41b096bfddffc4d04290, State: Running, Role: LEADER
I20250627 13:54:47.852644 3556 consensus_queue.cc:260] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1941, Committed index: 1941, Last appended: 1.1942, Last appended by leader: 1942, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:54:47.854350 3556 raft_consensus.cc:3058] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 1 FOLLOWER]: Advancing to term 3
W20250627 13:54:47.858371 2909 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51560: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.009s ago.
I20250627 13:54:47.861853 1897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "5f1fab906378491e919a4a3364723428" candidate_term: 2 candidate_status { last_received { term: 1 index: 1941 } } ignore_live_leader: true dest_uuid: "0204b303a14a41b096bfddffc4d04290"
I20250627 13:54:47.864213 1897 raft_consensus.cc:2366] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f1fab906378491e919a4a3364723428 for earlier term 2. Current term is 3.
I20250627 13:54:47.865568 2749 leader_election.cc:400] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [CANDIDATE]: Term 2 election: Vote denied by peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207) with higher term. Message: Invalid argument: T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f1fab906378491e919a4a3364723428 for earlier term 2. Current term is 3.
I20250627 13:54:47.865689 2749 leader_election.cc:403] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20250627 13:54:47.866806 2909 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51560: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.018s ago.
I20250627 13:54:47.874902 1897 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1941 } } ignore_live_leader: true dest_uuid: "0204b303a14a41b096bfddffc4d04290"
I20250627 13:54:47.875022 1896 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1941 } } ignore_live_leader: true dest_uuid: "0204b303a14a41b096bfddffc4d04290"
I20250627 13:54:47.875201 1897 raft_consensus.cc:2408] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 373a9aec2682487a8ff2a4f0d1ee87ab for term 3 because replica has last-logged OpId of term: 1 index: 1942, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1941.
I20250627 13:54:47.875211 3648 raft_consensus.cc:3058] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 2 FOLLOWER]: Advancing to term 3
W20250627 13:54:47.878867 2769 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34530: Illegal state: replica 5f1fab906378491e919a4a3364723428 is not leader of this config: current role FOLLOWER
W20250627 13:54:47.878896 2770 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34530: Illegal state: replica 5f1fab906378491e919a4a3364723428 is not leader of this config: current role FOLLOWER
I20250627 13:54:47.882228 2814 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" candidate_term: 3 candidate_status { last_received { term: 1 index: 1941 } } ignore_live_leader: true dest_uuid: "5f1fab906378491e919a4a3364723428"
I20250627 13:54:47.882580 2812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1941 } } ignore_live_leader: true dest_uuid: "5f1fab906378491e919a4a3364723428"
W20250627 13:54:47.883644 2511 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47486: Illegal state: replica 373a9aec2682487a8ff2a4f0d1ee87ab is not leader of this config: current role FOLLOWER
I20250627 13:54:47.884682 3648 raft_consensus.cc:2747] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207) with higher term. Message: Invalid argument: T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f1fab906378491e919a4a3364723428 for earlier term 2. Current term is 3.
I20250627 13:54:47.884814 2486 leader_election.cc:304] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 373a9aec2682487a8ff2a4f0d1ee87ab; no voters: 0204b303a14a41b096bfddffc4d04290, 5f1fab906378491e919a4a3364723428
I20250627 13:54:47.886910 2552 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "5f1fab906378491e919a4a3364723428" candidate_term: 2 candidate_status { last_received { term: 1 index: 1941 } } ignore_live_leader: true dest_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab"
I20250627 13:54:47.887241 2552 raft_consensus.cc:2366] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f1fab906378491e919a4a3364723428 for earlier term 2. Current term is 3.
I20250627 13:54:47.887429 2814 raft_consensus.cc:2466] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 373a9aec2682487a8ff2a4f0d1ee87ab in term 3.
I20250627 13:54:47.887712 2750 leader_election.cc:400] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [CANDIDATE]: Term 2 election: Vote denied by peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009) with higher term. Message: Invalid argument: T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5f1fab906378491e919a4a3364723428 for earlier term 2. Current term is 3.
W20250627 13:54:47.888363 2511 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47486: Illegal state: replica 373a9aec2682487a8ff2a4f0d1ee87ab is not leader of this config: current role FOLLOWER
W20250627 13:54:47.894476 1857 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51560: Illegal state: replica 0204b303a14a41b096bfddffc4d04290 is not leader of this config: current role FOLLOWER
W20250627 13:54:47.904903 2767 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34530: Illegal state: replica 5f1fab906378491e919a4a3364723428 is not leader of this config: current role FOLLOWER
I20250627 13:54:47.905215 3649 raft_consensus.cc:2747] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250627 13:54:47.905712 2486 leader_election.cc:304] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 373a9aec2682487a8ff2a4f0d1ee87ab, 5f1fab906378491e919a4a3364723428; no voters: 0204b303a14a41b096bfddffc4d04290
I20250627 13:54:47.906111 3649 raft_consensus.cc:2802] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 3 FOLLOWER]: Leader election won for term 3
I20250627 13:54:47.913105 3649 raft_consensus.cc:695] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 3 LEADER]: Becoming Leader. State: Replica: 373a9aec2682487a8ff2a4f0d1ee87ab, State: Running, Role: LEADER
W20250627 13:54:47.914305 1857 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51560: Illegal state: replica 0204b303a14a41b096bfddffc4d04290 is not leader of this config: current role FOLLOWER
W20250627 13:54:47.914319 2511 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47486: Service unavailable: leader is not yet ready
I20250627 13:54:47.915447 3649 consensus_queue.cc:237] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1941, Committed index: 1941, Last appended: 1.1941, Last appended by leader: 1941, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:54:47.919555 1617 catalog_manager.cc:5582] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab reported cstate change: term changed from 1 to 3, leader changed from 0204b303a14a41b096bfddffc4d04290 (127.1.140.130) to 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135). New cstate: current_term: 3 leader_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } health_report { overall_health: UNKNOWN } } }
I20250627 13:54:47.927014 2814 raft_consensus.cc:1273] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 3 FOLLOWER]: Refusing update from remote peer 373a9aec2682487a8ff2a4f0d1ee87ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1941. Preceding OpId from leader: term: 3 index: 1943. (index mismatch)
I20250627 13:54:47.927279 1896 raft_consensus.cc:1273] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 3 FOLLOWER]: Refusing update from remote peer 373a9aec2682487a8ff2a4f0d1ee87ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1942. Preceding OpId from leader: term: 3 index: 1943. (index mismatch)
I20250627 13:54:47.927533 3649 consensus_queue.cc:1035] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1942, Last known committed idx: 1941, Time since last communication: 0.000s
I20250627 13:54:47.929112 3649 consensus_queue.cc:1035] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1942, Last known committed idx: 1941, Time since last communication: 0.000s
I20250627 13:54:47.929519 3649 consensus_queue.cc:1230] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [LEADER]: Peer 0204b303a14a41b096bfddffc4d04290 log is divergent from this leader: its last log entry 1.1942 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1941
I20250627 13:54:47.932454 1896 pending_rounds.cc:85] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Aborting all ops after (but not including) 1941
I20250627 13:54:47.932600 1896 pending_rounds.cc:107] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Aborting uncommitted WRITE_OP operation due to leader change: 1.1942
W20250627 13:54:47.932822 1896 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:51560: Aborted: Op aborted by new leader
I20250627 13:54:47.942050 2911 mvcc.cc:204] Tried to move back new op lower bound from 7172229070530334720 to 7172229070507765760. Current Snapshot: MvccSnapshot[applied={T|T < 7172229070530334720 or (T in {7172229070530334720})}]
I20250627 13:54:47.945852 2912 mvcc.cc:204] Tried to move back new op lower bound from 7172229070530334720 to 7172229070507765760. Current Snapshot: MvccSnapshot[applied={T|T < 7172229070530334720 or (T in {7172229070530334720})}]
I20250627 13:54:49.081399 1746 tablet_service.cc:1423] Tablet server 978726a17dd44a5891b685d3b2ae2b99 set to quiescing
I20250627 13:54:49.081683 1746 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:54:52.362360 1877 tablet_service.cc:1423] Tablet server 0204b303a14a41b096bfddffc4d04290 set to quiescing
I20250627 13:54:52.362640 1877 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:54:53.747291 2487 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.140.130:42207, user_credentials={real_user=slave}} blocked reactor thread for 52593.5us
I20250627 13:54:53.877768 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 1814
I20250627 13:54:53.930471 1937 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:54:54.345569 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 1814
W20250627 13:54:54.428789 2487 connection.cc:537] client connection to 127.1.140.130:42207 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:54:54.429075 2487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:54:54.436770 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.130:42207
--local_ip_for_outbound_sockets=127.1.140.130
--tserver_master_addrs=127.1.140.190:35061
--webserver_port=46771
--webserver_interface=127.1.140.130
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:54:54.449919 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.130:42207: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:54:54.938784 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.130:42207: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:54:54.942090 3719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:54.942531 3719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:54.942735 3719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:54.958457 3719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:54.958772 3719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.130
I20250627 13:54:54.972128 3719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.130:42207
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.140.130
--webserver_port=46771
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.130
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:54.974582 3719 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:54.977145 3719 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:55.019204 3727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:55.019722 3726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:55.038152 3730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:55.045017 3719 server_base.cc:1048] running on GCE node
I20250627 13:54:55.046674 3719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:55.050830 3719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:55.054329 3719 hybrid_clock.cc:648] HybridClock initialized: now 1751032495054112 us; error 203 us; skew 500 ppm
I20250627 13:54:55.054755 3719 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:55.066630 3719 webserver.cc:469] Webserver started at http://127.1.140.130:46771/ using document root <none> and password file <none>
I20250627 13:54:55.067611 3719 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:55.067790 3719 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:55.082613 3719 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.002s sys 0.007s
I20250627 13:54:55.091207 3737 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:55.094372 3719 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.002s sys 0.004s
I20250627 13:54:55.094525 3719 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal
uuid: "0204b303a14a41b096bfddffc4d04290"
format_stamp: "Formatted at 2025-06-27 13:54:18 on dist-test-slave-lj78"
I20250627 13:54:55.095468 3719 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:55.158326 3719 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:55.159101 3719 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:55.159338 3719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:55.160286 3719 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:55.162438 3744 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:54:55.172335 3719 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:54:55.172679 3719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.000s sys 0.001s
I20250627 13:54:55.172914 3719 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:54:55.187904 3719 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:54:55.188051 3719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.015s user 0.003s sys 0.001s
I20250627 13:54:55.192420 3744 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Bootstrap starting.
W20250627 13:54:55.393589 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.130:42207: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:54:55.864426 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.130:42207: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250627 13:54:56.319203 3852 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.130:42207 every 8 connection(s)
I20250627 13:54:56.319563 3719 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.130:42207
I20250627 13:54:56.321780 3719 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data/info.pb
I20250627 13:54:56.332007 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 3719
I20250627 13:54:56.332219 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 1945
I20250627 13:54:56.384483 3853 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:54:56.384874 3853 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:56.385916 3853 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:56.389068 1616 ts_manager.cc:194] Re-registered known tserver with Master: 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207)
I20250627 13:54:56.390657 1616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.130:47269
W20250627 13:54:56.433171 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
I20250627 13:54:56.439988 2068 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:54:56.696055 3744 log.cc:826] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Log is configured to *not* fsync() on all Append() calls
W20250627 13:54:56.858057 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
I20250627 13:54:56.889055 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 1945
I20250627 13:54:56.946452 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.131:33589
--local_ip_for_outbound_sockets=127.1.140.131
--tserver_master_addrs=127.1.140.190:35061
--webserver_port=41547
--webserver_interface=127.1.140.131
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250627 13:54:57.395947 3853 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
W20250627 13:54:57.399766 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
W20250627 13:54:57.698926 3863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:54:57.699460 3863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:54:57.699644 3863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:54:57.711614 3863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:54:57.711923 3863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.131
I20250627 13:54:57.727264 3863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.131:33589
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.140.131
--webserver_port=41547
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.131
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:54:57.735725 3863 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:54:57.740445 3863 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:54:57.842339 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
W20250627 13:54:57.903507 3869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:54:57.956424 3870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:58.002883 3863 server_base.cc:1048] running on GCE node
W20250627 13:54:58.013584 3872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:54:58.014531 3863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:54:58.016609 3863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:54:58.032192 3863 hybrid_clock.cc:648] HybridClock initialized: now 1751032498025521 us; error 6632 us; skew 500 ppm
I20250627 13:54:58.032619 3863 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:54:58.065542 3863 webserver.cc:469] Webserver started at http://127.1.140.131:41547/ using document root <none> and password file <none>
I20250627 13:54:58.066304 3863 fs_manager.cc:362] Metadata directory not provided
I20250627 13:54:58.066421 3863 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:54:58.087327 3863 fs_manager.cc:714] Time spent opening directory manager: real 0.018s user 0.005s sys 0.005s
I20250627 13:54:58.102855 3879 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:58.110558 3863 fs_manager.cc:730] Time spent opening block manager: real 0.021s user 0.007s sys 0.000s
I20250627 13:54:58.110824 3863 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/wal
uuid: "e879d1cb3a7c4226884531badda84252"
format_stamp: "Formatted at 2025-06-27 13:54:19 on dist-test-slave-lj78"
I20250627 13:54:58.111838 3863 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:54:58.252570 3863 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:54:58.264474 3863 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:54:58.264959 3863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:54:58.298637 3863 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:54:58.324088 3720 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [LEADER]: Peer 0204b303a14a41b096bfddffc4d04290 is lagging by at least 2 ops behind the committed index
I20250627 13:54:58.334370 3863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:54:58.334554 3863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:54:58.334900 3863 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:54:58.335041 3863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20250627 13:54:58.354061 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20250627 13:54:58.890237 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
I20250627 13:54:59.525609 3995 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.131:33589 every 8 connection(s)
I20250627 13:54:59.525926 3863 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.131:33589
I20250627 13:54:59.527851 3863 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data/info.pb
I20250627 13:54:59.535686 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 3863
I20250627 13:54:59.535846 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 1684
I20250627 13:54:59.582453 3996 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:54:59.582854 3996 heartbeater.cc:461] Registering TS with master...
I20250627 13:54:59.583925 3996 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:54:59.585598 1616 ts_manager.cc:194] Re-registered known tserver with Master: e879d1cb3a7c4226884531badda84252 (127.1.140.131:33589)
I20250627 13:54:59.586550 1616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.131:56109
I20250627 13:54:59.623317 1805 generic_service.cc:196] Checking for leaks (request via RPC)
W20250627 13:54:59.707657 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
I20250627 13:55:00.034526 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 1684
I20250627 13:55:00.074875 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.129:35871
--local_ip_for_outbound_sockets=127.1.140.129
--tserver_master_addrs=127.1.140.190:35061
--webserver_port=42277
--webserver_interface=127.1.140.129
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:55:00.229157 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
I20250627 13:55:00.589432 3996 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
W20250627 13:55:00.778736 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
W20250627 13:55:01.012126 4008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:55:01.013533 4008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:55:01.013773 4008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:55:01.025651 4008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:55:01.025897 4008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.129
I20250627 13:55:01.039669 4008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.129:35871
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.140.129
--webserver_port=42277
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.129
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:55:01.042207 4008 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:55:01.044698 4008 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:55:01.117426 4014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:55:01.153182 4015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:55:01.212585 4008 server_base.cc:1048] running on GCE node
W20250627 13:55:01.221879 4018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:55:01.223026 4008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:55:01.244837 4008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:55:01.246219 4008 hybrid_clock.cc:648] HybridClock initialized: now 1751032501246103 us; error 106 us; skew 500 ppm
I20250627 13:55:01.256479 4008 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:55:01.389119 4006 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [LEADER]: Peer 0204b303a14a41b096bfddffc4d04290 is lagging by at least 261 ops behind the committed index [suppressed 24 similar messages]
I20250627 13:55:01.413805 4008 webserver.cc:469] Webserver started at http://127.1.140.129:42277/ using document root <none> and password file <none>
I20250627 13:55:01.415073 4008 fs_manager.cc:362] Metadata directory not provided
I20250627 13:55:01.415305 4008 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250627 13:55:01.428535 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
I20250627 13:55:01.430823 4008 fs_manager.cc:714] Time spent opening directory manager: real 0.013s user 0.006s sys 0.006s
I20250627 13:55:01.469828 4025 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:01.475212 4008 fs_manager.cc:730] Time spent opening block manager: real 0.040s user 0.002s sys 0.023s
I20250627 13:55:01.475435 4008 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/wal
uuid: "978726a17dd44a5891b685d3b2ae2b99"
format_stamp: "Formatted at 2025-06-27 13:54:18 on dist-test-slave-lj78"
I20250627 13:55:01.478976 4008 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:55:01.669142 4008 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:55:01.670228 4008 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:55:01.670565 4008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:55:01.682739 4008 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:55:01.707399 3744 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Bootstrap replaying log segment 1/1 (1.47M/4.19M this segment, stats: ops{read=850 overwritten=0 applied=848 ignored=0} inserts{seen=42350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250627 13:55:01.743306 4008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:55:01.743510 4008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:01.743729 4008 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:55:01.743830 4008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20250627 13:55:02.184873 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
W20250627 13:55:02.805819 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
I20250627 13:55:03.078260 4008 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.129:35871
I20250627 13:55:03.080124 4008 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data/info.pb
I20250627 13:55:03.089663 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 4008
I20250627 13:55:03.090562 4139 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.129:35871 every 8 connection(s)
I20250627 13:55:03.120136 4140 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:55:03.120515 4140 heartbeater.cc:461] Registering TS with master...
I20250627 13:55:03.121583 4140 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:03.123337 1616 ts_manager.cc:194] Re-registered known tserver with Master: 978726a17dd44a5891b685d3b2ae2b99 (127.1.140.129:35871)
I20250627 13:55:03.124365 1616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.129:34891
W20250627 13:55:03.359856 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250627 13:55:03.837419 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
I20250627 13:55:04.129860 4140 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
W20250627 13:55:04.340029 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
I20250627 13:55:04.411477 4160 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [LEADER]: Peer 0204b303a14a41b096bfddffc4d04290 is lagging by at least 498 ops behind the committed index [suppressed 25 similar messages]
I20250627 13:55:04.489444 3786 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:04.517554 2794 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:55:04.553543 2532 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:55:04.608242 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:04.636103 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:04.658334 2663 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:04.661072 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:04.682138 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:04.747761 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:55:05.189604 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
W20250627 13:55:05.731683 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
W20250627 13:55:06.245852 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
I20250627 13:55:06.707542 3744 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Bootstrap replaying log segment 1/1 (3.22M/4.19M this segment, stats: ops{read=1859 overwritten=0 applied=1857 ignored=0} inserts{seen=92800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250627 13:55:06.717496 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
W20250627 13:55:07.229826 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
I20250627 13:55:07.426043 4188 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [LEADER]: Peer 0204b303a14a41b096bfddffc4d04290 is lagging by at least 785 ops behind the committed index [suppressed 25 similar messages]
W20250627 13:55:07.761217 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
W20250627 13:55:08.291229 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
W20250627 13:55:08.984387 2487 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
I20250627 13:55:09.171430 3744 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Bootstrap replayed 1/1 log segments. Stats: ops{read=2429 overwritten=1 applied=2427 ignored=0} inserts{seen=121250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250627 13:55:09.172284 3744 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Bootstrap complete.
I20250627 13:55:09.174409 3744 ts_tablet_manager.cc:1397] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Time spent bootstrapping tablet: real 13.983s user 9.406s sys 0.570s
I20250627 13:55:09.179162 3744 raft_consensus.cc:357] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 3 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:55:09.186466 3744 raft_consensus.cc:738] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0204b303a14a41b096bfddffc4d04290, State: Initialized, Role: FOLLOWER
I20250627 13:55:09.187235 3744 consensus_queue.cc:260] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2427, Last appended: 3.2428, Last appended by leader: 2428, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:55:09.188784 3744 ts_tablet_manager.cc:1428] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20250627 13:55:09.987159 4214 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 373a9aec2682487a8ff2a4f0d1ee87ab)
I20250627 13:55:09.987442 4214 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:55:09.991988 4214 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611), 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009)
I20250627 13:55:10.083334 2814 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "0204b303a14a41b096bfddffc4d04290" candidate_term: 4 candidate_status { last_received { term: 3 index: 3278 } } ignore_live_leader: false dest_uuid: "5f1fab906378491e919a4a3364723428" is_pre_election: true
I20250627 13:55:10.083652 2552 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "0204b303a14a41b096bfddffc4d04290" candidate_term: 4 candidate_status { last_received { term: 3 index: 3278 } } ignore_live_leader: false dest_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" is_pre_election: true
I20250627 13:55:10.084367 3741 leader_election.cc:304] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0204b303a14a41b096bfddffc4d04290; no voters: 373a9aec2682487a8ff2a4f0d1ee87ab, 5f1fab906378491e919a4a3364723428
I20250627 13:55:10.191478 4214 raft_consensus.cc:2747] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250627 13:55:12.338654 3786 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:12.360713 2532 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:55:12.439689 2794 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:55:12.505707 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:12.546962 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:12.554880 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:12.598215 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:12.608718 2663 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:12.681006 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:14.599277 1616 ts_manager.cc:284] Unset tserver state for 0204b303a14a41b096bfddffc4d04290 from MAINTENANCE_MODE
I20250627 13:55:14.638276 3996 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:14.715934 3306 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:14.749931 1616 ts_manager.cc:284] Unset tserver state for e879d1cb3a7c4226884531badda84252 from MAINTENANCE_MODE
I20250627 13:55:15.130596 2859 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:15.166532 4140 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:15.229825 3447 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:15.392059 3163 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:15.456333 3853 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:15.482887 2597 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:15.559615 2728 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:15.641993 3996 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:15.720356 3306 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:17.050501 1616 ts_manager.cc:284] Unset tserver state for 978726a17dd44a5891b685d3b2ae2b99 from MAINTENANCE_MODE
I20250627 13:55:17.140744 2859 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:17.171257 4140 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:17.235347 3447 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:17.397555 3163 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:17.464710 3853 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:17.506246 2597 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:17.566776 2728 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:17.646698 3996 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:17.725306 3306 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:19.518343 1586 maintenance_mode-itest.cc:745] Restarting batch of 1 tservers: 373a9aec2682487a8ff2a4f0d1ee87ab
W20250627 13:55:19.633518 2871 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.140.135:34009, user_credentials={real_user=slave}} blocked reactor thread for 87504.4us
I20250627 13:55:20.486048 1616 ts_manager.cc:295] Set tserver state for 373a9aec2682487a8ff2a4f0d1ee87ab to MAINTENANCE_MODE
I20250627 13:55:22.885213 2532 tablet_service.cc:1423] Tablet server 373a9aec2682487a8ff2a4f0d1ee87ab set to quiescing
I20250627 13:55:22.885462 2532 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:55:22.888418 4264 raft_consensus.cc:991] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: : Instructing follower 0204b303a14a41b096bfddffc4d04290 to start an election
I20250627 13:55:22.888577 4264 raft_consensus.cc:1079] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 3 LEADER]: Signalling peer 0204b303a14a41b096bfddffc4d04290 to start an election
I20250627 13:55:22.889302 3805 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8dca13501f184a25a621c700a1939986"
dest_uuid: "0204b303a14a41b096bfddffc4d04290"
from {username='slave'} at 127.1.140.135:42891
I20250627 13:55:22.889556 3805 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250627 13:55:22.889673 3805 raft_consensus.cc:3058] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 3 FOLLOWER]: Advancing to term 4
I20250627 13:55:22.894079 3805 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:55:22.895663 3806 raft_consensus.cc:1238] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 4 FOLLOWER]: Rejecting Update request from peer 373a9aec2682487a8ff2a4f0d1ee87ab for earlier term 3. Current term is 4. Ops: []
I20250627 13:55:22.896086 3805 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [CANDIDATE]: Term 4 election: Requested vote from peers 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611), 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009)
I20250627 13:55:22.896443 2552 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "0204b303a14a41b096bfddffc4d04290" candidate_term: 4 candidate_status { last_received { term: 3 index: 4958 } } ignore_live_leader: true dest_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab"
I20250627 13:55:22.896610 2552 raft_consensus.cc:3053] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 3 LEADER]: Stepping down as leader of term 3
I20250627 13:55:22.896669 2552 raft_consensus.cc:738] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 373a9aec2682487a8ff2a4f0d1ee87ab, State: Running, Role: LEADER
I20250627 13:55:22.896670 2814 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "0204b303a14a41b096bfddffc4d04290" candidate_term: 4 candidate_status { last_received { term: 3 index: 4958 } } ignore_live_leader: true dest_uuid: "5f1fab906378491e919a4a3364723428"
I20250627 13:55:22.896806 2814 raft_consensus.cc:3058] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 3 FOLLOWER]: Advancing to term 4
I20250627 13:55:22.896911 2552 consensus_queue.cc:260] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4958, Committed index: 4958, Last appended: 3.4958, Last appended by leader: 4958, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:55:22.897094 2552 logging.cc:424] LogThrottler TrackedPeer Lag: suppressed but not reported on 18 messages since previous log ~15 seconds ago
I20250627 13:55:22.897171 2552 raft_consensus.cc:3058] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 3 FOLLOWER]: Advancing to term 4
I20250627 13:55:22.899852 2552 raft_consensus.cc:2466] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0204b303a14a41b096bfddffc4d04290 in term 4.
I20250627 13:55:22.899852 2814 raft_consensus.cc:2466] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0204b303a14a41b096bfddffc4d04290 in term 4.
I20250627 13:55:22.900375 3739 leader_election.cc:304] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0204b303a14a41b096bfddffc4d04290, 5f1fab906378491e919a4a3364723428; no voters:
W20250627 13:55:22.902768 2512 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47486: Illegal state: replica 373a9aec2682487a8ff2a4f0d1ee87ab is not leader of this config: current role FOLLOWER
I20250627 13:55:22.903808 4387 raft_consensus.cc:2802] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 4 FOLLOWER]: Leader election won for term 4
I20250627 13:55:22.907548 4387 raft_consensus.cc:695] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [term 4 LEADER]: Becoming Leader. State: Replica: 0204b303a14a41b096bfddffc4d04290, State: Running, Role: LEADER
I20250627 13:55:22.907996 4387 consensus_queue.cc:237] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4957, Committed index: 4957, Last appended: 3.4958, Last appended by leader: 4958, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
W20250627 13:55:22.908648 2770 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34530: Illegal state: replica 5f1fab906378491e919a4a3364723428 is not leader of this config: current role FOLLOWER
W20250627 13:55:22.908715 2768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34530: Illegal state: replica 5f1fab906378491e919a4a3364723428 is not leader of this config: current role FOLLOWER
W20250627 13:55:22.909653 2768 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:34530: Illegal state: replica 5f1fab906378491e919a4a3364723428 is not leader of this config: current role FOLLOWER
I20250627 13:55:22.911675 1617 catalog_manager.cc:5582] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 reported cstate change: term changed from 3 to 4, leader changed from 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135) to 0204b303a14a41b096bfddffc4d04290 (127.1.140.130). New cstate: current_term: 4 leader_uuid: "0204b303a14a41b096bfddffc4d04290" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } health_report { overall_health: HEALTHY } } }
W20250627 13:55:22.913861 2512 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47486: Illegal state: replica 373a9aec2682487a8ff2a4f0d1ee87ab is not leader of this config: current role FOLLOWER
I20250627 13:55:22.945541 2552 raft_consensus.cc:1273] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 4 FOLLOWER]: Refusing update from remote peer 0204b303a14a41b096bfddffc4d04290: Log matching property violated. Preceding OpId in replica: term: 3 index: 4958. Preceding OpId from leader: term: 4 index: 4960. (index mismatch)
I20250627 13:55:22.946705 2814 raft_consensus.cc:1273] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Refusing update from remote peer 0204b303a14a41b096bfddffc4d04290: Log matching property violated. Preceding OpId in replica: term: 3 index: 4958. Preceding OpId from leader: term: 4 index: 4960. (index mismatch)
I20250627 13:55:22.946739 4388 consensus_queue.cc:1035] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4959, Last known committed idx: 4958, Time since last communication: 0.000s
I20250627 13:55:22.947427 4387 consensus_queue.cc:1035] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4959, Last known committed idx: 4957, Time since last communication: 0.000s
I20250627 13:55:22.972422 4213 mvcc.cc:204] Tried to move back new op lower bound from 7172229213975146496 to 7172229213835722752. Current Snapshot: MvccSnapshot[applied={T|T < 7172229213975146496 or (T in {7172229213975146496})}]
I20250627 13:55:26.181972 2532 tablet_service.cc:1423] Tablet server 373a9aec2682487a8ff2a4f0d1ee87ab set to quiescing
I20250627 13:55:26.182175 2532 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:27.771159 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 2469
I20250627 13:55:27.841548 2591 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:55:28.295630 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 2469
W20250627 13:55:28.402490 3741 connection.cc:537] client connection to 127.1.140.135:34009 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:55:28.402761 3741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:55:28.404132 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.135:34009
--local_ip_for_outbound_sockets=127.1.140.135
--tserver_master_addrs=127.1.140.190:35061
--webserver_port=45365
--webserver_interface=127.1.140.135
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:55:28.416301 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.135:34009: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:55:28.938632 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.135:34009: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:55:28.992513 4441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:55:28.992965 4441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:55:28.993160 4441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:55:29.006283 4441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:55:29.006625 4441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.135
I20250627 13:55:29.020349 4441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.135:34009
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data/info.pb
--webserver_interface=127.1.140.135
--webserver_port=45365
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.135
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:55:29.022396 4441 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:55:29.024847 4441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:55:29.053565 4448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:55:29.054313 4449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:55:29.080395 4441 server_base.cc:1048] running on GCE node
W20250627 13:55:29.089766 4451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:55:29.092943 4441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:55:29.096711 4441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:55:29.100982 4441 hybrid_clock.cc:648] HybridClock initialized: now 1751032529100854 us; error 123 us; skew 500 ppm
I20250627 13:55:29.101480 4441 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:55:29.107478 4441 webserver.cc:469] Webserver started at http://127.1.140.135:45365/ using document root <none> and password file <none>
I20250627 13:55:29.108456 4441 fs_manager.cc:362] Metadata directory not provided
I20250627 13:55:29.109126 4441 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:55:29.119463 4441 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.004s sys 0.003s
I20250627 13:55:29.135146 4457 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:55:29.136998 4441 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20250627 13:55:29.137161 4441 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/wal
uuid: "373a9aec2682487a8ff2a4f0d1ee87ab"
format_stamp: "Formatted at 2025-06-27 13:54:20 on dist-test-slave-lj78"
I20250627 13:55:29.138247 4441 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:55:29.202610 4441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:55:29.203395 4441 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:55:29.203677 4441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:55:29.204655 4441 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:55:29.210932 4464 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:55:29.254043 4441 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:55:29.254179 4441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.048s user 0.003s sys 0.000s
I20250627 13:55:29.254312 4441 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:55:29.258564 4441 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:55:29.258765 4441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.004s sys 0.000s
I20250627 13:55:29.259912 4464 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Bootstrap starting.
W20250627 13:55:29.448930 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.135:34009: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:55:29.990646 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.135:34009: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250627 13:55:30.172160 4464 log.cc:826] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Log is configured to *not* fsync() on all Append() calls
W20250627 13:55:30.482174 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.135:34009: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250627 13:55:30.724279 4577 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.135:34009 every 8 connection(s)
I20250627 13:55:30.724586 4441 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.135:34009
I20250627 13:55:30.726369 4441 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data/info.pb
I20250627 13:55:30.734100 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 4441
I20250627 13:55:30.774004 4578 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:55:30.774438 4578 heartbeater.cc:461] Registering TS with master...
I20250627 13:55:30.775676 4578 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:55:30.778291 1618 ts_manager.cc:194] Re-registered known tserver with Master: 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009)
I20250627 13:55:30.779795 1618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.135:34237
W20250627 13:55:31.006685 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
W20250627 13:55:31.601167 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250627 13:55:31.782990 4578 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
W20250627 13:55:32.265290 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20250627 13:55:32.365080 4486 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 373a9aec2682487a8ff2a4f0d1ee87ab is lagging by at least 11 ops behind the committed index
I20250627 13:55:32.610867 4509 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:32.629129 3786 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:55:32.646411 2794 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:55:32.653131 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:32.711449 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:32.778266 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:32.788544 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:32.905397 2663 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:32.955029 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:55:32.996773 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20250627 13:55:33.633399 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250627 13:55:34.164882 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
W20250627 13:55:34.709439 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
W20250627 13:55:35.267623 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
I20250627 13:55:35.269533 4464 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Bootstrap replaying log segment 1/2 (1.19M/8.00M this segment, stats: ops{read=701 overwritten=0 applied=701 ignored=0} inserts{seen=35000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250627 13:55:35.448287 4635 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 373a9aec2682487a8ff2a4f0d1ee87ab is lagging by at least 326 ops behind the committed index [suppressed 25 similar messages]
W20250627 13:55:35.838794 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
W20250627 13:55:36.327595 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
W20250627 13:55:36.874161 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
W20250627 13:55:37.310809 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250627 13:55:37.846791 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
W20250627 13:55:38.423345 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
I20250627 13:55:38.486893 4635 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 373a9aec2682487a8ff2a4f0d1ee87ab is lagging by at least 662 ops behind the committed index [suppressed 28 similar messages]
W20250627 13:55:39.049762 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
W20250627 13:55:39.636166 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
I20250627 13:55:39.683233 3786 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:55:39.738313 4509 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:39.778029 2794 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:55:39.815376 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:39.866482 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:39.883880 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:39.889997 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:39.891090 2663 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:39.924494 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:55:40.225399 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
I20250627 13:55:40.271011 4464 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Bootstrap replaying log segment 1/2 (3.04M/8.00M this segment, stats: ops{read=1790 overwritten=0 applied=1790 ignored=0} inserts{seen=89450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250627 13:55:40.721263 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
W20250627 13:55:41.273399 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
I20250627 13:55:41.580189 4644 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 373a9aec2682487a8ff2a4f0d1ee87ab is lagging by at least 926 ops behind the committed index [suppressed 26 similar messages]
W20250627 13:55:41.845057 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
W20250627 13:55:42.416052 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
W20250627 13:55:42.928288 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
W20250627 13:55:43.429419 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
W20250627 13:55:44.003050 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
W20250627 13:55:44.543349 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
I20250627 13:55:44.634550 4702 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 373a9aec2682487a8ff2a4f0d1ee87ab is lagging by at least 1222 ops behind the committed index [suppressed 27 similar messages]
W20250627 13:55:45.113745 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
I20250627 13:55:45.276605 4464 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Bootstrap replaying log segment 1/2 (5.09M/8.00M this segment, stats: ops{read=2976 overwritten=0 applied=2975 ignored=0} inserts{seen=148650 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250627 13:55:45.686371 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 156: this message will repeat every 5th retry.
W20250627 13:55:46.225082 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 161: this message will repeat every 5th retry.
W20250627 13:55:46.776754 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 166: this message will repeat every 5th retry.
W20250627 13:55:47.322207 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 171: this message will repeat every 5th retry.
I20250627 13:55:47.612876 3786 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:55:47.619509 4509 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:47.698063 4714 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 373a9aec2682487a8ff2a4f0d1ee87ab is lagging by at least 1511 ops behind the committed index [suppressed 27 similar messages]
I20250627 13:55:47.732272 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:47.733460 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:47.746713 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:47.757496 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:47.791810 2663 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:47.805442 2794 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
W20250627 13:55:47.831703 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 176: this message will repeat every 5th retry.
I20250627 13:55:47.854772 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:55:48.353397 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 181: this message will repeat every 5th retry.
W20250627 13:55:48.789387 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 186: this message will repeat every 5th retry.
W20250627 13:55:48.848330 2870 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.140.130:42207, user_credentials={real_user=slave}} blocked reactor thread for 54721.8us
W20250627 13:55:49.286460 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 191: this message will repeat every 5th retry.
W20250627 13:55:49.843608 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 196: this message will repeat every 5th retry.
I20250627 13:55:50.277559 4464 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Bootstrap replaying log segment 1/2 (7.10M/8.00M this segment, stats: ops{read=4138 overwritten=0 applied=4138 ignored=0} inserts{seen=206800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250627 13:55:50.414415 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 201: this message will repeat every 5th retry.
I20250627 13:55:50.777344 4767 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 373a9aec2682487a8ff2a4f0d1ee87ab is lagging by at least 1844 ops behind the committed index [suppressed 29 similar messages]
W20250627 13:55:50.990164 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 206: this message will repeat every 5th retry.
W20250627 13:55:51.475804 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 211: this message will repeat every 5th retry.
W20250627 13:55:51.968700 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 216: this message will repeat every 5th retry.
W20250627 13:55:52.513501 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 221: this message will repeat every 5th retry.
I20250627 13:55:52.537567 4464 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Bootstrap replayed 1/2 log segments. Stats: ops{read=4659 overwritten=0 applied=4658 ignored=0} inserts{seen=232800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20250627 13:55:53.091852 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 226: this message will repeat every 5th retry.
W20250627 13:55:53.577402 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 231: this message will repeat every 5th retry.
I20250627 13:55:53.842883 4768 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 373a9aec2682487a8ff2a4f0d1ee87ab is lagging by at least 2176 ops behind the committed index [suppressed 28 similar messages]
W20250627 13:55:54.053409 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 236: this message will repeat every 5th retry.
I20250627 13:55:54.502693 3786 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:55:54.515455 4509 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:55:54.539325 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 241: this message will repeat every 5th retry.
I20250627 13:55:54.590435 2794 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:55:54.627086 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:54.674871 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:54.684908 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:54.697834 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:54.725170 2663 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:55:54.748997 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:55:55.043939 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 246: this message will repeat every 5th retry.
I20250627 13:55:55.278615 4464 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Bootstrap replaying log segment 2/2 (1.04M/1.33M this segment, stats: ops{read=5268 overwritten=0 applied=5268 ignored=0} inserts{seen=263250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250627 13:55:55.512176 3741 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 251: this message will repeat every 5th retry.
I20250627 13:55:55.915277 4464 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Bootstrap replayed 2/2 log segments. Stats: ops{read=5438 overwritten=0 applied=5438 ignored=0} inserts{seen=271750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:55:55.916054 4464 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Bootstrap complete.
I20250627 13:55:55.919911 4464 ts_tablet_manager.cc:1397] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Time spent bootstrapping tablet: real 26.661s user 19.435s sys 0.709s
I20250627 13:55:55.924459 4464 raft_consensus.cc:357] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:55:55.925371 4464 raft_consensus.cc:738] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 373a9aec2682487a8ff2a4f0d1ee87ab, State: Initialized, Role: FOLLOWER
I20250627 13:55:55.926126 4464 consensus_queue.cc:260] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5438, Last appended: 4.5438, Last appended by leader: 5438, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:55:55.936409 4464 ts_tablet_manager.cc:1428] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab: Time spent starting tablet: real 0.016s user 0.014s sys 0.004s
I20250627 13:55:56.642441 4827 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0204b303a14a41b096bfddffc4d04290)
I20250627 13:55:56.642724 4827 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:55:56.644647 4827 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611), 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207)
I20250627 13:55:56.721038 3806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" candidate_term: 5 candidate_status { last_received { term: 4 index: 6288 } } ignore_live_leader: false dest_uuid: "0204b303a14a41b096bfddffc4d04290" is_pre_election: true
I20250627 13:55:56.722996 2812 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" candidate_term: 5 candidate_status { last_received { term: 4 index: 6288 } } ignore_live_leader: false dest_uuid: "5f1fab906378491e919a4a3364723428" is_pre_election: true
I20250627 13:55:56.725103 4459 leader_election.cc:304] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 373a9aec2682487a8ff2a4f0d1ee87ab; no voters: 0204b303a14a41b096bfddffc4d04290, 5f1fab906378491e919a4a3364723428
I20250627 13:55:56.725910 4827 raft_consensus.cc:2747] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250627 13:55:57.051909 4826 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 373a9aec2682487a8ff2a4f0d1ee87ab is lagging by at least 1192 ops behind the committed index [suppressed 23 similar messages]
I20250627 13:55:57.235517 4827 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0204b303a14a41b096bfddffc4d04290)
I20250627 13:55:57.235705 4827 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:55:57.236404 4827 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611), 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207)
I20250627 13:55:57.241410 2813 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" candidate_term: 5 candidate_status { last_received { term: 4 index: 7138 } } ignore_live_leader: false dest_uuid: "5f1fab906378491e919a4a3364723428" is_pre_election: true
I20250627 13:55:57.242045 3806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" candidate_term: 5 candidate_status { last_received { term: 4 index: 7138 } } ignore_live_leader: false dest_uuid: "0204b303a14a41b096bfddffc4d04290" is_pre_election: true
I20250627 13:55:57.242774 4460 leader_election.cc:304] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 373a9aec2682487a8ff2a4f0d1ee87ab; no voters: 0204b303a14a41b096bfddffc4d04290, 5f1fab906378491e919a4a3364723428
I20250627 13:55:57.243501 4827 raft_consensus.cc:2747] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250627 13:55:57.319394 3739 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.140.137:40611, user_credentials={real_user=slave}} blocked reactor thread for 68671.3us
I20250627 13:56:01.712733 3786 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:56:01.762006 4509 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:01.793574 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:01.808183 2794 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:56:01.817629 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:01.903132 2663 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:01.923105 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:01.949124 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:02.078728 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:03.793913 1616 ts_manager.cc:284] Unset tserver state for 373a9aec2682487a8ff2a4f0d1ee87ab from MAINTENANCE_MODE
I20250627 13:56:03.794898 2728 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:56:03.795516 3996 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:56:03.833442 3306 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:56:04.185209 2859 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:56:04.231986 4578 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:56:04.248479 3853 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:56:04.269922 4140 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:56:04.339726 3447 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:56:04.577512 3163 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:56:05.801339 1586 maintenance_mode-itest.cc:745] Restarting batch of 2 tservers: 5f1fab906378491e919a4a3364723428,f9d895c7b77646baaff4f9a677e8c69d
I20250627 13:56:06.932665 1618 ts_manager.cc:295] Set tserver state for f9d895c7b77646baaff4f9a677e8c69d to MAINTENANCE_MODE
I20250627 13:56:08.108805 1616 ts_manager.cc:295] Set tserver state for 5f1fab906378491e919a4a3364723428 to MAINTENANCE_MODE
I20250627 13:56:09.771251 2663 tablet_service.cc:1423] Tablet server f9d895c7b77646baaff4f9a677e8c69d set to quiescing
I20250627 13:56:09.771456 2663 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:09.860514 2794 tablet_service.cc:1423] Tablet server 5f1fab906378491e919a4a3364723428 set to quiescing
I20250627 13:56:09.860769 2794 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:56:14.846521 2794 tablet_service.cc:1423] Tablet server 5f1fab906378491e919a4a3364723428 set to quiescing
I20250627 13:56:14.846762 2794 tablet_service.cc:1430] Tablet server has 0 leaders and 2 scanners
W20250627 13:56:15.167816 2870 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.140.130:42207, user_credentials={real_user=slave}} blocked reactor thread for 91806.7us
I20250627 13:56:18.157387 1614 maintenance_manager.cc:419] P c49acafbbcb74268bd768406057f1bce: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033401
I20250627 13:56:18.246102 2794 tablet_service.cc:1423] Tablet server 5f1fab906378491e919a4a3364723428 set to quiescing
I20250627 13:56:18.246371 2794 tablet_service.cc:1430] Tablet server has 0 leaders and 1 scanners
I20250627 13:56:18.281620 1613 maintenance_manager.cc:643] P c49acafbbcb74268bd768406057f1bce: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.123s user 0.042s sys 0.038s Metrics: {"bytes_written":7319,"cfile_init":1,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1157,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":14,"thread_start_us":84443,"threads_started":1,"wal-append.queue_time_us":84544}
I20250627 13:56:18.283224 1614 maintenance_manager.cc:419] P c49acafbbcb74268bd768406057f1bce: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 687 bytes on disk
I20250627 13:56:18.285499 1613 maintenance_manager.cc:643] P c49acafbbcb74268bd768406057f1bce: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20250627 13:56:20.455353 2794 tablet_service.cc:1423] Tablet server 5f1fab906378491e919a4a3364723428 set to quiescing
I20250627 13:56:20.455575 2794 tablet_service.cc:1430] Tablet server has 0 leaders and 1 scanners
I20250627 13:56:21.984875 2860 maintenance_manager.cc:419] P 5f1fab906378491e919a4a3364723428: Scheduling FlushMRSOp(8dca13501f184a25a621c700a1939986): perf score=0.053253
I20250627 13:56:24.303143 2794 tablet_service.cc:1423] Tablet server 5f1fab906378491e919a4a3364723428 set to quiescing
I20250627 13:56:24.303380 2794 tablet_service.cc:1430] Tablet server has 0 leaders and 1 scanners
I20250627 13:56:29.236724 2794 tablet_service.cc:1423] Tablet server 5f1fab906378491e919a4a3364723428 set to quiescing
I20250627 13:56:29.236963 2794 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:31.068595 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 2731
I20250627 13:56:31.277351 2854 generic_service.cc:196] Checking for leaks (request via RPC)
W20250627 13:56:31.420806 2870 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.140.130:42207, user_credentials={real_user=slave}} blocked reactor thread for 53481.4us
I20250627 13:56:31.683365 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 2731
W20250627 13:56:31.785416 3739 connection.cc:537] client connection to 127.1.140.137:40611 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:56:31.785550 3739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 22 similar messages]
I20250627 13:56:31.788825 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.137:40611
--local_ip_for_outbound_sockets=127.1.140.137
--tserver_master_addrs=127.1.140.190:35061
--webserver_port=34499
--webserver_interface=127.1.140.137
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:56:31.794549 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.137:40611: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250627 13:56:32.305313 5142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:32.305742 5142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:32.305912 5142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:32.314827 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.137:40611: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250627 13:56:32.317835 5142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:32.318141 5142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.137
I20250627 13:56:32.332907 5142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.137:40611
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data/info.pb
--webserver_interface=127.1.140.137
--webserver_port=34499
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.137
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:32.335291 5142 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:32.337720 5142 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:56:32.382318 5151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:32.387436 5152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:32.392164 5155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:32.411818 5142 server_base.cc:1048] running on GCE node
I20250627 13:56:32.413185 5142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:32.421140 5142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:56:32.422480 5142 hybrid_clock.cc:648] HybridClock initialized: now 1751032592422403 us; error 63 us; skew 500 ppm
I20250627 13:56:32.422952 5142 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:56:32.444955 5142 webserver.cc:469] Webserver started at http://127.1.140.137:34499/ using document root <none> and password file <none>
I20250627 13:56:32.446074 5142 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:32.446302 5142 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:32.460616 5142 fs_manager.cc:714] Time spent opening directory manager: real 0.011s user 0.001s sys 0.011s
I20250627 13:56:32.494576 5163 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:32.496421 5142 fs_manager.cc:730] Time spent opening block manager: real 0.034s user 0.010s sys 0.004s
I20250627 13:56:32.496665 5142 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/wal
uuid: "5f1fab906378491e919a4a3364723428"
format_stamp: "Formatted at 2025-06-27 13:54:21 on dist-test-slave-lj78"
I20250627 13:56:32.497874 5142 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:56:32.579268 5142 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:32.580384 5142 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:32.580744 5142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:32.588083 5142 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:56:32.617950 5170 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250627 13:56:32.634908 5142 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250627 13:56:32.635095 5142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.044s user 0.006s sys 0.021s
I20250627 13:56:32.635300 5142 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250627 13:56:32.639065 5142 ts_tablet_manager.cc:610] Registered 1 tablets
I20250627 13:56:32.639251 5142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.004s sys 0.000s
I20250627 13:56:32.640000 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap starting.
W20250627 13:56:32.805476 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.137:40611: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250627 13:56:33.299710 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.137:40611: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250627 13:56:33.845698 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.137:40611: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250627 13:56:33.981582 2899 meta_cache.cc:1510] marking tablet server 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611) as failed
W20250627 13:56:33.981866 2899 meta_cache.cc:302] tablet 8dca13501f184a25a621c700a1939986: replica 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.1.140.137:40611: connect: Connection refused (error 111)
I20250627 13:56:34.263257 5170 log.cc:826] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Log is configured to *not* fsync() on all Append() calls
I20250627 13:56:34.279752 5280 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.137:40611 every 8 connection(s)
I20250627 13:56:34.280066 5142 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.137:40611
I20250627 13:56:34.282039 5142 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data/info.pb
I20250627 13:56:34.285179 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 5142
I20250627 13:56:34.285385 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 2600
I20250627 13:56:34.349068 2723 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:56:34.362599 5281 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:56:34.363054 5281 heartbeater.cc:461] Registering TS with master...
I20250627 13:56:34.364166 5281 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:56:34.367074 1616 ts_manager.cc:194] Re-registered known tserver with Master: 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611)
I20250627 13:56:34.368733 1616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.137:49471
W20250627 13:56:34.389513 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
I20250627 13:56:34.708175 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 2600
I20250627 13:56:34.737510 1586 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
/tmp/dist-test-taskbElLHZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.140.136:39253
--local_ip_for_outbound_sockets=127.1.140.136
--tserver_master_addrs=127.1.140.190:35061
--webserver_port=33675
--webserver_interface=127.1.140.136
--builtin_ntp_servers=127.1.140.148:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250627 13:56:34.874701 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250627 13:56:35.372332 5281 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
W20250627 13:56:35.410387 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
W20250627 13:56:35.558166 5289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:56:35.558616 5289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250627 13:56:35.558805 5289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:56:35.570844 5289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250627 13:56:35.571120 5289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.140.136
I20250627 13:56:35.585062 5289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.140.148:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.140.136:39253
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data/info.pb
--webserver_interface=127.1.140.136
--webserver_port=33675
--tserver_master_addrs=127.1.140.190:35061
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.140.136
--log_dir=/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision f813aeead00d01482b21a4c28bdb5620b28c8aaf
build type FASTDEBUG
built by None at 27 Jun 2025 13:43:23 UTC on 24a791456cd2
build id 6796
ASAN enabled
I20250627 13:56:35.587357 5289 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250627 13:56:35.589586 5289 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250627 13:56:35.705773 5295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:35.706904 5294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250627 13:56:35.715102 5298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250627 13:56:35.715653 5289 server_base.cc:1048] running on GCE node
I20250627 13:56:35.716967 5289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250627 13:56:35.739813 5289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250627 13:56:35.742679 5289 hybrid_clock.cc:648] HybridClock initialized: now 1751032595742630 us; error 37 us; skew 500 ppm
I20250627 13:56:35.743141 5289 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250627 13:56:35.794881 5289 webserver.cc:469] Webserver started at http://127.1.140.136:33675/ using document root <none> and password file <none>
I20250627 13:56:35.796959 5289 fs_manager.cc:362] Metadata directory not provided
I20250627 13:56:35.797186 5289 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250627 13:56:35.817068 5289 fs_manager.cc:714] Time spent opening directory manager: real 0.014s user 0.005s sys 0.005s
I20250627 13:56:35.831997 5305 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250627 13:56:35.836349 5289 fs_manager.cc:730] Time spent opening block manager: real 0.016s user 0.003s sys 0.006s
I20250627 13:56:35.836586 5289 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data,/tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/wal
uuid: "f9d895c7b77646baaff4f9a677e8c69d"
format_stamp: "Formatted at 2025-06-27 13:54:21 on dist-test-slave-lj78"
I20250627 13:56:35.838428 5289 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/wal
metadata directory: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/wal
1 data directories: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250627 13:56:35.902263 5289 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250627 13:56:35.903528 5289 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250627 13:56:35.904807 5289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250627 13:56:35.915231 5289 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250627 13:56:35.922034 5289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250627 13:56:35.922214 5289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250627 13:56:35.922389 5289 ts_tablet_manager.cc:610] Registered 0 tablets
I20250627 13:56:35.922494 5289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20250627 13:56:35.967839 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
I20250627 13:56:36.376964 5036 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 1 ops behind the committed index
W20250627 13:56:36.514904 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250627 13:56:37.106184 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
I20250627 13:56:37.123852 5419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.140.136:39253 every 8 connection(s)
I20250627 13:56:37.124158 5289 rpc_server.cc:307] RPC server started. Bound to: 127.1.140.136:39253
I20250627 13:56:37.127810 5289 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data/info.pb
I20250627 13:56:37.134528 1586 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu as pid 5289
I20250627 13:56:37.221478 5420 heartbeater.cc:344] Connected to a master server at 127.1.140.190:35061
I20250627 13:56:37.221871 5420 heartbeater.cc:461] Registering TS with master...
I20250627 13:56:37.222941 5420 heartbeater.cc:507] Master 127.1.140.190:35061 requested a full tablet report, sending...
I20250627 13:56:37.225126 1616 ts_manager.cc:194] Re-registered known tserver with Master: f9d895c7b77646baaff4f9a677e8c69d (127.1.140.136:39253)
I20250627 13:56:37.226121 1616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.140.136:39257
W20250627 13:56:37.728963 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
I20250627 13:56:38.229470 5420 heartbeater.cc:499] Master 127.1.140.190:35061 was elected leader, sending a full tablet report...
W20250627 13:56:38.417683 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
I20250627 13:56:38.904287 3786 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:56:38.921352 4509 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:56:38.935554 5214 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:38.935905 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:38.948182 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:56:38.977468 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
I20250627 13:56:38.985179 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:39.002049 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:39.012102 5353 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:39.113729 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:39.286896 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap replaying log segment 1/3 (1.45M/8.00M this segment, stats: ops{read=854 overwritten=0 applied=851 ignored=0} inserts{seen=42500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250627 13:56:39.420003 5426 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 238 ops behind the committed index [suppressed 23 similar messages]
W20250627 13:56:39.680864 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
W20250627 13:56:40.193181 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
W20250627 13:56:40.842983 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250627 13:56:41.399303 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
W20250627 13:56:41.955895 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
W20250627 13:56:42.473438 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
I20250627 13:56:42.511591 5487 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 576 ops behind the committed index [suppressed 27 similar messages]
W20250627 13:56:42.963080 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
W20250627 13:56:43.383168 2870 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.140.130:42207, user_credentials={real_user=slave}} blocked reactor thread for 86793.7us
W20250627 13:56:43.538582 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20250627 13:56:44.111141 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
I20250627 13:56:44.291528 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap replaying log segment 1/3 (3.37M/8.00M this segment, stats: ops{read=1979 overwritten=0 applied=1975 ignored=0} inserts{seen=98650 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250627 13:56:44.644193 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
W20250627 13:56:45.122869 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
I20250627 13:56:45.603384 5487 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 867 ops behind the committed index [suppressed 28 similar messages]
W20250627 13:56:45.642874 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
I20250627 13:56:45.941648 3786 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:56:45.990499 4509 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:56:46.006328 5214 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:46.029503 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:46.044325 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:46.112952 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:46.114341 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:46.164377 5353 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:46.231989 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:56:46.251287 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
W20250627 13:56:46.821558 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
W20250627 13:56:47.397410 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
W20250627 13:56:48.001729 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
W20250627 13:56:48.550187 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
I20250627 13:56:48.632825 5494 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 1137 ops behind the committed index [suppressed 25 similar messages]
W20250627 13:56:49.102929 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 156: this message will repeat every 5th retry.
I20250627 13:56:49.293485 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap replaying log segment 1/3 (5.20M/8.00M this segment, stats: ops{read=3060 overwritten=0 applied=3059 ignored=0} inserts{seen=152850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250627 13:56:49.325688 2898 meta_cache.cc:1261] Time spent looking up entry by key: real 0.059s user 0.001s sys 0.000s
W20250627 13:56:49.650599 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 161: this message will repeat every 5th retry.
W20250627 13:56:50.202651 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 166: this message will repeat every 5th retry.
W20250627 13:56:50.745602 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 171: this message will repeat every 5th retry.
W20250627 13:56:51.328675 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 176: this message will repeat every 5th retry.
I20250627 13:56:51.636271 5557 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 1462 ops behind the committed index [suppressed 27 similar messages]
W20250627 13:56:51.785422 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 181: this message will repeat every 5th retry.
W20250627 13:56:52.291718 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 186: this message will repeat every 5th retry.
W20250627 13:56:52.838706 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 191: this message will repeat every 5th retry.
I20250627 13:56:53.096737 3786 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:56:53.108575 4509 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:56:53.146960 5214 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:53.264467 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:53.271983 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:53.364221 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:53.372360 5353 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:53.395552 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:56:53.437826 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:56:53.524438 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 196: this message will repeat every 5th retry.
W20250627 13:56:54.012264 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 201: this message will repeat every 5th retry.
I20250627 13:56:54.295150 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap replaying log segment 1/3 (7.14M/8.00M this segment, stats: ops{read=4198 overwritten=0 applied=4197 ignored=0} inserts{seen=209750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250627 13:56:54.531195 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 206: this message will repeat every 5th retry.
I20250627 13:56:54.715657 5609 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 1729 ops behind the committed index [suppressed 27 similar messages]
W20250627 13:56:55.051895 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 211: this message will repeat every 5th retry.
W20250627 13:56:55.454325 2870 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.140.130:42207, user_credentials={real_user=slave}} blocked reactor thread for 67591.1us
W20250627 13:56:55.532271 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 216: this message will repeat every 5th retry.
W20250627 13:56:56.019280 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 221: this message will repeat every 5th retry.
W20250627 13:56:56.629796 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 226: this message will repeat every 5th retry.
I20250627 13:56:56.746690 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap replayed 1/3 log segments. Stats: ops{read=4704 overwritten=0 applied=4703 ignored=0} inserts{seen=235050 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20250627 13:56:57.162058 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 231: this message will repeat every 5th retry.
W20250627 13:56:57.662279 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 236: this message will repeat every 5th retry.
I20250627 13:56:57.735718 5615 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 2065 ops behind the committed index [suppressed 28 similar messages]
W20250627 13:56:58.151764 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 241: this message will repeat every 5th retry.
W20250627 13:56:58.685034 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 246: this message will repeat every 5th retry.
W20250627 13:56:59.165412 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 251: this message will repeat every 5th retry.
I20250627 13:56:59.298592 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap replaying log segment 2/3 (1.07M/8.00M this segment, stats: ops{read=5335 overwritten=0 applied=5334 ignored=0} inserts{seen=266550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250627 13:56:59.636183 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 256: this message will repeat every 5th retry.
W20250627 13:57:00.249459 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 261: this message will repeat every 5th retry.
I20250627 13:57:00.516681 5214 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:00.525658 3786 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:57:00.535296 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:00.555023 4509 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:57:00.587834 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:00.599151 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:00.656302 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:00.657871 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:00.680239 5353 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:00.783674 5615 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 2317 ops behind the committed index [suppressed 28 similar messages]
W20250627 13:57:00.825539 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 266: this message will repeat every 5th retry.
W20250627 13:57:01.520628 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 271: this message will repeat every 5th retry.
W20250627 13:57:02.049723 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 276: this message will repeat every 5th retry.
W20250627 13:57:02.617694 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 281: this message will repeat every 5th retry.
W20250627 13:57:03.129153 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 286: this message will repeat every 5th retry.
W20250627 13:57:03.643483 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 291: this message will repeat every 5th retry.
I20250627 13:57:03.815114 5556 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 2641 ops behind the committed index [suppressed 26 similar messages]
W20250627 13:57:04.193449 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 296: this message will repeat every 5th retry.
I20250627 13:57:04.298942 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap replaying log segment 2/3 (2.87M/8.00M this segment, stats: ops{read=6394 overwritten=0 applied=6392 ignored=0} inserts{seen=319450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250627 13:57:04.677393 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 301: this message will repeat every 5th retry.
W20250627 13:57:05.168005 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 306: this message will repeat every 5th retry.
W20250627 13:57:05.685395 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 311: this message will repeat every 5th retry.
W20250627 13:57:06.243467 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 316: this message will repeat every 5th retry.
W20250627 13:57:06.748610 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 321: this message will repeat every 5th retry.
I20250627 13:57:06.830734 5684 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 2947 ops behind the committed index [suppressed 28 similar messages]
W20250627 13:57:07.332978 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 326: this message will repeat every 5th retry.
I20250627 13:57:07.722254 3786 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:57:07.801007 4509 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:57:07.820614 5214 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:57:07.847985 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 331: this message will repeat every 5th retry.
I20250627 13:57:07.874796 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:07.946714 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:08.005153 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:08.017972 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:08.034791 5353 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:08.084261 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:57:08.423864 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 336: this message will repeat every 5th retry.
W20250627 13:57:08.897361 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 341: this message will repeat every 5th retry.
I20250627 13:57:09.302281 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap replaying log segment 2/3 (4.54M/8.00M this segment, stats: ops{read=7377 overwritten=0 applied=7375 ignored=0} inserts{seen=368600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250627 13:57:09.422653 3854 maintenance_manager.cc:419] P 0204b303a14a41b096bfddffc4d04290: Scheduling FlushMRSOp(8dca13501f184a25a621c700a1939986): perf score=0.074569
W20250627 13:57:09.506518 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 346: this message will repeat every 5th retry.
I20250627 13:57:09.850605 5644 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 3237 ops behind the committed index [suppressed 27 similar messages]
W20250627 13:57:09.989074 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 351: this message will repeat every 5th retry.
W20250627 13:57:10.473237 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 356: this message will repeat every 5th retry.
W20250627 13:57:10.983502 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 361: this message will repeat every 5th retry.
W20250627 13:57:11.519944 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 366: this message will repeat every 5th retry.
W20250627 13:57:12.136036 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 371: this message will repeat every 5th retry.
W20250627 13:57:12.689478 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 376: this message will repeat every 5th retry.
I20250627 13:57:12.925092 5725 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 3508 ops behind the committed index [suppressed 28 similar messages]
W20250627 13:57:13.159446 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 381: this message will repeat every 5th retry.
W20250627 13:57:13.696296 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 386: this message will repeat every 5th retry.
I20250627 13:57:14.313439 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap replaying log segment 2/3 (6.01M/8.00M this segment, stats: ops{read=8240 overwritten=0 applied=8239 ignored=0} inserts{seen=411800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250627 13:57:14.585435 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 391: this message will repeat every 5th retry.
I20250627 13:57:14.959208 4509 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:57:14.984305 3786 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:57:14.993332 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:15.035075 5214 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:57:15.052345 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 396: this message will repeat every 5th retry.
I20250627 13:57:15.147826 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:15.193888 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:15.199137 5353 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:15.231253 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:15.293890 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:57:15.672649 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 401: this message will repeat every 5th retry.
I20250627 13:57:15.981238 5743 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 3728 ops behind the committed index [suppressed 24 similar messages]
W20250627 13:57:16.226998 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 406: this message will repeat every 5th retry.
W20250627 13:57:16.504935 3739 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.140.137:40611, user_credentials={real_user=slave}} blocked reactor thread for 73352.7us
W20250627 13:57:16.765411 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 411: this message will repeat every 5th retry.
W20250627 13:57:17.220520 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 416: this message will repeat every 5th retry.
W20250627 13:57:17.787730 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 421: this message will repeat every 5th retry.
W20250627 13:57:18.248521 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 426: this message will repeat every 5th retry.
W20250627 13:57:18.814349 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 431: this message will repeat every 5th retry.
I20250627 13:57:19.048719 5795 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 4001 ops behind the committed index [suppressed 29 similar messages]
W20250627 13:57:19.232122 2896 meta_cache.cc:1261] Time spent looking up entry by key: real 0.076s user 0.001s sys 0.000s
W20250627 13:57:19.303834 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 436: this message will repeat every 5th retry.
I20250627 13:57:19.305303 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap replaying log segment 2/3 (7.32M/8.00M this segment, stats: ops{read=9011 overwritten=0 applied=9007 ignored=0} inserts{seen=450200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250627 13:57:19.793512 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 441: this message will repeat every 5th retry.
W20250627 13:57:20.370196 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 446: this message will repeat every 5th retry.
W20250627 13:57:20.851809 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 451: this message will repeat every 5th retry.
I20250627 13:57:21.337252 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap replayed 2/3 log segments. Stats: ops{read=9408 overwritten=0 applied=9407 ignored=0} inserts{seen=470200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20250627 13:57:21.378545 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 456: this message will repeat every 5th retry.
W20250627 13:57:21.977128 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 461: this message will repeat every 5th retry.
I20250627 13:57:22.213284 5814 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 4268 ops behind the committed index [suppressed 28 similar messages]
W20250627 13:57:22.778383 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 466: this message will repeat every 5th retry.
W20250627 13:57:23.307534 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 471: this message will repeat every 5th retry.
I20250627 13:57:23.646231 3786 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:57:23.698534 4509 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250627 13:57:23.724025 5214 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:23.741976 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:23.779831 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:23.803330 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:23.854195 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:57:23.883052 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 476: this message will repeat every 5th retry.
I20250627 13:57:23.891913 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:23.929133 5353 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:24.314419 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap replaying log segment 3/3 (671.0K/2.80M this segment, stats: ops{read=9791 overwritten=0 applied=9790 ignored=0} inserts{seen=489350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250627 13:57:24.406165 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 481: this message will repeat every 5th retry.
W20250627 13:57:24.902841 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 486: this message will repeat every 5th retry.
I20250627 13:57:25.323660 5813 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 4515 ops behind the committed index [suppressed 26 similar messages]
W20250627 13:57:25.489734 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 491: this message will repeat every 5th retry.
W20250627 13:57:25.946753 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 496: this message will repeat every 5th retry.
W20250627 13:57:26.524953 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 501: this message will repeat every 5th retry.
W20250627 13:57:27.029548 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 506: this message will repeat every 5th retry.
W20250627 13:57:27.564563 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 511: this message will repeat every 5th retry.
W20250627 13:57:28.063961 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 516: this message will repeat every 5th retry.
I20250627 13:57:28.391709 5811 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 4788 ops behind the committed index [suppressed 28 similar messages]
W20250627 13:57:28.607847 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 521: this message will repeat every 5th retry.
W20250627 13:57:29.092576 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 526: this message will repeat every 5th retry.
I20250627 13:57:29.319764 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap replaying log segment 3/3 (1.85M/2.80M this segment, stats: ops{read=10496 overwritten=0 applied=10496 ignored=0} inserts{seen=524650 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250627 13:57:29.418040 2870 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.140.130:42207, user_credentials={real_user=slave}} blocked reactor thread for 78245.6us
W20250627 13:57:29.671506 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 531: this message will repeat every 5th retry.
W20250627 13:57:30.308199 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 536: this message will repeat every 5th retry.
W20250627 13:57:30.820848 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 541: this message will repeat every 5th retry.
W20250627 13:57:31.399362 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 546: this message will repeat every 5th retry.
I20250627 13:57:31.433557 5873 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 5048 ops behind the committed index [suppressed 27 similar messages]
W20250627 13:57:31.869794 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 551: this message will repeat every 5th retry.
I20250627 13:57:31.921007 3786 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250627 13:57:31.973366 4509 tablet_service.cc:1430] Tablet server has 0 leaders and 2 scanners
I20250627 13:57:32.002794 5214 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:32.031318 4073 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:32.074734 3095 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:32.105494 3379 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:32.137296 3238 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:32.178076 3929 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250627 13:57:32.186478 5353 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250627 13:57:32.549995 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 556: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/maintenance_mode-itest.cc:753: Failure
Failed
Bad status: Timed out: cluster didn't become healthy: Running 'cluster ksck 127.1.140.190:35061' did not succeed in 60 seconds: Runtime error: error running 'cluster ksck 127.1.140.190:35061': stdout: Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
c49acafbbcb74268bd768406057f1bce | 127.1.140.190:35061 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.1.140.148:44053 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server States
Server | State
----------------------------------+------------------
5f1fab906378491e919a4a3364723428 | MAINTENANCE_MODE
f9d895c7b77646baaff4f9a677e8c69d | MAINTENANCE_MODE
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0204b303a14a41b096bfddffc4d04290 | 127.1.140.130:42207 | HEALTHY | /L0 | 1 | 0
373a9aec2682487a8ff2a4f0d1ee87ab | 127.1.140.135:34009 | HEALTHY | /L1 | 0 | 2
5f1fab906378491e919a4a3364723428 | 127.1.140.137:40611 | HEALTHY | /L2 | 0 | 0
978726a17dd44a5891b685d3b2ae2b99 | 127.1.140.129:35871 | HEALTHY | /L0 | 0 | 0
99bf3fe9a41d4ce1b61e440b31778689 | 127.1.140.132:44007 | HEALTHY | /L1 | 0 | 0
aa90f1d263644537b32ad1a4595aef92 | 127.1.140.134:36295 | HEALTHY | /L1 | 0 | 0
e879d1cb3a7c4226884531badda84252 | 127.1.140.131:33589 | HEALTHY | /L0 | 0 | 0
f9d895c7b77646baaff4f9a677e8c69d | 127.1.140.136:39253 | HEALTHY | /L2 | 0 | 0
ffa5b64156aa4e0dbef89b3fb8111e44 | 127.1.140.133:44765 | HEALTHY | /L1 | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/L2 | 2
/L0 | 3
/L1 | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 9 server(s) checked
local_ip_for_outbound_sockets | 127.1.140.129 | experimental | 127.1.140.129:35871
local_ip_for_outbound_sockets | 127.1.140.130 | experimental | 127.1.140.130:42207
local_ip_for_outbound_sockets | 127.1.140.131 | experimental | 127.1.140.131:33589
local_ip_for_outbound_sockets | 127.1.140.132 | experimental | 127.1.140.132:44007
local_ip_for_outbound_sockets | 127.1.140.133 | experimental | 127.1.140.133:44765
local_ip_for_outbound_sockets | 127.1.140.134 | experimental | 127.1.140.134:36295
local_ip_for_outbound_sockets | 127.1.140.135 | experimental | 127.1.140.135:34009
local_ip_for_outbound_sockets | 127.1.140.136 | experimental | 127.1.140.136:39253
local_ip_for_outbound_sockets | 127.1.140.137 | experimental | 127.1.140.137:40611
never_fsync | true | unsafe,advanced | all 9 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 9 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked
server_dump_info_format | pb | hidden | all 9 server(s) checked
server_dump_info_path | /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-0/data/info.pb | hidden | 127.1.140.129:35871
server_dump_info_path | /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/data/info.pb | hidden | 127.1.140.130:42207
server_dump_info_path | /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-2/data/info.pb | hidden | 127.1.140.131:33589
server_dump_info_path | /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-3/data/info.pb | hidden | 127.1.140.132:44007
server_dump_info_path | /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-4/data/info.pb | hidden | 127.1.140.133:44765
server_dump_info_path | /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-5/data/info.pb | hidden | 127.1.140.134:36295
server_dump_info_path | /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-6/data/info.pb | hidden | 127.1.140.135:34009
server_dump_info_path | /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-7/data/info.pb | hidden | 127.1.140.136:39253
server_dump_info_path | /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-8/data/info.pb | hidden | 127.1.140.137:40611
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.1.140.148:44053 | all 9 server(s) checked
time_source | builtin | all 9 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------
1.18.0-SNAPSHOT | all 10 server(s) checked
Tablet Summary
Tablet 8dca13501f184a25a621c700a1939986 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): not running
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap replaying log segment 3/3 (1.85M/2.80M this segment, stats: ops{read=10496 overwritten=0 applied=10496 ignored=0} inserts{seen=524650 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): RUNNING
0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): RUNNING [LEADER]
All reported replicas are:
A = 5f1fab906378491e919a4a3364723428
B = 373a9aec2682487a8ff2a4f0d1ee87ab
C = 0204b303a14a41b096bfddffc4d04290
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C | 4 | -1 | Yes
B | A B C* | 4 | -1 | Yes
C | A B C* | 4 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 9
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
, stderr: W20250627 13:57:30.998128 5871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250627 13:57:30.999509 5871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250627 13:57:31.045104 5871 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Runtime error: ksck discovered errors
: /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu: process exited with non-zero status 1
W20250627 13:57:33.065359 3739 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 -> Peer 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611): Couldn't send request to peer 5f1fab906378491e919a4a3364723428. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 561: this message will repeat every 5th retry.
I20250627 13:57:33.264884 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap replayed 3/3 log segments. Stats: ops{read=11052 overwritten=0 applied=11052 ignored=0} inserts{seen=552450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250627 13:57:33.266141 5170 tablet_bootstrap.cc:492] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Bootstrap complete.
I20250627 13:57:33.271783 5170 ts_tablet_manager.cc:1397] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Time spent bootstrapping tablet: real 60.632s user 40.115s sys 1.300s
I20250627 13:57:33.281687 5170 raft_consensus.cc:357] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:57:33.284373 5170 raft_consensus.cc:738] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1fab906378491e919a4a3364723428, State: Initialized, Role: FOLLOWER
I20250627 13:57:33.285514 5170 consensus_queue.cc:260] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11052, Last appended: 4.11052, Last appended by leader: 11052, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:57:33.310447 5170 ts_tablet_manager.cc:1428] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428: Time spent starting tablet: real 0.038s user 0.014s sys 0.004s
I20250627 13:57:33.705127 5925 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0204b303a14a41b096bfddffc4d04290)
I20250627 13:57:33.705421 5925 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:57:33.707052 5925 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009), 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207)
I20250627 13:57:33.779798 4529 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "5f1fab906378491e919a4a3364723428" candidate_term: 5 candidate_status { last_received { term: 4 index: 11477 } } ignore_live_leader: false dest_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" is_pre_election: true
I20250627 13:57:33.784194 3806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "5f1fab906378491e919a4a3364723428" candidate_term: 5 candidate_status { last_received { term: 4 index: 11477 } } ignore_live_leader: false dest_uuid: "0204b303a14a41b096bfddffc4d04290" is_pre_election: true
I20250627 13:57:33.784933 5166 leader_election.cc:304] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f1fab906378491e919a4a3364723428; no voters: 0204b303a14a41b096bfddffc4d04290, 373a9aec2682487a8ff2a4f0d1ee87ab
I20250627 13:57:34.145274 5925 raft_consensus.cc:2747] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250627 13:57:34.436236 5938 consensus_queue.cc:786] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290 [LEADER]: Peer 5f1fab906378491e919a4a3364723428 is lagging by at least 3840 ops behind the committed index [suppressed 19 similar messages]
I20250627 13:57:35.817276 5940 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0204b303a14a41b096bfddffc4d04290)
I20250627 13:57:35.817485 5940 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:57:35.818401 5940 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009), 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207)
I20250627 13:57:35.819299 4529 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "5f1fab906378491e919a4a3364723428" candidate_term: 5 candidate_status { last_received { term: 4 index: 14452 } } ignore_live_leader: false dest_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" is_pre_election: true
I20250627 13:57:35.819453 3806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "5f1fab906378491e919a4a3364723428" candidate_term: 5 candidate_status { last_received { term: 4 index: 14452 } } ignore_live_leader: false dest_uuid: "0204b303a14a41b096bfddffc4d04290" is_pre_election: true
I20250627 13:57:35.820274 5166 leader_election.cc:304] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f1fab906378491e919a4a3364723428; no voters: 0204b303a14a41b096bfddffc4d04290, 373a9aec2682487a8ff2a4f0d1ee87ab
I20250627 13:57:35.820729 5940 raft_consensus.cc:2747] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250627 13:57:36.660077 5943 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0204b303a14a41b096bfddffc4d04290)
I20250627 13:57:36.660372 5943 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:57:36.661623 4529 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "5f1fab906378491e919a4a3364723428" candidate_term: 5 candidate_status { last_received { term: 4 index: 15727 } } ignore_live_leader: false dest_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" is_pre_election: true
I20250627 13:57:36.673967 3806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "5f1fab906378491e919a4a3364723428" candidate_term: 5 candidate_status { last_received { term: 4 index: 15727 } } ignore_live_leader: false dest_uuid: "0204b303a14a41b096bfddffc4d04290" is_pre_election: true
I20250627 13:57:36.674598 5166 leader_election.cc:304] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f1fab906378491e919a4a3364723428; no voters: 0204b303a14a41b096bfddffc4d04290, 373a9aec2682487a8ff2a4f0d1ee87ab
I20250627 13:57:36.680423 5943 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009), 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207)
I20250627 13:57:36.680619 5943 raft_consensus.cc:2747] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250627 13:57:36.997586 3741 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.140.135:34009, user_credentials={real_user=slave}} blocked reactor thread for 47172.4us
I20250627 13:57:47.371847 3742 maintenance_manager.cc:643] P 0204b303a14a41b096bfddffc4d04290: FlushMRSOp(8dca13501f184a25a621c700a1939986) complete. Timing: real 37.947s user 23.932s sys 0.803s Metrics: {"bytes_written":20340386,"cfile_init":1,"compiler_manager_pool.queue_time_us":9765,"compiler_manager_pool.run_cpu_time_us":692434,"compiler_manager_pool.run_wall_time_us":2672895,"dirs.queue_time_us":11218,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":2082,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":176967,"lbm_writes_1-10_ms":10,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":875,"peak_mem_usage":0,"rows_written":733500,"thread_start_us":43242,"threads_started":3,"wal-append.queue_time_us":1122}
I20250627 13:57:47.373654 3854 maintenance_manager.cc:419] P 0204b303a14a41b096bfddffc4d04290: Scheduling LogGCOp(8dca13501f184a25a621c700a1939986): free 25166404 bytes of WAL
I20250627 13:57:47.373955 3742 log_reader.cc:385] T 8dca13501f184a25a621c700a1939986: removed 3 log segments from log reader
I20250627 13:57:47.374047 3742 log.cc:1079] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Deleting log segment in path: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal/wals/8dca13501f184a25a621c700a1939986/wal-000000001 (ops 1-4890)
I20250627 13:57:47.374167 3742 log.cc:1079] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Deleting log segment in path: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal/wals/8dca13501f184a25a621c700a1939986/wal-000000002 (ops 4891-9511)
I20250627 13:57:47.374220 3742 log.cc:1079] T 8dca13501f184a25a621c700a1939986 P 0204b303a14a41b096bfddffc4d04290: Deleting log segment in path: /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0/minicluster-data/ts-1/wal/wals/8dca13501f184a25a621c700a1939986/wal-000000003 (ops 9512-14130)
I20250627 13:57:47.380424 3742 maintenance_manager.cc:643] P 0204b303a14a41b096bfddffc4d04290: LogGCOp(8dca13501f184a25a621c700a1939986) complete. Timing: real 0.007s user 0.000s sys 0.003s Metrics: {}
I20250627 13:57:47.380764 3854 maintenance_manager.cc:419] P 0204b303a14a41b096bfddffc4d04290: Scheduling UndoDeltaBlockGCOp(8dca13501f184a25a621c700a1939986): 8404302 bytes on disk
I20250627 13:57:47.381597 3742 maintenance_manager.cc:643] P 0204b303a14a41b096bfddffc4d04290: UndoDeltaBlockGCOp(8dca13501f184a25a621c700a1939986) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":155,"lbm_reads_lt_1ms":4}
I20250627 13:57:48.859692 1586 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20250627 13:57:48.859911 1586 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 978726a17dd44a5891b685d3b2ae2b99 and pid 4008
************************ BEGIN STACKS **************************
[New LWP 4010]
[New LWP 4011]
[New LWP 4012]
[New LWP 4013]
[New LWP 4020]
[New LWP 4021]
[New LWP 4022]
[New LWP 4026]
[New LWP 4027]
[New LWP 4028]
[New LWP 4029]
[New LWP 4030]
[New LWP 4031]
[New LWP 4032]
[New LWP 4033]
[New LWP 4034]
[New LWP 4035]
[New LWP 4036]
[New LWP 4037]
[New LWP 4038]
[New LWP 4039]
[New LWP 4040]
[New LWP 4041]
[New LWP 4042]
[New LWP 4043]
[New LWP 4044]
[New LWP 4045]
[New LWP 4046]
[New LWP 4047]
[New LWP 4048]
[New LWP 4049]
[New LWP 4050]
[New LWP 4051]
[New LWP 4052]
[New LWP 4053]
[New LWP 4054]
[New LWP 4055]
[New LWP 4056]
[New LWP 4057]
[New LWP 4058]
[New LWP 4059]
[New LWP 4060]
[New LWP 4061]
[New LWP 4062]
[New LWP 4063]
[New LWP 4064]
[New LWP 4065]
[New LWP 4066]
[New LWP 4067]
[New LWP 4068]
[New LWP 4069]
[New LWP 4070]
[New LWP 4071]
[New LWP 4072]
[New LWP 4073]
[New LWP 4074]
[New LWP 4075]
[New LWP 4076]
[New LWP 4077]
[New LWP 4078]
[New LWP 4079]
[New LWP 4080]
[New LWP 4081]
[New LWP 4082]
[New LWP 4083]
[New LWP 4084]
[New LWP 4085]
[New LWP 4086]
[New LWP 4087]
[New LWP 4088]
[New LWP 4089]
[New LWP 4090]
[New LWP 4091]
[New LWP 4092]
[New LWP 4093]
[New LWP 4094]
[New LWP 4095]
[New LWP 4096]
[New LWP 4097]
[New LWP 4098]
[New LWP 4099]
[New LWP 4100]
[New LWP 4101]
[New LWP 4103]
[New LWP 4104]
[New LWP 4105]
[New LWP 4106]
[New LWP 4107]
[New LWP 4108]
[New LWP 4109]
[New LWP 4110]
[New LWP 4111]
[New LWP 4112]
[New LWP 4113]
[New LWP 4114]
[New LWP 4115]
[New LWP 4116]
[New LWP 4117]
[New LWP 4118]
[New LWP 4119]
[New LWP 4120]
[New LWP 4121]
[New LWP 4122]
[New LWP 4123]
[New LWP 4124]
[New LWP 4125]
[New LWP 4126]
[New LWP 4127]
[New LWP 4128]
[New LWP 4129]
[New LWP 4130]
[New LWP 4131]
[New LWP 4132]
[New LWP 4133]
[New LWP 4134]
[New LWP 4135]
[New LWP 4137]
[New LWP 4138]
[New LWP 4139]
[New LWP 4140]
[New LWP 4141]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007fedb61f2d50 in ?? ()
Id Target Id Frame
* 1 LWP 4008 "kudu" 0x00007fedb61f2d50 in ?? ()
2 LWP 4010 "kudu" 0x00007fedb61eefb9 in ?? ()
3 LWP 4011 "kudu" 0x00007fedb61eefb9 in ?? ()
4 LWP 4012 "kudu" 0x00007fedb61eefb9 in ?? ()
5 LWP 4013 "kernel-watcher-" 0x00007fedb61eefb9 in ?? ()
6 LWP 4020 "ntp client-4020" 0x00007fedb61f29e2 in ?? ()
7 LWP 4021 "file cache-evic" 0x00007fedb61eefb9 in ?? ()
8 LWP 4022 "sq_acceptor" 0x00007fedaef14cb9 in ?? ()
9 LWP 4026 "rpc reactor-402" 0x00007fedaef21a47 in ?? ()
10 LWP 4027 "rpc reactor-402" 0x00007fedaef21a47 in ?? ()
11 LWP 4028 "rpc reactor-402" 0x00007fedaef21a47 in ?? ()
12 LWP 4029 "rpc reactor-402" 0x00007fedaef21a47 in ?? ()
13 LWP 4030 "MaintenanceMgr " 0x00007fedb61eead3 in ?? ()
14 LWP 4031 "txn-status-mana" 0x00007fedb61eefb9 in ?? ()
15 LWP 4032 "collect_and_rem" 0x00007fedb61eefb9 in ?? ()
16 LWP 4033 "tc-session-exp-" 0x00007fedb61eefb9 in ?? ()
17 LWP 4034 "rpc worker-4034" 0x00007fedb61eead3 in ?? ()
18 LWP 4035 "rpc worker-4035" 0x00007fedb61eead3 in ?? ()
19 LWP 4036 "rpc worker-4036" 0x00007fedb61eead3 in ?? ()
20 LWP 4037 "rpc worker-4037" 0x00007fedb61eead3 in ?? ()
21 LWP 4038 "rpc worker-4038" 0x00007fedb61eead3 in ?? ()
22 LWP 4039 "rpc worker-4039" 0x00007fedb61eead3 in ?? ()
23 LWP 4040 "rpc worker-4040" 0x00007fedb61eead3 in ?? ()
24 LWP 4041 "rpc worker-4041" 0x00007fedb61eead3 in ?? ()
25 LWP 4042 "rpc worker-4042" 0x00007fedb61eead3 in ?? ()
26 LWP 4043 "rpc worker-4043" 0x00007fedb61eead3 in ?? ()
27 LWP 4044 "rpc worker-4044" 0x00007fedb61eead3 in ?? ()
28 LWP 4045 "rpc worker-4045" 0x00007fedb61eead3 in ?? ()
29 LWP 4046 "rpc worker-4046" 0x00007fedb61eead3 in ?? ()
30 LWP 4047 "rpc worker-4047" 0x00007fedb61eead3 in ?? ()
31 LWP 4048 "rpc worker-4048" 0x00007fedb61eead3 in ?? ()
32 LWP 4049 "rpc worker-4049" 0x00007fedb61eead3 in ?? ()
33 LWP 4050 "rpc worker-4050" 0x00007fedb61eead3 in ?? ()
34 LWP 4051 "rpc worker-4051" 0x00007fedb61eead3 in ?? ()
35 LWP 4052 "rpc worker-4052" 0x00007fedb61eead3 in ?? ()
36 LWP 4053 "rpc worker-4053" 0x00007fedb61eead3 in ?? ()
37 LWP 4054 "rpc worker-4054" 0x00007fedb61eead3 in ?? ()
38 LWP 4055 "rpc worker-4055" 0x00007fedb61eead3 in ?? ()
39 LWP 4056 "rpc worker-4056" 0x00007fedb61eead3 in ?? ()
40 LWP 4057 "rpc worker-4057" 0x00007fedb61eead3 in ?? ()
41 LWP 4058 "rpc worker-4058" 0x00007fedb61eead3 in ?? ()
42 LWP 4059 "rpc worker-4059" 0x00007fedb61eead3 in ?? ()
43 LWP 4060 "rpc worker-4060" 0x00007fedb61eead3 in ?? ()
44 LWP 4061 "rpc worker-4061" 0x00007fedb61eead3 in ?? ()
45 LWP 4062 "rpc worker-4062" 0x00007fedb61eead3 in ?? ()
46 LWP 4063 "rpc worker-4063" 0x00007fedb61eead3 in ?? ()
47 LWP 4064 "rpc worker-4064" 0x00007fedb61eead3 in ?? ()
48 LWP 4065 "rpc worker-4065" 0x00007fedb61eead3 in ?? ()
49 LWP 4066 "rpc worker-4066" 0x00007fedb61eead3 in ?? ()
50 LWP 4067 "rpc worker-4067" 0x00007fedb61eead3 in ?? ()
51 LWP 4068 "rpc worker-4068" 0x00007fedb61eead3 in ?? ()
52 LWP 4069 "rpc worker-4069" 0x00007fedb61eead3 in ?? ()
53 LWP 4070 "rpc worker-4070" 0x00007fedb61eead3 in ?? ()
54 LWP 4071 "rpc worker-4071" 0x00007fedb61eead3 in ?? ()
55 LWP 4072 "rpc worker-4072" 0x00007fedb61eead3 in ?? ()
56 LWP 4073 "rpc worker-4073" 0x00007fedb61eead3 in ?? ()
57 LWP 4074 "rpc worker-4074" 0x00007fedb61eead3 in ?? ()
58 LWP 4075 "rpc worker-4075" 0x00007fedb61eead3 in ?? ()
59 LWP 4076 "rpc worker-4076" 0x00007fedb61eead3 in ?? ()
60 LWP 4077 "rpc worker-4077" 0x00007fedb61eead3 in ?? ()
61 LWP 4078 "rpc worker-4078" 0x00007fedb61eead3 in ?? ()
62 LWP 4079 "rpc worker-4079" 0x00007fedb61eead3 in ?? ()
63 LWP 4080 "rpc worker-4080" 0x00007fedb61eead3 in ?? ()
64 LWP 4081 "rpc worker-4081" 0x00007fedb61eead3 in ?? ()
65 LWP 4082 "rpc worker-4082" 0x00007fedb61eead3 in ?? ()
66 LWP 4083 "rpc worker-4083" 0x00007fedb61eead3 in ?? ()
67 LWP 4084 "rpc worker-4084" 0x00007fedb61eead3 in ?? ()
68 LWP 4085 "rpc worker-4085" 0x00007fedb61eead3 in ?? ()
69 LWP 4086 "rpc worker-4086" 0x00007fedb61eead3 in ?? ()
70 LWP 4087 "rpc worker-4087" 0x00007fedb61eead3 in ?? ()
71 LWP 4088 "rpc worker-4088" 0x00007fedb61eead3 in ?? ()
72 LWP 4089 "rpc worker-4089" 0x00007fedb61eead3 in ?? ()
73 LWP 4090 "rpc worker-4090" 0x00007fedb61eead3 in ?? ()
74 LWP 4091 "rpc worker-4091" 0x00007fedb61eead3 in ?? ()
75 LWP 4092 "rpc worker-4092" 0x00007fedb61eead3 in ?? ()
76 LWP 4093 "rpc worker-4093" 0x00007fedb61eead3 in ?? ()
77 LWP 4094 "rpc worker-4094" 0x00007fedb61eead3 in ?? ()
78 LWP 4095 "rpc worker-4095" 0x00007fedb61eead3 in ?? ()
79 LWP 4096 "rpc worker-4096" 0x00007fedb61eead3 in ?? ()
80 LWP 4097 "rpc worker-4097" 0x00007fedb61eead3 in ?? ()
81 LWP 4098 "rpc worker-4098" 0x00007fedb61eead3 in ?? ()
82 LWP 4099 "rpc worker-4099" 0x00007fedb61eead3 in ?? ()
83 LWP 4100 "rpc worker-4100" 0x00007fedb61eead3 in ?? ()
84 LWP 4101 "rpc worker-4101" 0x00007fedb61eead3 in ?? ()
85 LWP 4103 "rpc worker-4103" 0x00007fedb61eead3 in ?? ()
86 LWP 4104 "rpc worker-4104" 0x00007fedb61eead3 in ?? ()
87 LWP 4105 "rpc worker-4105" 0x00007fedb61eead3 in ?? ()
88 LWP 4106 "rpc worker-4106" 0x00007fedb61eead3 in ?? ()
89 LWP 4107 "rpc worker-4107" 0x00007fedb61eead3 in ?? ()
90 LWP 4108 "rpc worker-4108" 0x00007fedb61eead3 in ?? ()
91 LWP 4109 "rpc worker-4109" 0x00007fedb61eead3 in ?? ()
92 LWP 4110 "rpc worker-4110" 0x00007fedb61eead3 in ?? ()
93 LWP 4111 "rpc worker-4111" 0x00007fedb61eead3 in ?? ()
94 LWP 4112 "rpc worker-4112" 0x00007fedb61eead3 in ?? ()
95 LWP 4113 "rpc worker-4113" 0x00007fedb61eead3 in ?? ()
96 LWP 4114 "rpc worker-4114" 0x00007fedb61eead3 in ?? ()
97 LWP 4115 "rpc worker-4115" 0x00007fedb61eead3 in ?? ()
98 LWP 4116 "rpc worker-4116" 0x00007fedb61eead3 in ?? ()
99 LWP 4117 "rpc worker-4117" 0x00007fedb61eead3 in ?? ()
100 LWP 4118 "rpc worker-4118" 0x00007fedb61eead3 in ?? ()
101 LWP 4119 "rpc worker-4119" 0x00007fedb61eead3 in ?? ()
102 LWP 4120 "rpc worker-4120" 0x00007fedb61eead3 in ?? ()
103 LWP 4121 "rpc worker-4121" 0x00007fedb61eead3 in ?? ()
104 LWP 4122 "rpc worker-4122" 0x00007fedb61eead3 in ?? ()
105 LWP 4123 "rpc worker-4123" 0x00007fedb61eead3 in ?? ()
106 LWP 4124 "rpc worker-4124" 0x00007fedb61eead3 in ?? ()
107 LWP 4125 "rpc worker-4125" 0x00007fedb61eead3 in ?? ()
108 LWP 4126 "rpc worker-4126" 0x00007fedb61eead3 in ?? ()
109 LWP 4127 "rpc worker-4127" 0x00007fedb61eead3 in ?? ()
110 LWP 4128 "rpc worker-4128" 0x00007fedb61eead3 in ?? ()
111 LWP 4129 "rpc worker-4129" 0x00007fedb61eead3 in ?? ()
112 LWP 4130 "rpc worker-4130" 0x00007fedb61eead3 in ?? ()
113 LWP 4131 "rpc worker-4131" 0x00007fedb61eead3 in ?? ()
114 LWP 4132 "rpc worker-4132" 0x00007fedb61eead3 in ?? ()
115 LWP 4133 "rpc worker-4133" 0x00007fedb61eead3 in ?? ()
116 LWP 4134 "rpc worker-4134" 0x00007fedb61eead3 in ?? ()
117 LWP 4135 "diag-logger-413" 0x00007fedb61eefb9 in ?? ()
118 LWP 4137 "result-tracker-" 0x00007fedb61eefb9 in ?? ()
119 LWP 4138 "excess-log-dele" 0x00007fedb61eefb9 in ?? ()
120 LWP 4139 "acceptor-4139" 0x00007fedaef230c7 in ?? ()
121 LWP 4140 "heartbeat-4140" 0x00007fedb61eefb9 in ?? ()
122 LWP 4141 "maintenance_sch" 0x00007fedb61eefb9 in ?? ()
Thread 122 (LWP 4141):
#0 0x00007fedb61eefb9 in ?? ()
#1 0x00007fed6ba966e0 in ?? ()
#2 0x0000000000000298 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007fed6ba96730 in ?? ()
#6 0x0000000000000530 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 4140):
#0 0x00007fedb61eefb9 in ?? ()
#1 0x26b5f4687947e954 in ?? ()
#2 0x00000000000000a5 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019e80 in ?? ()
#5 0x00007fed6c2ae5f0 in ?? ()
#6 0x000000000000014a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 4139):
#0 0x00007fedaef230c7 in ?? ()
#1 0x00006140000214a8 in ?? ()
#2 0x00007fed6cad5d90 in ?? ()
#3 0x00007fed6cad5dc0 in ?? ()
#4 0x00007fed6cad5ec0 in ?? ()
#5 0x00007fed6cad5db0 in ?? ()
#6 0x00007fed6cad5e20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 4138):
#0 0x00007fedb61eefb9 in ?? ()
#1 0x00007fed6d2eff40 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffe68329350 in ?? ()
#5 0x00007fed6d2eff30 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 4137):
#0 0x00007fedb61eefb9 in ?? ()
#1 0x00007fed6db08100 in ?? ()
#2 0x00000000000000a6 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007fed6db080f0 in ?? ()
#6 0x000000000000014c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 4135):
#0 0x00007fedb61eefb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 4134):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000027 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007fed6eb37e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fed6eb37eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007fedb61ee770 in ?? ()
#10 0x00007fed6eb37eb0 in ?? ()
#11 0x00000ffdadd66fd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 4133):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000015 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007fed6f34fe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fed6f34feb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007fedb61ee770 in ?? ()
#10 0x00007fed6f34feb0 in ?? ()
#11 0x00000ffdade69fd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 4132):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 4131):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 4130):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 4129):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 4128):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 4127):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 4126):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 4125):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 4124):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 4123):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 4122):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 4121):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 4120):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 4119):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 4118):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 4117):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 4116):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 4115):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 4114):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 4113):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 4112):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 4111):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 4110):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 4109):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 4108):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 4107):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 4106):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 4105):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 4104):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 4103):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 4101):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 4100):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 4099):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 4098):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 4097):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 4096):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 4095):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 4094):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 4093):
#0 0x00007fedb61eead3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a688c in ?? ()
#4 0x00007fed82f08e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fed82f08eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a6840 in ?? ()
#9 0x00007fedb61ee770 in ?? ()
#10 0x00007fed82f08eb0 in ?? ()
#11 0x00000ffdb05e11d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 4092):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 4091):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 4090):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 4089):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 4088):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 4087):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 4086):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 4085):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 4084):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 4083):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 4082):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 4081):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 4080):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 4079):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 4078):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 4077):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 4076):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 4075):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 4074):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 4073):
#0 0x00007fedb61eead3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012003c in ?? ()
#4 0x00007fed8d0e9e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fed8d0e9eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00011fff0 in ?? ()
#9 0x00007fedb61ee770 in ?? ()
#10 0x00007fed8d0e9eb0 in ?? ()
#11 0x00000ffdb1a1d3d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 4072):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 4071):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 4070):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 4069):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 4068):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 4067):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 4066):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 4065):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 4064):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 4063):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 4062):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 4061):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 4060):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 4059):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 4058):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 4057):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 4056):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 4055):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 4054):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 4053):
#0 0x00007fedb61eead3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007fed972cae90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fed972caeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007fedb61ee770 in ?? ()
#10 0x00007fed972caeb0 in ?? ()
#11 0x00000ffdb2e595d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 4052):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 4051):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 4050):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 4049):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 4048):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 4047):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 4046):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 4045):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 4044):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 4043):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 4042):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 4041):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 4040):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 4039):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 4038):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 4037):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 4036):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 4035):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 4034):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 4033):
#0 0x00007fedb61eefb9 in ?? ()
#1 0x00007feda0cadcc0 in ?? ()
#2 0x0000000000000010 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a720 in ?? ()
#5 0x00007feda0cadcb0 in ?? ()
#6 0x0000000000000020 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 4032):
#0 0x00007fedb61eefb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 4031):
#0 0x00007fedb61eefb9 in ?? ()
#1 0x00007feda1caf240 in ?? ()
#2 0x0000000000000010 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007feda1caf230 in ?? ()
#6 0x0000000000000020 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 4030):
#0 0x00007fedb61eead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 4029):
#0 0x00007fedaef21a47 in ?? ()
#1 0x00007feda2cb2320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007feda2cb2310 in ?? ()
#4 0x00007feda2cb2520 in ?? ()
#5 0x00007feda2cb2360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007feda2cb23e0 in ?? ()
#8 0x00007fedb159425d in ?? ()
#9 0x3fb9746e40685000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x00000000da000000 in ?? ()
#14 0x00007fed00000000 in ?? ()
#15 0x41da17a869707644 in ?? ()
#16 0x00000ffe3458e47c in ?? ()
#17 0x00007feda2cb23c0 in ?? ()
#18 0x00007fedb1598ba3 in ?? ()
#19 0x00007feda2cb2390 in ?? ()
#20 0x3fb9746e40685000 in ?? ()
#21 0x00000000a2cb23e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb9746e40685000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 4028):
#0 0x00007fedaef21a47 in ?? ()
#1 0x00007feda34b3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007feda34b3310 in ?? ()
#4 0x00007feda34b3520 in ?? ()
#5 0x00007feda34b3360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007feda34b33e0 in ?? ()
#8 0x00007fedb159425d in ?? ()
#9 0x3fb9890f1932c000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x00000000ed000000 in ?? ()
#14 0x00007fed00000000 in ?? ()
#15 0x41da17a869707645 in ?? ()
#16 0x00000ffe3468e67c in ?? ()
#17 0x00007feda34b33c0 in ?? ()
#18 0x00007fedb1598ba3 in ?? ()
#19 0x00007feda34b3390 in ?? ()
#20 0x3fb9890f1932c000 in ?? ()
#21 0x00000000a34b33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb9890f1932c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 4027):
#0 0x00007fedaef21a47 in ?? ()
#1 0x00007feda3cb4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007feda3cb4310 in ?? ()
#4 0x00007feda3cb4520 in ?? ()
#5 0x00007feda3cb4360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007feda3cb43e0 in ?? ()
#8 0x00007fedb159425d in ?? ()
#9 0x3fb96a6bfc1e0000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x0000000094000000 in ?? ()
#14 0x00007fed00000000 in ?? ()
#15 0x41da17a869707644 in ?? ()
#16 0x00000ffe3478e87c in ?? ()
#17 0x00007feda3cb43c0 in ?? ()
#18 0x00007fedb1598ba3 in ?? ()
#19 0x00007feda3cb4390 in ?? ()
#20 0x3fb96a6bfc1e0000 in ?? ()
#21 0x00000000a3cb43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb96a6bfc1e0000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 4026):
#0 0x00007fedaef21a47 in ?? ()
#1 0x00007feda5cad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007feda5cad310 in ?? ()
#4 0x00007feda5cad520 in ?? ()
#5 0x00007feda5cad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007feda5cad3e0 in ?? ()
#8 0x00007fedb159425d in ?? ()
#9 0x3fb9894443f32000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x000000006e000000 in ?? ()
#14 0x00007fed00000000 in ?? ()
#15 0x41da17a869707647 in ?? ()
#16 0x00000ffe34b8da7c in ?? ()
#17 0x00007feda5cad3c0 in ?? ()
#18 0x00007fedb1598ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 4022):
#0 0x00007fedaef14cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007feda76b77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 4021):
#0 0x00007fedb61eefb9 in ?? ()
#1 0x00007feda6eb60c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001cf40 in ?? ()
#5 0x00007feda6eb60b0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 4020):
#0 0x00007fedb61f29e2 in ?? ()
#1 0x00007feda66b3ba0 in ?? ()
#2 0x00007feda66b3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007feda66b3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007feda66b3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007feda66b3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007fedbbb05edc in ?? ()
#12 0x00007fedbbaf58d9 in ?? ()
#13 0x00007fedbbaf9ec6 in ?? ()
#14 0x00007fedbbafe900 in ?? ()
#15 0x00007fedbbafe729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007fedb2dca4de in ?? ()
#18 0x00007fedb61e86db in ?? ()
#19 0x00000ffe34cce684 in ?? ()
#20 0x00007feda66b3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 4013):
#0 0x00007fedb61eefb9 in ?? ()
#1 0x00007feda86b8c80 in ?? ()
#2 0x0000000000000347 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001e5a8 in ?? ()
#5 0x00007feda86b8c70 in ?? ()
#6 0x000000000000068e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 4012):
#0 0x00007fedb61eefb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 4011):
#0 0x00007fedb61eefb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 4010):
#0 0x00007fedb61eefb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007feda9ebc450 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 4008):
#0 0x00007fedb61f2d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 13:57:49.565671 1586 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID 0204b303a14a41b096bfddffc4d04290 and pid 3719
************************ BEGIN STACKS **************************
[New LWP 3722]
[New LWP 3723]
[New LWP 3724]
[New LWP 3725]
[New LWP 3732]
[New LWP 3733]
[New LWP 3734]
[New LWP 3738]
[New LWP 3739]
[New LWP 3740]
[New LWP 3741]
[New LWP 3742]
[New LWP 3743]
[New LWP 3745]
[New LWP 3746]
[New LWP 3747]
[New LWP 3748]
[New LWP 3749]
[New LWP 3750]
[New LWP 3751]
[New LWP 3752]
[New LWP 3753]
[New LWP 3754]
[New LWP 3755]
[New LWP 3756]
[New LWP 3757]
[New LWP 3758]
[New LWP 3759]
[New LWP 3760]
[New LWP 3761]
[New LWP 3762]
[New LWP 3763]
[New LWP 3764]
[New LWP 3765]
[New LWP 3766]
[New LWP 3767]
[New LWP 3768]
[New LWP 3769]
[New LWP 3770]
[New LWP 3771]
[New LWP 3772]
[New LWP 3773]
[New LWP 3774]
[New LWP 3775]
[New LWP 3776]
[New LWP 3777]
[New LWP 3778]
[New LWP 3779]
[New LWP 3780]
[New LWP 3781]
[New LWP 3782]
[New LWP 3783]
[New LWP 3784]
[New LWP 3785]
[New LWP 3786]
[New LWP 3787]
[New LWP 3788]
[New LWP 3789]
[New LWP 3790]
[New LWP 3791]
[New LWP 3792]
[New LWP 3793]
[New LWP 3794]
[New LWP 3795]
[New LWP 3796]
[New LWP 3797]
[New LWP 3798]
[New LWP 3799]
[New LWP 3800]
[New LWP 3801]
[New LWP 3802]
[New LWP 3803]
[New LWP 3804]
[New LWP 3805]
[New LWP 3806]
[New LWP 3807]
[New LWP 3808]
[New LWP 3809]
[New LWP 3810]
[New LWP 3811]
[New LWP 3812]
[New LWP 3813]
[New LWP 3814]
[New LWP 3815]
[New LWP 3816]
[New LWP 3817]
[New LWP 3818]
[New LWP 3819]
[New LWP 3820]
[New LWP 3821]
[New LWP 3822]
[New LWP 3823]
[New LWP 3824]
[New LWP 3825]
[New LWP 3826]
[New LWP 3827]
[New LWP 3828]
[New LWP 3829]
[New LWP 3830]
[New LWP 3831]
[New LWP 3832]
[New LWP 3833]
[New LWP 3834]
[New LWP 3835]
[New LWP 3836]
[New LWP 3837]
[New LWP 3838]
[New LWP 3839]
[New LWP 3840]
[New LWP 3841]
[New LWP 3842]
[New LWP 3843]
[New LWP 3845]
[New LWP 3847]
[New LWP 3848]
[New LWP 3849]
[New LWP 3850]
[New LWP 3851]
[New LWP 3852]
[New LWP 3853]
[New LWP 3854]
[New LWP 5945]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007f5de9836d50 in ?? ()
Id Target Id Frame
* 1 LWP 3719 "kudu" 0x00007f5de9836d50 in ?? ()
2 LWP 3722 "kudu" 0x00007f5de9832fb9 in ?? ()
3 LWP 3723 "kudu" 0x00007f5de9832fb9 in ?? ()
4 LWP 3724 "kudu" 0x00007f5de9832fb9 in ?? ()
5 LWP 3725 "kernel-watcher-" 0x00007f5de9832fb9 in ?? ()
6 LWP 3732 "ntp client-3732" 0x00007f5de98369e2 in ?? ()
7 LWP 3733 "file cache-evic" 0x00007f5de9832fb9 in ?? ()
8 LWP 3734 "sq_acceptor" 0x00007f5de2558cb9 in ?? ()
9 LWP 3738 "rpc reactor-373" 0x00007f5de2565a47 in ?? ()
10 LWP 3739 "rpc reactor-373" 0x00007f5de2565a47 in ?? ()
11 LWP 3740 "rpc reactor-374" 0x00007f5de2565a47 in ?? ()
12 LWP 3741 "rpc reactor-374" 0x00007f5de2565a47 in ?? ()
13 LWP 3742 "MaintenanceMgr " 0x00007f5de9832ad3 in ?? ()
14 LWP 3743 "txn-status-mana" 0x00007f5de9832fb9 in ?? ()
15 LWP 3745 "collect_and_rem" 0x00007f5de9832fb9 in ?? ()
16 LWP 3746 "tc-session-exp-" 0x00007f5de9832fb9 in ?? ()
17 LWP 3747 "rpc worker-3747" 0x00007f5de9832ad3 in ?? ()
18 LWP 3748 "rpc worker-3748" 0x00007f5de9832ad3 in ?? ()
19 LWP 3749 "rpc worker-3749" 0x00007f5de9832ad3 in ?? ()
20 LWP 3750 "rpc worker-3750" 0x00007f5de9832ad3 in ?? ()
21 LWP 3751 "rpc worker-3751" 0x00007f5de9832ad3 in ?? ()
22 LWP 3752 "rpc worker-3752" 0x00007f5de9832ad3 in ?? ()
23 LWP 3753 "rpc worker-3753" 0x00007f5de9832ad3 in ?? ()
24 LWP 3754 "rpc worker-3754" 0x00007f5de9832ad3 in ?? ()
25 LWP 3755 "rpc worker-3755" 0x00007f5de9832ad3 in ?? ()
26 LWP 3756 "rpc worker-3756" 0x00007f5de9832ad3 in ?? ()
27 LWP 3757 "rpc worker-3757" 0x00007f5de9832ad3 in ?? ()
28 LWP 3758 "rpc worker-3758" 0x00007f5de9832ad3 in ?? ()
29 LWP 3759 "rpc worker-3759" 0x00007f5de9832ad3 in ?? ()
30 LWP 3760 "rpc worker-3760" 0x00007f5de9832ad3 in ?? ()
31 LWP 3761 "rpc worker-3761" 0x00007f5de9832ad3 in ?? ()
32 LWP 3762 "rpc worker-3762" 0x00007f5de9832ad3 in ?? ()
33 LWP 3763 "rpc worker-3763" 0x00007f5de9832ad3 in ?? ()
34 LWP 3764 "rpc worker-3764" 0x00007f5de9832ad3 in ?? ()
35 LWP 3765 "rpc worker-3765" 0x00007f5de9832ad3 in ?? ()
36 LWP 3766 "rpc worker-3766" 0x00007f5de9832ad3 in ?? ()
37 LWP 3767 "rpc worker-3767" 0x00007f5de9832ad3 in ?? ()
38 LWP 3768 "rpc worker-3768" 0x00007f5de9832ad3 in ?? ()
39 LWP 3769 "rpc worker-3769" 0x00007f5de9832ad3 in ?? ()
40 LWP 3770 "rpc worker-3770" 0x00007f5de9832ad3 in ?? ()
41 LWP 3771 "rpc worker-3771" 0x00007f5de9832ad3 in ?? ()
42 LWP 3772 "rpc worker-3772" 0x00007f5de9832ad3 in ?? ()
43 LWP 3773 "rpc worker-3773" 0x00007f5de9832ad3 in ?? ()
44 LWP 3774 "rpc worker-3774" 0x00007f5de9832ad3 in ?? ()
45 LWP 3775 "rpc worker-3775" 0x00007f5de9832ad3 in ?? ()
46 LWP 3776 "rpc worker-3776" 0x00007f5de9832ad3 in ?? ()
47 LWP 3777 "rpc worker-3777" 0x00007f5de9832ad3 in ?? ()
48 LWP 3778 "rpc worker-3778" 0x00007f5de9832ad3 in ?? ()
49 LWP 3779 "rpc worker-3779" 0x00007f5de9832ad3 in ?? ()
50 LWP 3780 "rpc worker-3780" 0x00007f5de9832ad3 in ?? ()
51 LWP 3781 "rpc worker-3781" 0x00007f5de9832ad3 in ?? ()
52 LWP 3782 "rpc worker-3782" 0x00007f5de9832ad3 in ?? ()
53 LWP 3783 "rpc worker-3783" 0x00007f5de9832ad3 in ?? ()
54 LWP 3784 "rpc worker-3784" 0x00007f5de9832ad3 in ?? ()
55 LWP 3785 "rpc worker-3785" 0x00007f5de9832ad3 in ?? ()
56 LWP 3786 "rpc worker-3786" 0x00007f5de9832ad3 in ?? ()
57 LWP 3787 "rpc worker-3787" 0x00007f5de9832ad3 in ?? ()
58 LWP 3788 "rpc worker-3788" 0x00007f5de9832ad3 in ?? ()
59 LWP 3789 "rpc worker-3789" 0x00007f5de9832ad3 in ?? ()
60 LWP 3790 "rpc worker-3790" 0x00007f5de9832ad3 in ?? ()
61 LWP 3791 "rpc worker-3791" 0x00007f5de9832ad3 in ?? ()
62 LWP 3792 "rpc worker-3792" 0x00007f5de9832ad3 in ?? ()
63 LWP 3793 "rpc worker-3793" 0x00007f5de9832ad3 in ?? ()
64 LWP 3794 "rpc worker-3794" 0x00007f5de9832ad3 in ?? ()
65 LWP 3795 "rpc worker-3795" 0x00007f5de9832ad3 in ?? ()
66 LWP 3796 "rpc worker-3796" 0x00007f5de9832ad3 in ?? ()
67 LWP 3797 "rpc worker-3797" 0x00007f5de9832ad3 in ?? ()
68 LWP 3798 "rpc worker-3798" 0x00007f5de9832ad3 in ?? ()
69 LWP 3799 "rpc worker-3799" 0x00007f5de9832ad3 in ?? ()
70 LWP 3800 "rpc worker-3800" 0x00007f5de9832ad3 in ?? ()
71 LWP 3801 "rpc worker-3801" 0x00007f5de9832ad3 in ?? ()
72 LWP 3802 "rpc worker-3802" 0x00007f5de9832ad3 in ?? ()
73 LWP 3803 "rpc worker-3803" 0x00007f5de9832ad3 in ?? ()
74 LWP 3804 "rpc worker-3804" 0x00007f5de9832ad3 in ?? ()
75 LWP 3805 "rpc worker-3805" 0x00007f5de9832ad3 in ?? ()
76 LWP 3806 "rpc worker-3806" 0x00007f5de9832ad3 in ?? ()
77 LWP 3807 "rpc worker-3807" 0x00007f5de9832ad3 in ?? ()
78 LWP 3808 "rpc worker-3808" 0x00007f5de9832ad3 in ?? ()
79 LWP 3809 "rpc worker-3809" 0x00007f5de9832ad3 in ?? ()
80 LWP 3810 "rpc worker-3810" 0x00007f5de9832ad3 in ?? ()
81 LWP 3811 "rpc worker-3811" 0x00007f5de9832ad3 in ?? ()
82 LWP 3812 "rpc worker-3812" 0x00007f5de9832ad3 in ?? ()
83 LWP 3813 "rpc worker-3813" 0x00007f5de9832ad3 in ?? ()
84 LWP 3814 "rpc worker-3814" 0x00007f5de9832ad3 in ?? ()
85 LWP 3815 "rpc worker-3815" 0x00007f5de9832ad3 in ?? ()
86 LWP 3816 "rpc worker-3816" 0x00007f5de9832ad3 in ?? ()
87 LWP 3817 "rpc worker-3817" 0x00007f5de9832ad3 in ?? ()
88 LWP 3818 "rpc worker-3818" 0x00007f5de9832ad3 in ?? ()
89 LWP 3819 "rpc worker-3819" 0x00007f5de9832ad3 in ?? ()
90 LWP 3820 "rpc worker-3820" 0x00007f5de9832ad3 in ?? ()
91 LWP 3821 "rpc worker-3821" 0x00007f5de9832ad3 in ?? ()
92 LWP 3822 "rpc worker-3822" 0x00007f5de9832ad3 in ?? ()
93 LWP 3823 "rpc worker-3823" 0x00007f5de9832ad3 in ?? ()
94 LWP 3824 "rpc worker-3824" 0x00007f5de9832ad3 in ?? ()
95 LWP 3825 "rpc worker-3825" 0x00007f5de9832ad3 in ?? ()
96 LWP 3826 "rpc worker-3826" 0x00007f5de9832ad3 in ?? ()
97 LWP 3827 "rpc worker-3827" 0x00007f5de9832ad3 in ?? ()
98 LWP 3828 "rpc worker-3828" 0x00007f5de9832ad3 in ?? ()
99 LWP 3829 "rpc worker-3829" 0x00007f5de9832ad3 in ?? ()
100 LWP 3830 "rpc worker-3830" 0x00007f5de9832ad3 in ?? ()
101 LWP 3831 "rpc worker-3831" 0x00007f5de9832ad3 in ?? ()
102 LWP 3832 "rpc worker-3832" 0x00007f5de9832ad3 in ?? ()
103 LWP 3833 "rpc worker-3833" 0x00007f5de9832ad3 in ?? ()
104 LWP 3834 "rpc worker-3834" 0x00007f5de9832ad3 in ?? ()
105 LWP 3835 "rpc worker-3835" 0x00007f5de9832ad3 in ?? ()
106 LWP 3836 "rpc worker-3836" 0x00007f5de9832ad3 in ?? ()
107 LWP 3837 "rpc worker-3837" 0x00007f5de9832ad3 in ?? ()
108 LWP 3838 "rpc worker-3838" 0x00007f5de9832ad3 in ?? ()
109 LWP 3839 "rpc worker-3839" 0x00007f5de9832ad3 in ?? ()
110 LWP 3840 "rpc worker-3840" 0x00007f5de9832ad3 in ?? ()
111 LWP 3841 "rpc worker-3841" 0x00007f5de9832ad3 in ?? ()
112 LWP 3842 "rpc worker-3842" 0x00007f5de9832ad3 in ?? ()
113 LWP 3843 "rpc worker-3843" 0x00007f5de9832ad3 in ?? ()
114 LWP 3845 "rpc worker-3845" 0x00007f5de9832ad3 in ?? ()
115 LWP 3847 "rpc worker-3847" 0x00007f5de9832ad3 in ?? ()
116 LWP 3848 "rpc worker-3848" 0x00007f5de9832ad3 in ?? ()
117 LWP 3849 "diag-logger-384" 0x00007f5de9832fb9 in ?? ()
118 LWP 3850 "result-tracker-" 0x00007f5de9832fb9 in ?? ()
119 LWP 3851 "excess-log-dele" 0x00007f5de9832fb9 in ?? ()
120 LWP 3852 "acceptor-3852" 0x00007f5de25670c7 in ?? ()
121 LWP 3853 "heartbeat-3853" 0x00007f5de9832fb9 in ?? ()
122 LWP 3854 "maintenance_sch" 0x00007f5de9832fb9 in ?? ()
123 LWP 5945 "raft [worker]-5" 0x00007f5de9832fb9 in ?? ()
Thread 123 (LWP 5945):
#0 0x00007f5de9832fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000001e3 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00007f5d8de3bba4 in ?? ()
#5 0x00007f5d8de3b830 in ?? ()
#6 0x00000000000003c7 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f5d8de3b850 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 122 (LWP 3854):
#0 0x00007f5de9832fb9 in ?? ()
#1 0x00007f5d9e8806e0 in ?? ()
#2 0x00000000000002b8 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00006160000157a4 in ?? ()
#5 0x00007f5d9e880730 in ?? ()
#6 0x0000000000000571 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f5d9e880750 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007f5de62c23f0 in ?? ()
#12 0x00007f5d9e8810d0 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 121 (LWP 3853):
#0 0x00007f5de9832fb9 in ?? ()
#1 0x26b5f4687947e954 in ?? ()
#2 0x00000000000000b1 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019e80 in ?? ()
#5 0x00007f5d9f0985f0 in ?? ()
#6 0x0000000000000162 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 3852):
#0 0x00007f5de25670c7 in ?? ()
#1 0x00006140000210a8 in ?? ()
#2 0x00007f5d9f8bfd90 in ?? ()
#3 0x00007f5d9f8bfdc0 in ?? ()
#4 0x00007f5d9f8bfec0 in ?? ()
#5 0x00007f5d9f8bfdb0 in ?? ()
#6 0x00007f5d9f8bfe20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x00007f5de63ed960 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 3851):
#0 0x00007f5de9832fb9 in ?? ()
#1 0x00007f5da00d8f40 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffc8ef6fbb0 in ?? ()
#5 0x00007f5da00d8f30 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 3850):
#0 0x00007f5de9832fb9 in ?? ()
#1 0x00007f5da08f2100 in ?? ()
#2 0x00000000000000ad in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f5da08f20f0 in ?? ()
#6 0x000000000000015a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 3849):
#0 0x00007f5de9832fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 3848):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000017 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b684c in ?? ()
#4 0x00007f5da1921e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5da1921eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b6800 in ?? ()
#9 0x00007f5de9832770 in ?? ()
#10 0x00007f5da1921eb0 in ?? ()
#11 0x00000febb43243d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 3847):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000025 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007f5da2139e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5da2139eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007f5de9832770 in ?? ()
#10 0x00007f5da2139eb0 in ?? ()
#11 0x00000febb44273d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 3845):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 3843):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 3842):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 3841):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 3840):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 3839):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 3838):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 3837):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 3836):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 3835):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 3834):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 3833):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 3832):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 3831):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 3830):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 3829):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 3828):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 3827):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 3826):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 3825):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 3824):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 3823):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 3822):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 3821):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 3820):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 3819):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 3818):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 3817):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 3816):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 3815):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 3814):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 3813):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 3812):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 3811):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 3810):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 3809):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 3808):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 3807):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 3806):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x000000000000040b in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007f5db5cf2e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5db5cf2eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007f5de9832770 in ?? ()
#10 0x00007f5db5cf2eb0 in ?? ()
#11 0x00000febb6b9e5d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 3805):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x00000000000002f8 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a6888 in ?? ()
#4 0x00007f5db650ae90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5db650aeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 3804):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 3803):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 3802):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 3801):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 3800):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 3799):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 3798):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 3797):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 3796):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 3795):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 3794):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 3793):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 3792):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 3791):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 3790):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 3789):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 3788):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 3787):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 3786):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007f5dbfed3e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5dbfed3eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007f5de9832770 in ?? ()
#10 0x00007f5dbfed3eb0 in ?? ()
#11 0x00000febb7fda7d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 3785):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 3784):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 3783):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 3782):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 3781):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 3780):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 3779):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 3778):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 3777):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 3776):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 3775):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 3774):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 3773):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 3772):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 3771):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 3770):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 3769):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 3768):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 3767):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 3766):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000001057 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007f5dca0b4e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5dca0b4eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007f5de9832770 in ?? ()
#10 0x00007f5dca0b4eb0 in ?? ()
#11 0x00000febb94169d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 3765):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000e71 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007f5dca8cce90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5dca8cceb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007f5de9832770 in ?? ()
#10 0x00007f5dca8cceb0 in ?? ()
#11 0x00000febb95199d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 34 (LWP 3764):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000eb8 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000967f8 in ?? ()
#4 0x00007f5ddb5b7e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5ddb5b7eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 33 (LWP 3763):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 3762):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 3761):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 3760):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 3759):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 3758):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 3757):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 3756):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 3755):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 3754):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 3753):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 3752):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 3751):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 3750):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 3749):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 3748):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 3747):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 3746):
#0 0x00007f5de9832fb9 in ?? ()
#1 0x00007f5dd3ab2cc0 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ae20 in ?? ()
#5 0x00007f5dd3ab2cb0 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 3745):
#0 0x00007f5de9832fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 3743):
#0 0x00007f5de9832fb9 in ?? ()
#1 0x00007f5dd5337240 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007f5dd5337230 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 3742):
#0 0x00007f5de9832ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007f5dd5b5eba4 in ?? ()
#4 0x00007f5dd5b5e8b0 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f5dd5b5e8d0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00006160000159c0 in ?? ()
#9 0x00007f5de9832770 in ?? ()
#10 0x00007f5dd5b5e8d0 in ?? ()
#11 0x00007f5dd5b5e8d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 12 (LWP 3741):
#0 0x00007f5de2565a47 in ?? ()
#1 0x00007f5dd6386320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f5dd6386310 in ?? ()
#4 0x00007f5dd6386520 in ?? ()
#5 0x00007f5dd6386360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f5dd63863e0 in ?? ()
#8 0x00007f5de4bd825d in ?? ()
#9 0x3fad8544f0990000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x000000004a300000 in ?? ()
#14 0x00007f5d00000000 in ?? ()
#15 0x41da17a869707644 in ?? ()
#16 0x00000fec3ac68c7c in ?? ()
#17 0x00007f5dd63863c0 in ?? ()
#18 0x00007f5de4bdcba3 in ?? ()
#19 0x00007f5dd6386390 in ?? ()
#20 0x3fad8544f0990000 in ?? ()
#21 0x00000000d63863e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fad8544f0990000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 3740):
#0 0x00007f5de2565a47 in ?? ()
#1 0x00007f5dd6b9d320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f5dd6b9d310 in ?? ()
#4 0x00007f5dd6b9d520 in ?? ()
#5 0x00007f5dd6b9d360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f5dd6b9d3e0 in ?? ()
#8 0x00007f5de4bd825d in ?? ()
#9 0x3fb95abfdd599000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x00000000f9000000 in ?? ()
#14 0x00007f5d00000000 in ?? ()
#15 0x41da17a869707646 in ?? ()
#16 0x00000fec3ad6ba7c in ?? ()
#17 0x00007f5dd6b9d3c0 in ?? ()
#18 0x00007f5de4bdcba3 in ?? ()
#19 0x00007f5dd6b9d390 in ?? ()
#20 0x3fb95abfdd599000 in ?? ()
#21 0x00000000d6b9d3e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb95abfdd599000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 3739):
#0 0x00007f5de2565a47 in ?? ()
#1 0x00007f5dd73b4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f5dd73b4310 in ?? ()
#4 0x00007f5dd73b4520 in ?? ()
#5 0x00007f5dd73b4360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f5dd73b43e0 in ?? ()
#8 0x00007f5de4bd825d in ?? ()
#9 0x3fa4b546eb994000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x000000003c000000 in ?? ()
#14 0x00007f5d00000000 in ?? ()
#15 0x41da17a869707643 in ?? ()
#16 0x00000fec3ae6e87c in ?? ()
#17 0x00007f5dd73b43c0 in ?? ()
#18 0x00007f5de4bdcba3 in ?? ()
#19 0x00007f5dd73b4390 in ?? ()
#20 0x3fa4b546eb994000 in ?? ()
#21 0x00000000d73b43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fa4b546eb994000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 3738):
#0 0x00007f5de2565a47 in ?? ()
#1 0x00007f5dd93ad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f5dd93ad310 in ?? ()
#4 0x00007f5dd93ad520 in ?? ()
#5 0x00007f5dd93ad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007f5dd93ad3e0 in ?? ()
#8 0x00007f5de4bd825d in ?? ()
#9 0x3fb9659962121000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x000000002e000000 in ?? ()
#14 0x00007f5d00000000 in ?? ()
#15 0x41da17a869707646 in ?? ()
#16 0x00000fec3b26da7c in ?? ()
#17 0x00007f5dd93ad3c0 in ?? ()
#18 0x00007f5de4bdcba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 3734):
#0 0x00007f5de2558cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f5ddadb77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 3733):
#0 0x00007f5de9832fb9 in ?? ()
#1 0x00007f5dda5b60c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001cf40 in ?? ()
#5 0x00007f5dda5b60b0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 3732):
#0 0x00007f5de98369e2 in ?? ()
#1 0x00007f5dd9db3ba0 in ?? ()
#2 0x00007f5dd9db3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f5dd9db3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007f5dd9db3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f5dd9db3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007f5def149edc in ?? ()
#12 0x00007f5def1398d9 in ?? ()
#13 0x00007f5def13dec6 in ?? ()
#14 0x00007f5def142900 in ?? ()
#15 0x00007f5def142729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007f5de640e4de in ?? ()
#18 0x00007f5de982c6db in ?? ()
#19 0x00000fec3b3ae684 in ?? ()
#20 0x00007f5dd9db3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 3725):
#0 0x00007f5de9832fb9 in ?? ()
#1 0x00007f5ddbdb8c80 in ?? ()
#2 0x0000000000000369 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001e5a8 in ?? ()
#5 0x00007f5ddbdb8c70 in ?? ()
#6 0x00000000000006d2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 3724):
#0 0x00007f5de9832fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 3723):
#0 0x00007f5de9832fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 3722):
#0 0x00007f5de9832fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f5ddd5bc450 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 3719):
#0 0x00007f5de9836d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 13:57:50.253569 1586 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID e879d1cb3a7c4226884531badda84252 and pid 3863
************************ BEGIN STACKS **************************
[New LWP 3865]
[New LWP 3866]
[New LWP 3867]
[New LWP 3868]
[New LWP 3875]
[New LWP 3876]
[New LWP 3877]
[New LWP 3881]
[New LWP 3882]
[New LWP 3883]
[New LWP 3884]
[New LWP 3885]
[New LWP 3886]
[New LWP 3887]
[New LWP 3888]
[New LWP 3889]
[New LWP 3890]
[New LWP 3891]
[New LWP 3892]
[New LWP 3893]
[New LWP 3895]
[New LWP 3896]
[New LWP 3897]
[New LWP 3898]
[New LWP 3899]
[New LWP 3900]
[New LWP 3901]
[New LWP 3902]
[New LWP 3903]
[New LWP 3904]
[New LWP 3905]
[New LWP 3906]
[New LWP 3907]
[New LWP 3908]
[New LWP 3909]
[New LWP 3910]
[New LWP 3911]
[New LWP 3912]
[New LWP 3913]
[New LWP 3914]
[New LWP 3915]
[New LWP 3916]
[New LWP 3917]
[New LWP 3918]
[New LWP 3919]
[New LWP 3920]
[New LWP 3921]
[New LWP 3922]
[New LWP 3923]
[New LWP 3924]
[New LWP 3925]
[New LWP 3926]
[New LWP 3927]
[New LWP 3928]
[New LWP 3929]
[New LWP 3930]
[New LWP 3931]
[New LWP 3932]
[New LWP 3933]
[New LWP 3934]
[New LWP 3935]
[New LWP 3936]
[New LWP 3937]
[New LWP 3938]
[New LWP 3939]
[New LWP 3940]
[New LWP 3941]
[New LWP 3942]
[New LWP 3943]
[New LWP 3944]
[New LWP 3945]
[New LWP 3946]
[New LWP 3947]
[New LWP 3948]
[New LWP 3949]
[New LWP 3950]
[New LWP 3951]
[New LWP 3952]
[New LWP 3953]
[New LWP 3954]
[New LWP 3955]
[New LWP 3956]
[New LWP 3957]
[New LWP 3958]
[New LWP 3959]
[New LWP 3960]
[New LWP 3961]
[New LWP 3962]
[New LWP 3963]
[New LWP 3964]
[New LWP 3965]
[New LWP 3966]
[New LWP 3967]
[New LWP 3968]
[New LWP 3969]
[New LWP 3970]
[New LWP 3972]
[New LWP 3973]
[New LWP 3974]
[New LWP 3975]
[New LWP 3976]
[New LWP 3978]
[New LWP 3979]
[New LWP 3980]
[New LWP 3981]
[New LWP 3982]
[New LWP 3983]
[New LWP 3984]
[New LWP 3985]
[New LWP 3986]
[New LWP 3987]
[New LWP 3988]
[New LWP 3989]
[New LWP 3990]
[New LWP 3991]
[New LWP 3992]
[New LWP 3993]
[New LWP 3994]
[New LWP 3995]
[New LWP 3996]
[New LWP 4002]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007f3a17406d50 in ?? ()
Id Target Id Frame
* 1 LWP 3863 "kudu" 0x00007f3a17406d50 in ?? ()
2 LWP 3865 "kudu" 0x00007f3a17402fb9 in ?? ()
3 LWP 3866 "kudu" 0x00007f3a17402fb9 in ?? ()
4 LWP 3867 "kudu" 0x00007f3a17402fb9 in ?? ()
5 LWP 3868 "kernel-watcher-" 0x00007f3a17402fb9 in ?? ()
6 LWP 3875 "ntp client-3875" 0x00007f3a174069e2 in ?? ()
7 LWP 3876 "file cache-evic" 0x00007f3a17402fb9 in ?? ()
8 LWP 3877 "sq_acceptor" 0x00007f3a10128cb9 in ?? ()
9 LWP 3881 "rpc reactor-388" 0x00007f3a10135a47 in ?? ()
10 LWP 3882 "rpc reactor-388" 0x00007f3a10135a47 in ?? ()
11 LWP 3883 "rpc reactor-388" 0x00007f3a10135a47 in ?? ()
12 LWP 3884 "rpc reactor-388" 0x00007f3a10135a47 in ?? ()
13 LWP 3885 "MaintenanceMgr " 0x00007f3a17402ad3 in ?? ()
14 LWP 3886 "txn-status-mana" 0x00007f3a17402fb9 in ?? ()
15 LWP 3887 "collect_and_rem" 0x00007f3a17402fb9 in ?? ()
16 LWP 3888 "tc-session-exp-" 0x00007f3a17402fb9 in ?? ()
17 LWP 3889 "rpc worker-3889" 0x00007f3a17402ad3 in ?? ()
18 LWP 3890 "rpc worker-3890" 0x00007f3a17402ad3 in ?? ()
19 LWP 3891 "rpc worker-3891" 0x00007f3a17402ad3 in ?? ()
20 LWP 3892 "rpc worker-3892" 0x00007f3a17402ad3 in ?? ()
21 LWP 3893 "rpc worker-3893" 0x00007f3a17402ad3 in ?? ()
22 LWP 3895 "rpc worker-3895" 0x00007f3a17402ad3 in ?? ()
23 LWP 3896 "rpc worker-3896" 0x00007f3a17402ad3 in ?? ()
24 LWP 3897 "rpc worker-3897" 0x00007f3a17402ad3 in ?? ()
25 LWP 3898 "rpc worker-3898" 0x00007f3a17402ad3 in ?? ()
26 LWP 3899 "rpc worker-3899" 0x00007f3a17402ad3 in ?? ()
27 LWP 3900 "rpc worker-3900" 0x00007f3a17402ad3 in ?? ()
28 LWP 3901 "rpc worker-3901" 0x00007f3a17402ad3 in ?? ()
29 LWP 3902 "rpc worker-3902" 0x00007f3a17402ad3 in ?? ()
30 LWP 3903 "rpc worker-3903" 0x00007f3a17402ad3 in ?? ()
31 LWP 3904 "rpc worker-3904" 0x00007f3a17402ad3 in ?? ()
32 LWP 3905 "rpc worker-3905" 0x00007f3a17402ad3 in ?? ()
33 LWP 3906 "rpc worker-3906" 0x00007f3a17402ad3 in ?? ()
34 LWP 3907 "rpc worker-3907" 0x00007f3a17402ad3 in ?? ()
35 LWP 3908 "rpc worker-3908" 0x00007f3a17402ad3 in ?? ()
36 LWP 3909 "rpc worker-3909" 0x00007f3a17402ad3 in ?? ()
37 LWP 3910 "rpc worker-3910" 0x00007f3a17402ad3 in ?? ()
38 LWP 3911 "rpc worker-3911" 0x00007f3a17402ad3 in ?? ()
39 LWP 3912 "rpc worker-3912" 0x00007f3a17402ad3 in ?? ()
40 LWP 3913 "rpc worker-3913" 0x00007f3a17402ad3 in ?? ()
41 LWP 3914 "rpc worker-3914" 0x00007f3a17402ad3 in ?? ()
42 LWP 3915 "rpc worker-3915" 0x00007f3a17402ad3 in ?? ()
43 LWP 3916 "rpc worker-3916" 0x00007f3a17402ad3 in ?? ()
44 LWP 3917 "rpc worker-3917" 0x00007f3a17402ad3 in ?? ()
45 LWP 3918 "rpc worker-3918" 0x00007f3a17402ad3 in ?? ()
46 LWP 3919 "rpc worker-3919" 0x00007f3a17402ad3 in ?? ()
47 LWP 3920 "rpc worker-3920" 0x00007f3a17402ad3 in ?? ()
48 LWP 3921 "rpc worker-3921" 0x00007f3a17402ad3 in ?? ()
49 LWP 3922 "rpc worker-3922" 0x00007f3a17402ad3 in ?? ()
50 LWP 3923 "rpc worker-3923" 0x00007f3a17402ad3 in ?? ()
51 LWP 3924 "rpc worker-3924" 0x00007f3a17402ad3 in ?? ()
52 LWP 3925 "rpc worker-3925" 0x00007f3a17402ad3 in ?? ()
53 LWP 3926 "rpc worker-3926" 0x00007f3a17402ad3 in ?? ()
54 LWP 3927 "rpc worker-3927" 0x00007f3a17402ad3 in ?? ()
55 LWP 3928 "rpc worker-3928" 0x00007f3a17402ad3 in ?? ()
56 LWP 3929 "rpc worker-3929" 0x00007f3a17402ad3 in ?? ()
57 LWP 3930 "rpc worker-3930" 0x00007f3a17402ad3 in ?? ()
58 LWP 3931 "rpc worker-3931" 0x00007f3a17402ad3 in ?? ()
59 LWP 3932 "rpc worker-3932" 0x00007f3a17402ad3 in ?? ()
60 LWP 3933 "rpc worker-3933" 0x00007f3a17402ad3 in ?? ()
61 LWP 3934 "rpc worker-3934" 0x00007f3a17402ad3 in ?? ()
62 LWP 3935 "rpc worker-3935" 0x00007f3a17402ad3 in ?? ()
63 LWP 3936 "rpc worker-3936" 0x00007f3a17402ad3 in ?? ()
64 LWP 3937 "rpc worker-3937" 0x00007f3a17402ad3 in ?? ()
65 LWP 3938 "rpc worker-3938" 0x00007f3a17402ad3 in ?? ()
66 LWP 3939 "rpc worker-3939" 0x00007f3a17402ad3 in ?? ()
67 LWP 3940 "rpc worker-3940" 0x00007f3a17402ad3 in ?? ()
68 LWP 3941 "rpc worker-3941" 0x00007f3a17402ad3 in ?? ()
69 LWP 3942 "rpc worker-3942" 0x00007f3a17402ad3 in ?? ()
70 LWP 3943 "rpc worker-3943" 0x00007f3a17402ad3 in ?? ()
71 LWP 3944 "rpc worker-3944" 0x00007f3a17402ad3 in ?? ()
72 LWP 3945 "rpc worker-3945" 0x00007f3a17402ad3 in ?? ()
73 LWP 3946 "rpc worker-3946" 0x00007f3a17402ad3 in ?? ()
74 LWP 3947 "rpc worker-3947" 0x00007f3a17402ad3 in ?? ()
75 LWP 3948 "rpc worker-3948" 0x00007f3a17402ad3 in ?? ()
76 LWP 3949 "rpc worker-3949" 0x00007f3a17402ad3 in ?? ()
77 LWP 3950 "rpc worker-3950" 0x00007f3a17402ad3 in ?? ()
78 LWP 3951 "rpc worker-3951" 0x00007f3a17402ad3 in ?? ()
79 LWP 3952 "rpc worker-3952" 0x00007f3a17402ad3 in ?? ()
80 LWP 3953 "rpc worker-3953" 0x00007f3a17402ad3 in ?? ()
81 LWP 3954 "rpc worker-3954" 0x00007f3a17402ad3 in ?? ()
82 LWP 3955 "rpc worker-3955" 0x00007f3a17402ad3 in ?? ()
83 LWP 3956 "rpc worker-3956" 0x00007f3a17402ad3 in ?? ()
84 LWP 3957 "rpc worker-3957" 0x00007f3a17402ad3 in ?? ()
85 LWP 3958 "rpc worker-3958" 0x00007f3a17402ad3 in ?? ()
86 LWP 3959 "rpc worker-3959" 0x00007f3a17402ad3 in ?? ()
87 LWP 3960 "rpc worker-3960" 0x00007f3a17402ad3 in ?? ()
88 LWP 3961 "rpc worker-3961" 0x00007f3a17402ad3 in ?? ()
89 LWP 3962 "rpc worker-3962" 0x00007f3a17402ad3 in ?? ()
90 LWP 3963 "rpc worker-3963" 0x00007f3a17402ad3 in ?? ()
91 LWP 3964 "rpc worker-3964" 0x00007f3a17402ad3 in ?? ()
92 LWP 3965 "rpc worker-3965" 0x00007f3a17402ad3 in ?? ()
93 LWP 3966 "rpc worker-3966" 0x00007f3a17402ad3 in ?? ()
94 LWP 3967 "rpc worker-3967" 0x00007f3a17402ad3 in ?? ()
95 LWP 3968 "rpc worker-3968" 0x00007f3a17402ad3 in ?? ()
96 LWP 3969 "rpc worker-3969" 0x00007f3a17402ad3 in ?? ()
97 LWP 3970 "rpc worker-3970" 0x00007f3a17402ad3 in ?? ()
98 LWP 3972 "rpc worker-3972" 0x00007f3a17402ad3 in ?? ()
99 LWP 3973 "rpc worker-3973" 0x00007f3a17402ad3 in ?? ()
100 LWP 3974 "rpc worker-3974" 0x00007f3a17402ad3 in ?? ()
101 LWP 3975 "rpc worker-3975" 0x00007f3a17402ad3 in ?? ()
102 LWP 3976 "rpc worker-3976" 0x00007f3a17402ad3 in ?? ()
103 LWP 3978 "rpc worker-3978" 0x00007f3a17402ad3 in ?? ()
104 LWP 3979 "rpc worker-3979" 0x00007f3a17402ad3 in ?? ()
105 LWP 3980 "rpc worker-3980" 0x00007f3a17402ad3 in ?? ()
106 LWP 3981 "rpc worker-3981" 0x00007f3a17402ad3 in ?? ()
107 LWP 3982 "rpc worker-3982" 0x00007f3a17402ad3 in ?? ()
108 LWP 3983 "rpc worker-3983" 0x00007f3a17402ad3 in ?? ()
109 LWP 3984 "rpc worker-3984" 0x00007f3a17402ad3 in ?? ()
110 LWP 3985 "rpc worker-3985" 0x00007f3a17402ad3 in ?? ()
111 LWP 3986 "rpc worker-3986" 0x00007f3a17402ad3 in ?? ()
112 LWP 3987 "rpc worker-3987" 0x00007f3a17402ad3 in ?? ()
113 LWP 3988 "rpc worker-3988" 0x00007f3a17402ad3 in ?? ()
114 LWP 3989 "rpc worker-3989" 0x00007f3a17402ad3 in ?? ()
115 LWP 3990 "rpc worker-3990" 0x00007f3a17402ad3 in ?? ()
116 LWP 3991 "rpc worker-3991" 0x00007f3a17402ad3 in ?? ()
117 LWP 3992 "diag-logger-399" 0x00007f3a17402fb9 in ?? ()
118 LWP 3993 "result-tracker-" 0x00007f3a17402fb9 in ?? ()
119 LWP 3994 "excess-log-dele" 0x00007f3a17402fb9 in ?? ()
120 LWP 3995 "acceptor-3995" 0x00007f3a101370c7 in ?? ()
121 LWP 3996 "heartbeat-3996" 0x00007f3a17402fb9 in ?? ()
122 LWP 4002 "maintenance_sch" 0x00007f3a17402fb9 in ?? ()
Thread 122 (LWP 4002):
#0 0x00007f3a17402fb9 in ?? ()
#1 0x00007f39cc49b6e0 in ?? ()
#2 0x00000000000002ac in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f39cc49b730 in ?? ()
#6 0x0000000000000558 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 3996):
#0 0x00007f3a17402fb9 in ?? ()
#1 0x26b5f4687947e954 in ?? ()
#2 0x00000000000000aa in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019e80 in ?? ()
#5 0x00007f39cd4c25f0 in ?? ()
#6 0x0000000000000154 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 3995):
#0 0x00007f3a101370c7 in ?? ()
#1 0x00006140000216a8 in ?? ()
#2 0x00007f39cdce9d90 in ?? ()
#3 0x00007f39cdce9dc0 in ?? ()
#4 0x00007f39cdce9ec0 in ?? ()
#5 0x00007f39cdce9db0 in ?? ()
#6 0x00007f39cdce9e20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 3994):
#0 0x00007f3a17402fb9 in ?? ()
#1 0x00007f39ce503f40 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff50c986d0 in ?? ()
#5 0x00007f39ce503f30 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 3993):
#0 0x00007f3a17402fb9 in ?? ()
#1 0x00007f39ced1c100 in ?? ()
#2 0x00000000000000ab in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f39ced1c0f0 in ?? ()
#6 0x0000000000000156 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 3992):
#0 0x00007f3a17402fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 3991):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000028 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007f39cfd4be90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f39cfd4beb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 3990):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000014 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002a6858 in ?? ()
#4 0x00007f39d0563e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f39d0563eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 3989):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 3988):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 3987):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 3986):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 3985):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 3984):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 3983):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 3982):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 3981):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 3980):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 3979):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 3978):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 3976):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 3975):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 3974):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 3973):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 3972):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 3970):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 3969):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 3968):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 3967):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 3966):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 3965):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 3964):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 3963):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 3962):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 3961):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 3960):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 3959):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 3958):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 3957):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 3956):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 3955):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 3954):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 3953):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 3952):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 3951):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 3950):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 3949):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a688c in ?? ()
#4 0x00007f39e411ce90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f39e411ceb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a6840 in ?? ()
#9 0x00007f3a17402770 in ?? ()
#10 0x00007f39e411ceb0 in ?? ()
#11 0x00000fe73c8239d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 3948):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 3947):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 3946):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 3945):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 3944):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 3943):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 3942):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 3941):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 3940):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 3939):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 3938):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 3937):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 3936):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 3935):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 3934):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 3933):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 3932):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 3931):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 3930):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 3929):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012003c in ?? ()
#4 0x00007f39ee2fde90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f39ee2fdeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00011fff0 in ?? ()
#9 0x00007f3a17402770 in ?? ()
#10 0x00007f39ee2fdeb0 in ?? ()
#11 0x00000fe73dc5fbd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 3928):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 3927):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 3926):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 3925):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 3924):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 3923):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 3922):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 3921):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 3920):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 3919):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 3918):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 3917):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 3916):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 3915):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 3914):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 3913):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 3912):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 3911):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 3910):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 3909):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007f39f84dee90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f39f84deeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007f3a17402770 in ?? ()
#10 0x00007f39f84deeb0 in ?? ()
#11 0x00000fe73f09bdd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 3908):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 3907):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 3906):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 3905):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 3904):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 3903):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 3902):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 3901):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 3900):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 3899):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 3898):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 3897):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 3896):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 3895):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 3893):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 3892):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 3891):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 3890):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 3889):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 3888):
#0 0x00007f3a17402fb9 in ?? ()
#1 0x00007f3a01ecbcc0 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a720 in ?? ()
#5 0x00007f3a01ecbcb0 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 3887):
#0 0x00007f3a17402fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 3886):
#0 0x00007f3a17402fb9 in ?? ()
#1 0x00007f3a02f09240 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007f3a02f09230 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 3885):
#0 0x00007f3a17402ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 3884):
#0 0x00007f3a10135a47 in ?? ()
#1 0x00007f3a03f58320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f3a03f58310 in ?? ()
#4 0x00007f3a03f58520 in ?? ()
#5 0x00007f3a03f58360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f3a03f583e0 in ?? ()
#8 0x00007f3a127a825d in ?? ()
#9 0x3fb952bf372fa000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x00000000c4000000 in ?? ()
#14 0x00007f3a00000000 in ?? ()
#15 0x41da17a869707646 in ?? ()
#16 0x00000fe7c07e307c in ?? ()
#17 0x00007f3a03f583c0 in ?? ()
#18 0x00007f3a127acba3 in ?? ()
#19 0x00007f3a03f58390 in ?? ()
#20 0x3fb952bf372fa000 in ?? ()
#21 0x0000000003f583e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb952bf372fa000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 3883):
#0 0x00007f3a10135a47 in ?? ()
#1 0x00007f3a0476f320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f3a0476f310 in ?? ()
#4 0x00007f3a0476f520 in ?? ()
#5 0x00007f3a0476f360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f3a0476f3e0 in ?? ()
#8 0x00007f3a127a825d in ?? ()
#9 0x3fb97015dae68000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x0000000090000000 in ?? ()
#14 0x00007f3a00000000 in ?? ()
#15 0x41da17a869707646 in ?? ()
#16 0x00000fe7c08e5e7c in ?? ()
#17 0x00007f3a0476f3c0 in ?? ()
#18 0x00007f3a127acba3 in ?? ()
#19 0x00007f3a0476f390 in ?? ()
#20 0x3fb97015dae68000 in ?? ()
#21 0x000000000476f3e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb97015dae68000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 3882):
#0 0x00007f3a10135a47 in ?? ()
#1 0x00007f3a04f86320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f3a04f86310 in ?? ()
#4 0x00007f3a04f86520 in ?? ()
#5 0x00007f3a04f86360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f3a04f863e0 in ?? ()
#8 0x00007f3a127a825d in ?? ()
#9 0x3fb96db60887d000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x00000000a4000000 in ?? ()
#14 0x00007f3a00000000 in ?? ()
#15 0x41da17a869707644 in ?? ()
#16 0x00000fe7c09e8c7c in ?? ()
#17 0x00007f3a04f863c0 in ?? ()
#18 0x00007f3a127acba3 in ?? ()
#19 0x00007f3a04f86390 in ?? ()
#20 0x3fb96db60887d000 in ?? ()
#21 0x0000000004f863e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb96db60887d000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 3881):
#0 0x00007f3a10135a47 in ?? ()
#1 0x00007f3a06fa5320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f3a06fa5310 in ?? ()
#4 0x00007f3a06fa5520 in ?? ()
#5 0x00007f3a06fa5360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007f3a06fa53e0 in ?? ()
#8 0x00007f3a127a825d in ?? ()
#9 0x3fb96d35df467000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x000000008e000000 in ?? ()
#14 0x00007f3a00000000 in ?? ()
#15 0x41da17a869707647 in ?? ()
#16 0x00000fe7c0deca7c in ?? ()
#17 0x00007f3a06fa53c0 in ?? ()
#18 0x00007f3a127acba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 3877):
#0 0x00007f3a10128cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f3a089b77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 3876):
#0 0x00007f3a17402fb9 in ?? ()
#1 0x00007f3a081b60c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001cf40 in ?? ()
#5 0x00007f3a081b60b0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 3875):
#0 0x00007f3a174069e2 in ?? ()
#1 0x00007f3a079b3ba0 in ?? ()
#2 0x00007f3a079b3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f3a079b3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007f3a079b3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f3a079b3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007f3a1cd19edc in ?? ()
#12 0x00007f3a1cd098d9 in ?? ()
#13 0x00007f3a1cd0dec6 in ?? ()
#14 0x00007f3a1cd12900 in ?? ()
#15 0x00007f3a1cd12729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007f3a13fde4de in ?? ()
#18 0x00007f3a173fc6db in ?? ()
#19 0x00000fe7c0f2e684 in ?? ()
#20 0x00007f3a079b3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 3868):
#0 0x00007f3a17402fb9 in ?? ()
#1 0x00007f3a099b8c80 in ?? ()
#2 0x000000000000035e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001e5a8 in ?? ()
#5 0x00007f3a099b8c70 in ?? ()
#6 0x00000000000006bc in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 3867):
#0 0x00007f3a17402fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 3866):
#0 0x00007f3a17402fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 3865):
#0 0x00007f3a17402fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f3a0b1bc450 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 3863):
#0 0x00007f3a17406d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 13:57:50.958745 1586 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 3 with UUID 99bf3fe9a41d4ce1b61e440b31778689 and pid 3314
************************ BEGIN STACKS **************************
[New LWP 3316]
[New LWP 3317]
[New LWP 3318]
[New LWP 3319]
[New LWP 3326]
[New LWP 3327]
[New LWP 3328]
[New LWP 3331]
[New LWP 3332]
[New LWP 3334]
[New LWP 3335]
[New LWP 3336]
[New LWP 3337]
[New LWP 3338]
[New LWP 3339]
[New LWP 3340]
[New LWP 3341]
[New LWP 3342]
[New LWP 3343]
[New LWP 3344]
[New LWP 3345]
[New LWP 3346]
[New LWP 3347]
[New LWP 3348]
[New LWP 3349]
[New LWP 3350]
[New LWP 3351]
[New LWP 3352]
[New LWP 3353]
[New LWP 3354]
[New LWP 3355]
[New LWP 3356]
[New LWP 3357]
[New LWP 3358]
[New LWP 3359]
[New LWP 3360]
[New LWP 3361]
[New LWP 3362]
[New LWP 3363]
[New LWP 3364]
[New LWP 3365]
[New LWP 3366]
[New LWP 3367]
[New LWP 3368]
[New LWP 3369]
[New LWP 3370]
[New LWP 3371]
[New LWP 3372]
[New LWP 3373]
[New LWP 3374]
[New LWP 3375]
[New LWP 3376]
[New LWP 3377]
[New LWP 3378]
[New LWP 3379]
[New LWP 3380]
[New LWP 3381]
[New LWP 3382]
[New LWP 3384]
[New LWP 3386]
[New LWP 3387]
[New LWP 3388]
[New LWP 3389]
[New LWP 3390]
[New LWP 3391]
[New LWP 3392]
[New LWP 3393]
[New LWP 3394]
[New LWP 3395]
[New LWP 3396]
[New LWP 3397]
[New LWP 3398]
[New LWP 3399]
[New LWP 3400]
[New LWP 3401]
[New LWP 3402]
[New LWP 3403]
[New LWP 3404]
[New LWP 3406]
[New LWP 3407]
[New LWP 3408]
[New LWP 3409]
[New LWP 3410]
[New LWP 3411]
[New LWP 3412]
[New LWP 3413]
[New LWP 3414]
[New LWP 3415]
[New LWP 3416]
[New LWP 3417]
[New LWP 3418]
[New LWP 3419]
[New LWP 3420]
[New LWP 3421]
[New LWP 3422]
[New LWP 3423]
[New LWP 3424]
[New LWP 3425]
[New LWP 3426]
[New LWP 3427]
[New LWP 3428]
[New LWP 3429]
[New LWP 3430]
[New LWP 3431]
[New LWP 3432]
[New LWP 3433]
[New LWP 3434]
[New LWP 3435]
[New LWP 3436]
[New LWP 3437]
[New LWP 3438]
[New LWP 3439]
[New LWP 3440]
[New LWP 3441]
[New LWP 3442]
[New LWP 3443]
[New LWP 3444]
[New LWP 3445]
[New LWP 3446]
[New LWP 3447]
[New LWP 3450]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007f961c606d50 in ?? ()
Id Target Id Frame
* 1 LWP 3314 "kudu" 0x00007f961c606d50 in ?? ()
2 LWP 3316 "kudu" 0x00007f961c602fb9 in ?? ()
3 LWP 3317 "kudu" 0x00007f961c602fb9 in ?? ()
4 LWP 3318 "kudu" 0x00007f961c602fb9 in ?? ()
5 LWP 3319 "kernel-watcher-" 0x00007f961c602fb9 in ?? ()
6 LWP 3326 "ntp client-3326" 0x00007f961c6069e2 in ?? ()
7 LWP 3327 "file cache-evic" 0x00007f961c602fb9 in ?? ()
8 LWP 3328 "sq_acceptor" 0x00007f9615328cb9 in ?? ()
9 LWP 3331 "rpc reactor-333" 0x00007f9615335a47 in ?? ()
10 LWP 3332 "rpc reactor-333" 0x00007f9615335a47 in ?? ()
11 LWP 3334 "rpc reactor-333" 0x00007f9615335a47 in ?? ()
12 LWP 3335 "rpc reactor-333" 0x00007f9615335a47 in ?? ()
13 LWP 3336 "MaintenanceMgr " 0x00007f961c602ad3 in ?? ()
14 LWP 3337 "txn-status-mana" 0x00007f961c602fb9 in ?? ()
15 LWP 3338 "collect_and_rem" 0x00007f961c602fb9 in ?? ()
16 LWP 3339 "tc-session-exp-" 0x00007f961c602fb9 in ?? ()
17 LWP 3340 "rpc worker-3340" 0x00007f961c602ad3 in ?? ()
18 LWP 3341 "rpc worker-3341" 0x00007f961c602ad3 in ?? ()
19 LWP 3342 "rpc worker-3342" 0x00007f961c602ad3 in ?? ()
20 LWP 3343 "rpc worker-3343" 0x00007f961c602ad3 in ?? ()
21 LWP 3344 "rpc worker-3344" 0x00007f961c602ad3 in ?? ()
22 LWP 3345 "rpc worker-3345" 0x00007f961c602ad3 in ?? ()
23 LWP 3346 "rpc worker-3346" 0x00007f961c602ad3 in ?? ()
24 LWP 3347 "rpc worker-3347" 0x00007f961c602ad3 in ?? ()
25 LWP 3348 "rpc worker-3348" 0x00007f961c602ad3 in ?? ()
26 LWP 3349 "rpc worker-3349" 0x00007f961c602ad3 in ?? ()
27 LWP 3350 "rpc worker-3350" 0x00007f961c602ad3 in ?? ()
28 LWP 3351 "rpc worker-3351" 0x00007f961c602ad3 in ?? ()
29 LWP 3352 "rpc worker-3352" 0x00007f961c602ad3 in ?? ()
30 LWP 3353 "rpc worker-3353" 0x00007f961c602ad3 in ?? ()
31 LWP 3354 "rpc worker-3354" 0x00007f961c602ad3 in ?? ()
32 LWP 3355 "rpc worker-3355" 0x00007f961c602ad3 in ?? ()
33 LWP 3356 "rpc worker-3356" 0x00007f961c602ad3 in ?? ()
34 LWP 3357 "rpc worker-3357" 0x00007f961c602ad3 in ?? ()
35 LWP 3358 "rpc worker-3358" 0x00007f961c602ad3 in ?? ()
36 LWP 3359 "rpc worker-3359" 0x00007f961c602ad3 in ?? ()
37 LWP 3360 "rpc worker-3360" 0x00007f961c602ad3 in ?? ()
38 LWP 3361 "rpc worker-3361" 0x00007f961c602ad3 in ?? ()
39 LWP 3362 "rpc worker-3362" 0x00007f961c602ad3 in ?? ()
40 LWP 3363 "rpc worker-3363" 0x00007f961c602ad3 in ?? ()
41 LWP 3364 "rpc worker-3364" 0x00007f961c602ad3 in ?? ()
42 LWP 3365 "rpc worker-3365" 0x00007f961c602ad3 in ?? ()
43 LWP 3366 "rpc worker-3366" 0x00007f961c602ad3 in ?? ()
44 LWP 3367 "rpc worker-3367" 0x00007f961c602ad3 in ?? ()
45 LWP 3368 "rpc worker-3368" 0x00007f961c602ad3 in ?? ()
46 LWP 3369 "rpc worker-3369" 0x00007f961c602ad3 in ?? ()
47 LWP 3370 "rpc worker-3370" 0x00007f961c602ad3 in ?? ()
48 LWP 3371 "rpc worker-3371" 0x00007f961c602ad3 in ?? ()
49 LWP 3372 "rpc worker-3372" 0x00007f961c602ad3 in ?? ()
50 LWP 3373 "rpc worker-3373" 0x00007f961c602ad3 in ?? ()
51 LWP 3374 "rpc worker-3374" 0x00007f961c602ad3 in ?? ()
52 LWP 3375 "rpc worker-3375" 0x00007f961c602ad3 in ?? ()
53 LWP 3376 "rpc worker-3376" 0x00007f961c602ad3 in ?? ()
54 LWP 3377 "rpc worker-3377" 0x00007f961c602ad3 in ?? ()
55 LWP 3378 "rpc worker-3378" 0x00007f961c602ad3 in ?? ()
56 LWP 3379 "rpc worker-3379" 0x00007f961c602ad3 in ?? ()
57 LWP 3380 "rpc worker-3380" 0x00007f961c602ad3 in ?? ()
58 LWP 3381 "rpc worker-3381" 0x00007f961c602ad3 in ?? ()
59 LWP 3382 "rpc worker-3382" 0x00007f961c602ad3 in ?? ()
60 LWP 3384 "rpc worker-3384" 0x00007f961c602ad3 in ?? ()
61 LWP 3386 "rpc worker-3386" 0x00007f961c602ad3 in ?? ()
62 LWP 3387 "rpc worker-3387" 0x00007f961c602ad3 in ?? ()
63 LWP 3388 "rpc worker-3388" 0x00007f961c602ad3 in ?? ()
64 LWP 3389 "rpc worker-3389" 0x00007f961c602ad3 in ?? ()
65 LWP 3390 "rpc worker-3390" 0x00007f961c602ad3 in ?? ()
66 LWP 3391 "rpc worker-3391" 0x00007f961c602ad3 in ?? ()
67 LWP 3392 "rpc worker-3392" 0x00007f961c602ad3 in ?? ()
68 LWP 3393 "rpc worker-3393" 0x00007f961c602ad3 in ?? ()
69 LWP 3394 "rpc worker-3394" 0x00007f961c602ad3 in ?? ()
70 LWP 3395 "rpc worker-3395" 0x00007f961c602ad3 in ?? ()
71 LWP 3396 "rpc worker-3396" 0x00007f961c602ad3 in ?? ()
72 LWP 3397 "rpc worker-3397" 0x00007f961c602ad3 in ?? ()
73 LWP 3398 "rpc worker-3398" 0x00007f961c602ad3 in ?? ()
74 LWP 3399 "rpc worker-3399" 0x00007f961c602ad3 in ?? ()
75 LWP 3400 "rpc worker-3400" 0x00007f961c602ad3 in ?? ()
76 LWP 3401 "rpc worker-3401" 0x00007f961c602ad3 in ?? ()
77 LWP 3402 "rpc worker-3402" 0x00007f961c602ad3 in ?? ()
78 LWP 3403 "rpc worker-3403" 0x00007f961c602ad3 in ?? ()
79 LWP 3404 "rpc worker-3404" 0x00007f961c602ad3 in ?? ()
80 LWP 3406 "rpc worker-3406" 0x00007f961c602ad3 in ?? ()
81 LWP 3407 "rpc worker-3407" 0x00007f961c602ad3 in ?? ()
82 LWP 3408 "rpc worker-3408" 0x00007f961c602ad3 in ?? ()
83 LWP 3409 "rpc worker-3409" 0x00007f961c602ad3 in ?? ()
84 LWP 3410 "rpc worker-3410" 0x00007f961c602ad3 in ?? ()
85 LWP 3411 "rpc worker-3411" 0x00007f961c602ad3 in ?? ()
86 LWP 3412 "rpc worker-3412" 0x00007f961c602ad3 in ?? ()
87 LWP 3413 "rpc worker-3413" 0x00007f961c602ad3 in ?? ()
88 LWP 3414 "rpc worker-3414" 0x00007f961c602ad3 in ?? ()
89 LWP 3415 "rpc worker-3415" 0x00007f961c602ad3 in ?? ()
90 LWP 3416 "rpc worker-3416" 0x00007f961c602ad3 in ?? ()
91 LWP 3417 "rpc worker-3417" 0x00007f961c602ad3 in ?? ()
92 LWP 3418 "rpc worker-3418" 0x00007f961c602ad3 in ?? ()
93 LWP 3419 "rpc worker-3419" 0x00007f961c602ad3 in ?? ()
94 LWP 3420 "rpc worker-3420" 0x00007f961c602ad3 in ?? ()
95 LWP 3421 "rpc worker-3421" 0x00007f961c602ad3 in ?? ()
96 LWP 3422 "rpc worker-3422" 0x00007f961c602ad3 in ?? ()
97 LWP 3423 "rpc worker-3423" 0x00007f961c602ad3 in ?? ()
98 LWP 3424 "rpc worker-3424" 0x00007f961c602ad3 in ?? ()
99 LWP 3425 "rpc worker-3425" 0x00007f961c602ad3 in ?? ()
100 LWP 3426 "rpc worker-3426" 0x00007f961c602ad3 in ?? ()
101 LWP 3427 "rpc worker-3427" 0x00007f961c602ad3 in ?? ()
102 LWP 3428 "rpc worker-3428" 0x00007f961c602ad3 in ?? ()
103 LWP 3429 "rpc worker-3429" 0x00007f961c602ad3 in ?? ()
104 LWP 3430 "rpc worker-3430" 0x00007f961c602ad3 in ?? ()
105 LWP 3431 "rpc worker-3431" 0x00007f961c602ad3 in ?? ()
106 LWP 3432 "rpc worker-3432" 0x00007f961c602ad3 in ?? ()
107 LWP 3433 "rpc worker-3433" 0x00007f961c602ad3 in ?? ()
108 LWP 3434 "rpc worker-3434" 0x00007f961c602ad3 in ?? ()
109 LWP 3435 "rpc worker-3435" 0x00007f961c602ad3 in ?? ()
110 LWP 3436 "rpc worker-3436" 0x00007f961c602ad3 in ?? ()
111 LWP 3437 "rpc worker-3437" 0x00007f961c602ad3 in ?? ()
112 LWP 3438 "rpc worker-3438" 0x00007f961c602ad3 in ?? ()
113 LWP 3439 "rpc worker-3439" 0x00007f961c602ad3 in ?? ()
114 LWP 3440 "rpc worker-3440" 0x00007f961c602ad3 in ?? ()
115 LWP 3441 "rpc worker-3441" 0x00007f961c602ad3 in ?? ()
116 LWP 3442 "rpc worker-3442" 0x00007f961c602ad3 in ?? ()
117 LWP 3443 "diag-logger-344" 0x00007f961c602fb9 in ?? ()
118 LWP 3444 "result-tracker-" 0x00007f961c602fb9 in ?? ()
119 LWP 3445 "excess-log-dele" 0x00007f961c602fb9 in ?? ()
120 LWP 3446 "acceptor-3446" 0x00007f96153370c7 in ?? ()
121 LWP 3447 "heartbeat-3447" 0x00007f961c602fb9 in ?? ()
122 LWP 3450 "maintenance_sch" 0x00007f961c602fb9 in ?? ()
Thread 122 (LWP 3450):
#0 0x00007f961c602fb9 in ?? ()
#1 0x00007f95d169b6e0 in ?? ()
#2 0x000000000000030c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f95d169b730 in ?? ()
#6 0x0000000000000618 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 3447):
#0 0x00007f961c602fb9 in ?? ()
#1 0x26b5f4687947e954 in ?? ()
#2 0x00000000000000c3 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019e80 in ?? ()
#5 0x00007f95d26c25f0 in ?? ()
#6 0x0000000000000186 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 3446):
#0 0x00007f96153370c7 in ?? ()
#1 0x00006140000214a8 in ?? ()
#2 0x00007f95d2ee9d90 in ?? ()
#3 0x00007f95d2ee9dc0 in ?? ()
#4 0x00007f95d2ee9ec0 in ?? ()
#5 0x00007f95d2ee9db0 in ?? ()
#6 0x00007f95d2ee9e20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x00007f96191bd960 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 3445):
#0 0x00007f961c602fb9 in ?? ()
#1 0x00007f95d3703f40 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffeb618d2f0 in ?? ()
#5 0x00007f95d3703f30 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 3444):
#0 0x00007f961c602fb9 in ?? ()
#1 0x00007f95d3f1c100 in ?? ()
#2 0x00000000000000c3 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f95d3f1c0f0 in ?? ()
#6 0x0000000000000186 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 3443):
#0 0x00007f961c602fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 3442):
#0 0x00007f961c602ad3 in ?? ()
#1 0x000000000000001c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007f95d4f4be90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f95d4f4beb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 3441):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000024 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002a6858 in ?? ()
#4 0x00007f95d5763e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f95d5763eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 3440):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 3439):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 3438):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 3437):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 3436):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 3435):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 3434):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 3433):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 3432):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 3431):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 3430):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 3429):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 3428):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 3427):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 3426):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 3425):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 3424):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 3423):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 3422):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 3421):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 3420):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 3419):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 3418):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 3417):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 3416):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 3415):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 3414):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 3413):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 3412):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 3411):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 3410):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 3409):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 3408):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 3407):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 3406):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 3404):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 3403):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 3402):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 3401):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a6888 in ?? ()
#4 0x00007f95e931ce90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f95e931ceb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 3400):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 3399):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 3398):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 3397):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 3396):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 3395):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 3394):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 3393):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 3392):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 3391):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 3390):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 3389):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 3388):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 3387):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 3386):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 3384):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 3382):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 3381):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 3380):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 3379):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000120038 in ?? ()
#4 0x00007f95f34fae90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f95f34faeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 3378):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 3377):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 3376):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 3375):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 3374):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 3373):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 3372):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 3371):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 3370):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 3369):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 3368):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 3367):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 3366):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 3365):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 3364):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 3363):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 3362):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 3361):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 3360):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 3359):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00009ffe8 in ?? ()
#4 0x00007f95fcec6e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f95fcec6eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 3358):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 3357):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 3356):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 3355):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 3354):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 3353):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 3352):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 3351):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 3350):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 3349):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 3348):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 3347):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 3346):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 3345):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 3344):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 3343):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 3342):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 3341):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 3340):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 3339):
#0 0x00007f961c602fb9 in ?? ()
#1 0x00007f9606fc6cc0 in ?? ()
#2 0x0000000000000013 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a720 in ?? ()
#5 0x00007f9606fc6cb0 in ?? ()
#6 0x0000000000000026 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 3338):
#0 0x00007f961c602fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 3337):
#0 0x00007f961c602fb9 in ?? ()
#1 0x00007f9607fc8240 in ?? ()
#2 0x0000000000000013 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007f9607fc8230 in ?? ()
#6 0x0000000000000026 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 3336):
#0 0x00007f961c602ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 3335):
#0 0x00007f9615335a47 in ?? ()
#1 0x00007f9608fcb320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f9608fcb310 in ?? ()
#4 0x00007f9608fcb520 in ?? ()
#5 0x00007f9608fcb360 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007f9608fcb3e0 in ?? ()
#8 0x00007f96179a825d in ?? ()
#9 0x3fb95c5a33aaf000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x0000000025800000 in ?? ()
#14 0x00007f9600000000 in ?? ()
#15 0x41da17a869707648 in ?? ()
#16 0x00000ff3411f167c in ?? ()
#17 0x00007f9608fcb3c0 in ?? ()
#18 0x00007f96179acba3 in ?? ()
#19 0x00007f9608fcb390 in ?? ()
#20 0x3fb95c5a33aaf000 in ?? ()
#21 0x0000000008fcb3e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb95c5a33aaf000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 3334):
#0 0x00007f9615335a47 in ?? ()
#1 0x00007f96097cc320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f96097cc310 in ?? ()
#4 0x00007f96097cc520 in ?? ()
#5 0x00007f96097cc360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f96097cc3e0 in ?? ()
#8 0x00007f96179a825d in ?? ()
#9 0x3f9a1afe004e4000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x0000000013600000 in ?? ()
#14 0x00007f9600000000 in ?? ()
#15 0x41da17a869707646 in ?? ()
#16 0x00000ff3412f187c in ?? ()
#17 0x00007f96097cc3c0 in ?? ()
#18 0x00007f96179acba3 in ?? ()
#19 0x00007f96097cc390 in ?? ()
#20 0x3f9a1afe004e4000 in ?? ()
#21 0x00000000097cc3e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3f9a1afe004e4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 3332):
#0 0x00007f9615335a47 in ?? ()
#1 0x00007f9609fcd320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f9609fcd310 in ?? ()
#4 0x00007f9609fcd520 in ?? ()
#5 0x00007f9609fcd360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f9609fcd3e0 in ?? ()
#8 0x00007f96179a825d in ?? ()
#9 0x3fb95a127047e000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x0000000078000000 in ?? ()
#14 0x00007f9600000000 in ?? ()
#15 0x41da17a869707645 in ?? ()
#16 0x00000ff3413f1a7c in ?? ()
#17 0x00007f9609fcd3c0 in ?? ()
#18 0x00007f96179acba3 in ?? ()
#19 0x00007f9609fcd390 in ?? ()
#20 0x3fb95a127047e000 in ?? ()
#21 0x0000000009fcd3e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb95a127047e000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 3331):
#0 0x00007f9615335a47 in ?? ()
#1 0x00007f960bbb3320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f960bbb3310 in ?? ()
#4 0x00007f960bbb3520 in ?? ()
#5 0x00007f960bbb3360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f960bbb33e0 in ?? ()
#8 0x00007f96179a825d in ?? ()
#9 0x3fb9606b31cc7000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x00000000e2000000 in ?? ()
#14 0x00007f9600000000 in ?? ()
#15 0x41da17a869707646 in ?? ()
#16 0x00000ff34176e67c in ?? ()
#17 0x00007f960bbb33c0 in ?? ()
#18 0x00007f96179acba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 3328):
#0 0x00007f9615328cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f960dbb77c8 in ?? ()
#3 0x000060200001dcd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 3327):
#0 0x00007f961c602fb9 in ?? ()
#1 0x00007f960d3b60c0 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001cf40 in ?? ()
#5 0x00007f960d3b60b0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 3326):
#0 0x00007f961c6069e2 in ?? ()
#1 0x00007f960c3b2ba0 in ?? ()
#2 0x00007f960c3b2c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f960c3b2bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007f960c3b2c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f960c3b2c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007f9621f19edc in ?? ()
#12 0x00007f9621f098d9 in ?? ()
#13 0x00007f9621f0dec6 in ?? ()
#14 0x00007f9621f12900 in ?? ()
#15 0x00007f9621f12729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007f96191de4de in ?? ()
#18 0x00007f961c5fc6db in ?? ()
#19 0x00000ff34186e484 in ?? ()
#20 0x00007f960c3b2440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 3319):
#0 0x00007f961c602fb9 in ?? ()
#1 0x00007f960ebb8c80 in ?? ()
#2 0x00000000000003d6 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001e5a8 in ?? ()
#5 0x00007f960ebb8c70 in ?? ()
#6 0x00000000000007ac in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 3318):
#0 0x00007f961c602fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 3317):
#0 0x00007f961c602fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 3316):
#0 0x00007f961c602fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f96103bc450 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 3314):
#0 0x00007f961c606d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 13:57:51.657729 1586 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 4 with UUID ffa5b64156aa4e0dbef89b3fb8111e44 and pid 3171
************************ BEGIN STACKS **************************
[New LWP 3176]
[New LWP 3177]
[New LWP 3178]
[New LWP 3179]
[New LWP 3185]
[New LWP 3186]
[New LWP 3187]
[New LWP 3190]
[New LWP 3191]
[New LWP 3192]
[New LWP 3193]
[New LWP 3194]
[New LWP 3195]
[New LWP 3196]
[New LWP 3197]
[New LWP 3198]
[New LWP 3199]
[New LWP 3200]
[New LWP 3201]
[New LWP 3203]
[New LWP 3204]
[New LWP 3205]
[New LWP 3206]
[New LWP 3207]
[New LWP 3208]
[New LWP 3209]
[New LWP 3210]
[New LWP 3211]
[New LWP 3212]
[New LWP 3213]
[New LWP 3214]
[New LWP 3215]
[New LWP 3216]
[New LWP 3217]
[New LWP 3218]
[New LWP 3219]
[New LWP 3220]
[New LWP 3221]
[New LWP 3222]
[New LWP 3223]
[New LWP 3224]
[New LWP 3225]
[New LWP 3226]
[New LWP 3227]
[New LWP 3228]
[New LWP 3229]
[New LWP 3230]
[New LWP 3231]
[New LWP 3232]
[New LWP 3233]
[New LWP 3234]
[New LWP 3235]
[New LWP 3236]
[New LWP 3237]
[New LWP 3238]
[New LWP 3239]
[New LWP 3240]
[New LWP 3241]
[New LWP 3242]
[New LWP 3243]
[New LWP 3244]
[New LWP 3245]
[New LWP 3246]
[New LWP 3247]
[New LWP 3248]
[New LWP 3249]
[New LWP 3250]
[New LWP 3251]
[New LWP 3252]
[New LWP 3253]
[New LWP 3255]
[New LWP 3256]
[New LWP 3257]
[New LWP 3258]
[New LWP 3259]
[New LWP 3260]
[New LWP 3261]
[New LWP 3262]
[New LWP 3263]
[New LWP 3264]
[New LWP 3265]
[New LWP 3266]
[New LWP 3267]
[New LWP 3268]
[New LWP 3269]
[New LWP 3270]
[New LWP 3271]
[New LWP 3272]
[New LWP 3273]
[New LWP 3274]
[New LWP 3275]
[New LWP 3276]
[New LWP 3277]
[New LWP 3278]
[New LWP 3281]
[New LWP 3282]
[New LWP 3283]
[New LWP 3284]
[New LWP 3285]
[New LWP 3286]
[New LWP 3287]
[New LWP 3288]
[New LWP 3289]
[New LWP 3290]
[New LWP 3291]
[New LWP 3292]
[New LWP 3293]
[New LWP 3294]
[New LWP 3295]
[New LWP 3296]
[New LWP 3297]
[New LWP 3298]
[New LWP 3299]
[New LWP 3300]
[New LWP 3301]
[New LWP 3302]
[New LWP 3303]
[New LWP 3304]
[New LWP 3305]
[New LWP 3306]
[New LWP 3311]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007fd6c331dd50 in ?? ()
Id Target Id Frame
* 1 LWP 3171 "kudu" 0x00007fd6c331dd50 in ?? ()
2 LWP 3176 "kudu" 0x00007fd6c3319fb9 in ?? ()
3 LWP 3177 "kudu" 0x00007fd6c3319fb9 in ?? ()
4 LWP 3178 "kudu" 0x00007fd6c3319fb9 in ?? ()
5 LWP 3179 "kernel-watcher-" 0x00007fd6c3319fb9 in ?? ()
6 LWP 3185 "ntp client-3185" 0x00007fd6c331d9e2 in ?? ()
7 LWP 3186 "file cache-evic" 0x00007fd6c3319fb9 in ?? ()
8 LWP 3187 "sq_acceptor" 0x00007fd6bc03fcb9 in ?? ()
9 LWP 3190 "rpc reactor-319" 0x00007fd6bc04ca47 in ?? ()
10 LWP 3191 "rpc reactor-319" 0x00007fd6bc04ca47 in ?? ()
11 LWP 3192 "rpc reactor-319" 0x00007fd6bc04ca47 in ?? ()
12 LWP 3193 "rpc reactor-319" 0x00007fd6bc04ca47 in ?? ()
13 LWP 3194 "MaintenanceMgr " 0x00007fd6c3319ad3 in ?? ()
14 LWP 3195 "txn-status-mana" 0x00007fd6c3319fb9 in ?? ()
15 LWP 3196 "collect_and_rem" 0x00007fd6c3319fb9 in ?? ()
16 LWP 3197 "tc-session-exp-" 0x00007fd6c3319fb9 in ?? ()
17 LWP 3198 "rpc worker-3198" 0x00007fd6c3319ad3 in ?? ()
18 LWP 3199 "rpc worker-3199" 0x00007fd6c3319ad3 in ?? ()
19 LWP 3200 "rpc worker-3200" 0x00007fd6c3319ad3 in ?? ()
20 LWP 3201 "rpc worker-3201" 0x00007fd6c3319ad3 in ?? ()
21 LWP 3203 "rpc worker-3203" 0x00007fd6c3319ad3 in ?? ()
22 LWP 3204 "rpc worker-3204" 0x00007fd6c3319ad3 in ?? ()
23 LWP 3205 "rpc worker-3205" 0x00007fd6c3319ad3 in ?? ()
24 LWP 3206 "rpc worker-3206" 0x00007fd6c3319ad3 in ?? ()
25 LWP 3207 "rpc worker-3207" 0x00007fd6c3319ad3 in ?? ()
26 LWP 3208 "rpc worker-3208" 0x00007fd6c3319ad3 in ?? ()
27 LWP 3209 "rpc worker-3209" 0x00007fd6c3319ad3 in ?? ()
28 LWP 3210 "rpc worker-3210" 0x00007fd6c3319ad3 in ?? ()
29 LWP 3211 "rpc worker-3211" 0x00007fd6c3319ad3 in ?? ()
30 LWP 3212 "rpc worker-3212" 0x00007fd6c3319ad3 in ?? ()
31 LWP 3213 "rpc worker-3213" 0x00007fd6c3319ad3 in ?? ()
32 LWP 3214 "rpc worker-3214" 0x00007fd6c3319ad3 in ?? ()
33 LWP 3215 "rpc worker-3215" 0x00007fd6c3319ad3 in ?? ()
34 LWP 3216 "rpc worker-3216" 0x00007fd6c3319ad3 in ?? ()
35 LWP 3217 "rpc worker-3217" 0x00007fd6c3319ad3 in ?? ()
36 LWP 3218 "rpc worker-3218" 0x00007fd6c3319ad3 in ?? ()
37 LWP 3219 "rpc worker-3219" 0x00007fd6c3319ad3 in ?? ()
38 LWP 3220 "rpc worker-3220" 0x00007fd6c3319ad3 in ?? ()
39 LWP 3221 "rpc worker-3221" 0x00007fd6c3319ad3 in ?? ()
40 LWP 3222 "rpc worker-3222" 0x00007fd6c3319ad3 in ?? ()
41 LWP 3223 "rpc worker-3223" 0x00007fd6c3319ad3 in ?? ()
42 LWP 3224 "rpc worker-3224" 0x00007fd6c3319ad3 in ?? ()
43 LWP 3225 "rpc worker-3225" 0x00007fd6c3319ad3 in ?? ()
44 LWP 3226 "rpc worker-3226" 0x00007fd6c3319ad3 in ?? ()
45 LWP 3227 "rpc worker-3227" 0x00007fd6c3319ad3 in ?? ()
46 LWP 3228 "rpc worker-3228" 0x00007fd6c3319ad3 in ?? ()
47 LWP 3229 "rpc worker-3229" 0x00007fd6c3319ad3 in ?? ()
48 LWP 3230 "rpc worker-3230" 0x00007fd6c3319ad3 in ?? ()
49 LWP 3231 "rpc worker-3231" 0x00007fd6c3319ad3 in ?? ()
50 LWP 3232 "rpc worker-3232" 0x00007fd6c3319ad3 in ?? ()
51 LWP 3233 "rpc worker-3233" 0x00007fd6c3319ad3 in ?? ()
52 LWP 3234 "rpc worker-3234" 0x00007fd6c3319ad3 in ?? ()
53 LWP 3235 "rpc worker-3235" 0x00007fd6c3319ad3 in ?? ()
54 LWP 3236 "rpc worker-3236" 0x00007fd6c3319ad3 in ?? ()
55 LWP 3237 "rpc worker-3237" 0x00007fd6c3319ad3 in ?? ()
56 LWP 3238 "rpc worker-3238" 0x00007fd6c3319ad3 in ?? ()
57 LWP 3239 "rpc worker-3239" 0x00007fd6c3319ad3 in ?? ()
58 LWP 3240 "rpc worker-3240" 0x00007fd6c3319ad3 in ?? ()
59 LWP 3241 "rpc worker-3241" 0x00007fd6c3319ad3 in ?? ()
60 LWP 3242 "rpc worker-3242" 0x00007fd6c3319ad3 in ?? ()
61 LWP 3243 "rpc worker-3243" 0x00007fd6c3319ad3 in ?? ()
62 LWP 3244 "rpc worker-3244" 0x00007fd6c3319ad3 in ?? ()
63 LWP 3245 "rpc worker-3245" 0x00007fd6c3319ad3 in ?? ()
64 LWP 3246 "rpc worker-3246" 0x00007fd6c3319ad3 in ?? ()
65 LWP 3247 "rpc worker-3247" 0x00007fd6c3319ad3 in ?? ()
66 LWP 3248 "rpc worker-3248" 0x00007fd6c3319ad3 in ?? ()
67 LWP 3249 "rpc worker-3249" 0x00007fd6c3319ad3 in ?? ()
68 LWP 3250 "rpc worker-3250" 0x00007fd6c3319ad3 in ?? ()
69 LWP 3251 "rpc worker-3251" 0x00007fd6c3319ad3 in ?? ()
70 LWP 3252 "rpc worker-3252" 0x00007fd6c3319ad3 in ?? ()
71 LWP 3253 "rpc worker-3253" 0x00007fd6c3319ad3 in ?? ()
72 LWP 3255 "rpc worker-3255" 0x00007fd6c3319ad3 in ?? ()
73 LWP 3256 "rpc worker-3256" 0x00007fd6c3319ad3 in ?? ()
74 LWP 3257 "rpc worker-3257" 0x00007fd6c3319ad3 in ?? ()
75 LWP 3258 "rpc worker-3258" 0x00007fd6c3319ad3 in ?? ()
76 LWP 3259 "rpc worker-3259" 0x00007fd6c3319ad3 in ?? ()
77 LWP 3260 "rpc worker-3260" 0x00007fd6c3319ad3 in ?? ()
78 LWP 3261 "rpc worker-3261" 0x00007fd6c3319ad3 in ?? ()
79 LWP 3262 "rpc worker-3262" 0x00007fd6c3319ad3 in ?? ()
80 LWP 3263 "rpc worker-3263" 0x00007fd6c3319ad3 in ?? ()
81 LWP 3264 "rpc worker-3264" 0x00007fd6c3319ad3 in ?? ()
82 LWP 3265 "rpc worker-3265" 0x00007fd6c3319ad3 in ?? ()
83 LWP 3266 "rpc worker-3266" 0x00007fd6c3319ad3 in ?? ()
84 LWP 3267 "rpc worker-3267" 0x00007fd6c3319ad3 in ?? ()
85 LWP 3268 "rpc worker-3268" 0x00007fd6c3319ad3 in ?? ()
86 LWP 3269 "rpc worker-3269" 0x00007fd6c3319ad3 in ?? ()
87 LWP 3270 "rpc worker-3270" 0x00007fd6c3319ad3 in ?? ()
88 LWP 3271 "rpc worker-3271" 0x00007fd6c3319ad3 in ?? ()
89 LWP 3272 "rpc worker-3272" 0x00007fd6c3319ad3 in ?? ()
90 LWP 3273 "rpc worker-3273" 0x00007fd6c3319ad3 in ?? ()
91 LWP 3274 "rpc worker-3274" 0x00007fd6c3319ad3 in ?? ()
92 LWP 3275 "rpc worker-3275" 0x00007fd6c3319ad3 in ?? ()
93 LWP 3276 "rpc worker-3276" 0x00007fd6c3319ad3 in ?? ()
94 LWP 3277 "rpc worker-3277" 0x00007fd6c3319ad3 in ?? ()
95 LWP 3278 "rpc worker-3278" 0x00007fd6c3319ad3 in ?? ()
96 LWP 3281 "rpc worker-3281" 0x00007fd6c3319ad3 in ?? ()
97 LWP 3282 "rpc worker-3282" 0x00007fd6c3319ad3 in ?? ()
98 LWP 3283 "rpc worker-3283" 0x00007fd6c3319ad3 in ?? ()
99 LWP 3284 "rpc worker-3284" 0x00007fd6c3319ad3 in ?? ()
100 LWP 3285 "rpc worker-3285" 0x00007fd6c3319ad3 in ?? ()
101 LWP 3286 "rpc worker-3286" 0x00007fd6c3319ad3 in ?? ()
102 LWP 3287 "rpc worker-3287" 0x00007fd6c3319ad3 in ?? ()
103 LWP 3288 "rpc worker-3288" 0x00007fd6c3319ad3 in ?? ()
104 LWP 3289 "rpc worker-3289" 0x00007fd6c3319ad3 in ?? ()
105 LWP 3290 "rpc worker-3290" 0x00007fd6c3319ad3 in ?? ()
106 LWP 3291 "rpc worker-3291" 0x00007fd6c3319ad3 in ?? ()
107 LWP 3292 "rpc worker-3292" 0x00007fd6c3319ad3 in ?? ()
108 LWP 3293 "rpc worker-3293" 0x00007fd6c3319ad3 in ?? ()
109 LWP 3294 "rpc worker-3294" 0x00007fd6c3319ad3 in ?? ()
110 LWP 3295 "rpc worker-3295" 0x00007fd6c3319ad3 in ?? ()
111 LWP 3296 "rpc worker-3296" 0x00007fd6c3319ad3 in ?? ()
112 LWP 3297 "rpc worker-3297" 0x00007fd6c3319ad3 in ?? ()
113 LWP 3298 "rpc worker-3298" 0x00007fd6c3319ad3 in ?? ()
114 LWP 3299 "rpc worker-3299" 0x00007fd6c3319ad3 in ?? ()
115 LWP 3300 "rpc worker-3300" 0x00007fd6c3319ad3 in ?? ()
116 LWP 3301 "rpc worker-3301" 0x00007fd6c3319ad3 in ?? ()
117 LWP 3302 "diag-logger-330" 0x00007fd6c3319fb9 in ?? ()
118 LWP 3303 "result-tracker-" 0x00007fd6c3319fb9 in ?? ()
119 LWP 3304 "excess-log-dele" 0x00007fd6c3319fb9 in ?? ()
120 LWP 3305 "acceptor-3305" 0x00007fd6bc04e0c7 in ?? ()
121 LWP 3306 "heartbeat-3306" 0x00007fd6c3319fb9 in ?? ()
122 LWP 3311 "maintenance_sch" 0x00007fd6c3319fb9 in ?? ()
Thread 122 (LWP 3311):
#0 0x00007fd6c3319fb9 in ?? ()
#1 0x00007fd6783b26e0 in ?? ()
#2 0x0000000000000319 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007fd6783b2730 in ?? ()
#6 0x0000000000000632 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 3306):
#0 0x00007fd6c3319fb9 in ?? ()
#1 0x26b5f4687947e954 in ?? ()
#2 0x00000000000000c6 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019e80 in ?? ()
#5 0x00007fd6793d95f0 in ?? ()
#6 0x000000000000018c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 3305):
#0 0x00007fd6bc04e0c7 in ?? ()
#1 0x00006140000210a8 in ?? ()
#2 0x00007fd679c00d90 in ?? ()
#3 0x00007fd679c00dc0 in ?? ()
#4 0x00007fd679c00ec0 in ?? ()
#5 0x00007fd679c00db0 in ?? ()
#6 0x00007fd679c00e20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x00007fd6bfed4960 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 3304):
#0 0x00007fd6c3319fb9 in ?? ()
#1 0x00007fd67a41af40 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff27b0b550 in ?? ()
#5 0x00007fd67a41af30 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 3303):
#0 0x00007fd6c3319fb9 in ?? ()
#1 0x00007fd67ac33100 in ?? ()
#2 0x00000000000000c6 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007fd67ac330f0 in ?? ()
#6 0x000000000000018c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 3302):
#0 0x00007fd6c3319fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 3301):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x000000000000001c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007fd67bc62e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd67bc62eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 3300):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000024 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002a6858 in ?? ()
#4 0x00007fd67c47ae90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd67c47aeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 3299):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 3298):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 3297):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 3296):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 3295):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 3294):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 3293):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 3292):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 3291):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 3290):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 3289):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 3288):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 3287):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 3286):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 3285):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 3284):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 3283):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 3282):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 3281):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 3278):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 3277):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 3276):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 3275):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 3274):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 3273):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 3272):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 3271):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 3270):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 3269):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 3268):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 3267):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 3266):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 3265):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 3264):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 3263):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 3262):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 3261):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 3260):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 3259):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a6888 in ?? ()
#4 0x00007fd690033e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd690033eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 3258):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 3257):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 3256):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 3255):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 3253):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 3252):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 3251):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 3250):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 3249):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 3248):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 3247):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 3246):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 3245):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 3244):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 3243):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 3242):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 3241):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 3240):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 3239):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 3238):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000120038 in ?? ()
#4 0x00007fd6999fce90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd6999fceb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 3237):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 3236):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 3235):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 3234):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 3233):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 3232):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 3231):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 3230):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 3229):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 3228):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 3227):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 3226):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 3225):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 3224):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 3223):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 3222):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 3221):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 3220):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 3219):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 3218):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00009ffe8 in ?? ()
#4 0x00007fd6a3bdde90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd6a3bddeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 3217):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 3216):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 3215):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 3214):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 3213):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 3212):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 3211):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 3210):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 3209):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 3208):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 3207):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 3206):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 3205):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 3204):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 3203):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 3201):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 3200):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 3199):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 3198):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 3197):
#0 0x00007fd6c3319fb9 in ?? ()
#1 0x00007fd6adcc6cc0 in ?? ()
#2 0x0000000000000013 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a720 in ?? ()
#5 0x00007fd6adcc6cb0 in ?? ()
#6 0x0000000000000026 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 3196):
#0 0x00007fd6c3319fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 3195):
#0 0x00007fd6c3319fb9 in ?? ()
#1 0x00007fd6aecc8240 in ?? ()
#2 0x0000000000000013 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007fd6aecc8230 in ?? ()
#6 0x0000000000000026 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 3194):
#0 0x00007fd6c3319ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 3193):
#0 0x00007fd6bc04ca47 in ?? ()
#1 0x00007fd6afccb320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007fd6afccb310 in ?? ()
#4 0x00007fd6afccb520 in ?? ()
#5 0x00007fd6afccb360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fd6afccb3e0 in ?? ()
#8 0x00007fd6be6bf25d in ?? ()
#9 0x3fb950dc26340000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x0000000048000000 in ?? ()
#14 0x00007fd600000000 in ?? ()
#15 0x41da17a869707646 in ?? ()
#16 0x00000ffb55f9167c in ?? ()
#17 0x00007fd6afccb3c0 in ?? ()
#18 0x00007fd6be6c3ba3 in ?? ()
#19 0x00007fd6afccb390 in ?? ()
#20 0x3fb950dc26340000 in ?? ()
#21 0x00000000afccb3e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb950dc26340000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 3192):
#0 0x00007fd6bc04ca47 in ?? ()
#1 0x00007fd6b04cc320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007fd6b04cc310 in ?? ()
#4 0x00007fd6b04cc520 in ?? ()
#5 0x00007fd6b04cc360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007fd6b04cc3e0 in ?? ()
#8 0x00007fd6be6bf25d in ?? ()
#9 0x3f9731f26c244000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x0000000075e00000 in ?? ()
#14 0x00007fd600000000 in ?? ()
#15 0x41da17a869707646 in ?? ()
#16 0x00000ffb5609187c in ?? ()
#17 0x00007fd6b04cc3c0 in ?? ()
#18 0x00007fd6be6c3ba3 in ?? ()
#19 0x00007fd6b04cc390 in ?? ()
#20 0x3f9731f26c244000 in ?? ()
#21 0x00000000b04cc3e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3f9731f26c244000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 3191):
#0 0x00007fd6bc04ca47 in ?? ()
#1 0x00007fd6b0ccd320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007fd6b0ccd310 in ?? ()
#4 0x00007fd6b0ccd520 in ?? ()
#5 0x00007fd6b0ccd360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007fd6b0ccd3e0 in ?? ()
#8 0x00007fd6be6bf25d in ?? ()
#9 0x3fb98026ce703000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x0000000070000000 in ?? ()
#14 0x00007fd600000000 in ?? ()
#15 0x41da17a869707644 in ?? ()
#16 0x00000ffb56191a7c in ?? ()
#17 0x00007fd6b0ccd3c0 in ?? ()
#18 0x00007fd6be6c3ba3 in ?? ()
#19 0x00007fd6b0ccd390 in ?? ()
#20 0x3fb98026ce703000 in ?? ()
#21 0x00000000b0ccd3e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb98026ce703000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 3190):
#0 0x00007fd6bc04ca47 in ?? ()
#1 0x00007fd6b28b3320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007fd6b28b3310 in ?? ()
#4 0x00007fd6b28b3520 in ?? ()
#5 0x00007fd6b28b3360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007fd6b28b33e0 in ?? ()
#8 0x00007fd6be6bf25d in ?? ()
#9 0x3fb95e095c085000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x00000000d8000000 in ?? ()
#14 0x00007fd600000000 in ?? ()
#15 0x41da17a869707645 in ?? ()
#16 0x00000ffb5650e67c in ?? ()
#17 0x00007fd6b28b33c0 in ?? ()
#18 0x00007fd6be6c3ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 3187):
#0 0x00007fd6bc03fcb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fd6b48b77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 3186):
#0 0x00007fd6c3319fb9 in ?? ()
#1 0x00007fd6b40b60c0 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001cf40 in ?? ()
#5 0x00007fd6b40b60b0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 3185):
#0 0x00007fd6c331d9e2 in ?? ()
#1 0x00007fd6b30b2ba0 in ?? ()
#2 0x00007fd6b30b2c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fd6b30b2bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007fd6b30b2c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007fd6b30b2c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007fd6c8c30edc in ?? ()
#12 0x00007fd6c8c208d9 in ?? ()
#13 0x00007fd6c8c24ec6 in ?? ()
#14 0x00007fd6c8c29900 in ?? ()
#15 0x00007fd6c8c29729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007fd6bfef54de in ?? ()
#18 0x00007fd6c33136db in ?? ()
#19 0x00000ffb5660e484 in ?? ()
#20 0x00007fd6b30b2440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 3179):
#0 0x00007fd6c3319fb9 in ?? ()
#1 0x00007fd6b58b8c80 in ?? ()
#2 0x00000000000003e4 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001e5a8 in ?? ()
#5 0x00007fd6b58b8c70 in ?? ()
#6 0x00000000000007c8 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 3178):
#0 0x00007fd6c3319fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 3177):
#0 0x00007fd6c3319fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 3176):
#0 0x00007fd6c3319fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007fd6b70bc450 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 3171):
#0 0x00007fd6c331dd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 13:57:52.340159 1586 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 5 with UUID aa90f1d263644537b32ad1a4595aef92 and pid 3029
************************ BEGIN STACKS **************************
[New LWP 3033]
[New LWP 3034]
[New LWP 3035]
[New LWP 3036]
[New LWP 3042]
[New LWP 3043]
[New LWP 3044]
[New LWP 3048]
[New LWP 3049]
[New LWP 3050]
[New LWP 3051]
[New LWP 3052]
[New LWP 3053]
[New LWP 3054]
[New LWP 3055]
[New LWP 3056]
[New LWP 3057]
[New LWP 3058]
[New LWP 3059]
[New LWP 3060]
[New LWP 3061]
[New LWP 3062]
[New LWP 3063]
[New LWP 3064]
[New LWP 3065]
[New LWP 3066]
[New LWP 3067]
[New LWP 3068]
[New LWP 3069]
[New LWP 3070]
[New LWP 3071]
[New LWP 3072]
[New LWP 3073]
[New LWP 3074]
[New LWP 3075]
[New LWP 3076]
[New LWP 3077]
[New LWP 3078]
[New LWP 3079]
[New LWP 3080]
[New LWP 3081]
[New LWP 3082]
[New LWP 3083]
[New LWP 3084]
[New LWP 3085]
[New LWP 3086]
[New LWP 3087]
[New LWP 3088]
[New LWP 3089]
[New LWP 3090]
[New LWP 3091]
[New LWP 3092]
[New LWP 3093]
[New LWP 3094]
[New LWP 3095]
[New LWP 3096]
[New LWP 3097]
[New LWP 3098]
[New LWP 3099]
[New LWP 3100]
[New LWP 3101]
[New LWP 3102]
[New LWP 3103]
[New LWP 3104]
[New LWP 3105]
[New LWP 3108]
[New LWP 3109]
[New LWP 3110]
[New LWP 3111]
[New LWP 3112]
[New LWP 3113]
[New LWP 3114]
[New LWP 3115]
[New LWP 3116]
[New LWP 3117]
[New LWP 3118]
[New LWP 3119]
[New LWP 3120]
[New LWP 3121]
[New LWP 3122]
[New LWP 3123]
[New LWP 3124]
[New LWP 3125]
[New LWP 3126]
[New LWP 3127]
[New LWP 3128]
[New LWP 3129]
[New LWP 3130]
[New LWP 3131]
[New LWP 3132]
[New LWP 3133]
[New LWP 3134]
[New LWP 3135]
[New LWP 3136]
[New LWP 3137]
[New LWP 3138]
[New LWP 3139]
[New LWP 3140]
[New LWP 3141]
[New LWP 3142]
[New LWP 3143]
[New LWP 3144]
[New LWP 3145]
[New LWP 3146]
[New LWP 3147]
[New LWP 3148]
[New LWP 3149]
[New LWP 3151]
[New LWP 3152]
[New LWP 3153]
[New LWP 3154]
[New LWP 3155]
[New LWP 3156]
[New LWP 3157]
[New LWP 3158]
[New LWP 3159]
[New LWP 3160]
[New LWP 3161]
[New LWP 3162]
[New LWP 3163]
[New LWP 3164]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007f264dd8bd50 in ?? ()
Id Target Id Frame
* 1 LWP 3029 "kudu" 0x00007f264dd8bd50 in ?? ()
2 LWP 3033 "kudu" 0x00007f264dd87fb9 in ?? ()
3 LWP 3034 "kudu" 0x00007f264dd87fb9 in ?? ()
4 LWP 3035 "kudu" 0x00007f264dd87fb9 in ?? ()
5 LWP 3036 "kernel-watcher-" 0x00007f264dd87fb9 in ?? ()
6 LWP 3042 "ntp client-3042" 0x00007f264dd8b9e2 in ?? ()
7 LWP 3043 "file cache-evic" 0x00007f264dd87fb9 in ?? ()
8 LWP 3044 "sq_acceptor" 0x00007f2646aadcb9 in ?? ()
9 LWP 3048 "rpc reactor-304" 0x00007f2646abaa47 in ?? ()
10 LWP 3049 "rpc reactor-304" 0x00007f2646abaa47 in ?? ()
11 LWP 3050 "rpc reactor-305" 0x00007f2646abaa47 in ?? ()
12 LWP 3051 "rpc reactor-305" 0x00007f2646abaa47 in ?? ()
13 LWP 3052 "MaintenanceMgr " 0x00007f264dd87ad3 in ?? ()
14 LWP 3053 "txn-status-mana" 0x00007f264dd87fb9 in ?? ()
15 LWP 3054 "collect_and_rem" 0x00007f264dd87fb9 in ?? ()
16 LWP 3055 "tc-session-exp-" 0x00007f264dd87fb9 in ?? ()
17 LWP 3056 "rpc worker-3056" 0x00007f264dd87ad3 in ?? ()
18 LWP 3057 "rpc worker-3057" 0x00007f264dd87ad3 in ?? ()
19 LWP 3058 "rpc worker-3058" 0x00007f264dd87ad3 in ?? ()
20 LWP 3059 "rpc worker-3059" 0x00007f264dd87ad3 in ?? ()
21 LWP 3060 "rpc worker-3060" 0x00007f264dd87ad3 in ?? ()
22 LWP 3061 "rpc worker-3061" 0x00007f264dd87ad3 in ?? ()
23 LWP 3062 "rpc worker-3062" 0x00007f264dd87ad3 in ?? ()
24 LWP 3063 "rpc worker-3063" 0x00007f264dd87ad3 in ?? ()
25 LWP 3064 "rpc worker-3064" 0x00007f264dd87ad3 in ?? ()
26 LWP 3065 "rpc worker-3065" 0x00007f264dd87ad3 in ?? ()
27 LWP 3066 "rpc worker-3066" 0x00007f264dd87ad3 in ?? ()
28 LWP 3067 "rpc worker-3067" 0x00007f264dd87ad3 in ?? ()
29 LWP 3068 "rpc worker-3068" 0x00007f264dd87ad3 in ?? ()
30 LWP 3069 "rpc worker-3069" 0x00007f264dd87ad3 in ?? ()
31 LWP 3070 "rpc worker-3070" 0x00007f264dd87ad3 in ?? ()
32 LWP 3071 "rpc worker-3071" 0x00007f264dd87ad3 in ?? ()
33 LWP 3072 "rpc worker-3072" 0x00007f264dd87ad3 in ?? ()
34 LWP 3073 "rpc worker-3073" 0x00007f264dd87ad3 in ?? ()
35 LWP 3074 "rpc worker-3074" 0x00007f264dd87ad3 in ?? ()
36 LWP 3075 "rpc worker-3075" 0x00007f264dd87ad3 in ?? ()
37 LWP 3076 "rpc worker-3076" 0x00007f264dd87ad3 in ?? ()
38 LWP 3077 "rpc worker-3077" 0x00007f264dd87ad3 in ?? ()
39 LWP 3078 "rpc worker-3078" 0x00007f264dd87ad3 in ?? ()
40 LWP 3079 "rpc worker-3079" 0x00007f264dd87ad3 in ?? ()
41 LWP 3080 "rpc worker-3080" 0x00007f264dd87ad3 in ?? ()
42 LWP 3081 "rpc worker-3081" 0x00007f264dd87ad3 in ?? ()
43 LWP 3082 "rpc worker-3082" 0x00007f264dd87ad3 in ?? ()
44 LWP 3083 "rpc worker-3083" 0x00007f264dd87ad3 in ?? ()
45 LWP 3084 "rpc worker-3084" 0x00007f264dd87ad3 in ?? ()
46 LWP 3085 "rpc worker-3085" 0x00007f264dd87ad3 in ?? ()
47 LWP 3086 "rpc worker-3086" 0x00007f264dd87ad3 in ?? ()
48 LWP 3087 "rpc worker-3087" 0x00007f264dd87ad3 in ?? ()
49 LWP 3088 "rpc worker-3088" 0x00007f264dd87ad3 in ?? ()
50 LWP 3089 "rpc worker-3089" 0x00007f264dd87ad3 in ?? ()
51 LWP 3090 "rpc worker-3090" 0x00007f264dd87ad3 in ?? ()
52 LWP 3091 "rpc worker-3091" 0x00007f264dd87ad3 in ?? ()
53 LWP 3092 "rpc worker-3092" 0x00007f264dd87ad3 in ?? ()
54 LWP 3093 "rpc worker-3093" 0x00007f264dd87ad3 in ?? ()
55 LWP 3094 "rpc worker-3094" 0x00007f264dd87ad3 in ?? ()
56 LWP 3095 "rpc worker-3095" 0x00007f264dd87ad3 in ?? ()
57 LWP 3096 "rpc worker-3096" 0x00007f264dd87ad3 in ?? ()
58 LWP 3097 "rpc worker-3097" 0x00007f264dd87ad3 in ?? ()
59 LWP 3098 "rpc worker-3098" 0x00007f264dd87ad3 in ?? ()
60 LWP 3099 "rpc worker-3099" 0x00007f264dd87ad3 in ?? ()
61 LWP 3100 "rpc worker-3100" 0x00007f264dd87ad3 in ?? ()
62 LWP 3101 "rpc worker-3101" 0x00007f264dd87ad3 in ?? ()
63 LWP 3102 "rpc worker-3102" 0x00007f264dd87ad3 in ?? ()
64 LWP 3103 "rpc worker-3103" 0x00007f264dd87ad3 in ?? ()
65 LWP 3104 "rpc worker-3104" 0x00007f264dd87ad3 in ?? ()
66 LWP 3105 "rpc worker-3105" 0x00007f264dd87ad3 in ?? ()
67 LWP 3108 "rpc worker-3108" 0x00007f264dd87ad3 in ?? ()
68 LWP 3109 "rpc worker-3109" 0x00007f264dd87ad3 in ?? ()
69 LWP 3110 "rpc worker-3110" 0x00007f264dd87ad3 in ?? ()
70 LWP 3111 "rpc worker-3111" 0x00007f264dd87ad3 in ?? ()
71 LWP 3112 "rpc worker-3112" 0x00007f264dd87ad3 in ?? ()
72 LWP 3113 "rpc worker-3113" 0x00007f264dd87ad3 in ?? ()
73 LWP 3114 "rpc worker-3114" 0x00007f264dd87ad3 in ?? ()
74 LWP 3115 "rpc worker-3115" 0x00007f264dd87ad3 in ?? ()
75 LWP 3116 "rpc worker-3116" 0x00007f264dd87ad3 in ?? ()
76 LWP 3117 "rpc worker-3117" 0x00007f264dd87ad3 in ?? ()
77 LWP 3118 "rpc worker-3118" 0x00007f264dd87ad3 in ?? ()
78 LWP 3119 "rpc worker-3119" 0x00007f264dd87ad3 in ?? ()
79 LWP 3120 "rpc worker-3120" 0x00007f264dd87ad3 in ?? ()
80 LWP 3121 "rpc worker-3121" 0x00007f264dd87ad3 in ?? ()
81 LWP 3122 "rpc worker-3122" 0x00007f264dd87ad3 in ?? ()
82 LWP 3123 "rpc worker-3123" 0x00007f264dd87ad3 in ?? ()
83 LWP 3124 "rpc worker-3124" 0x00007f264dd87ad3 in ?? ()
84 LWP 3125 "rpc worker-3125" 0x00007f264dd87ad3 in ?? ()
85 LWP 3126 "rpc worker-3126" 0x00007f264dd87ad3 in ?? ()
86 LWP 3127 "rpc worker-3127" 0x00007f264dd87ad3 in ?? ()
87 LWP 3128 "rpc worker-3128" 0x00007f264dd87ad3 in ?? ()
88 LWP 3129 "rpc worker-3129" 0x00007f264dd87ad3 in ?? ()
89 LWP 3130 "rpc worker-3130" 0x00007f264dd87ad3 in ?? ()
90 LWP 3131 "rpc worker-3131" 0x00007f264dd87ad3 in ?? ()
91 LWP 3132 "rpc worker-3132" 0x00007f264dd87ad3 in ?? ()
92 LWP 3133 "rpc worker-3133" 0x00007f264dd87ad3 in ?? ()
93 LWP 3134 "rpc worker-3134" 0x00007f264dd87ad3 in ?? ()
94 LWP 3135 "rpc worker-3135" 0x00007f264dd87ad3 in ?? ()
95 LWP 3136 "rpc worker-3136" 0x00007f264dd87ad3 in ?? ()
96 LWP 3137 "rpc worker-3137" 0x00007f264dd87ad3 in ?? ()
97 LWP 3138 "rpc worker-3138" 0x00007f264dd87ad3 in ?? ()
98 LWP 3139 "rpc worker-3139" 0x00007f264dd87ad3 in ?? ()
99 LWP 3140 "rpc worker-3140" 0x00007f264dd87ad3 in ?? ()
100 LWP 3141 "rpc worker-3141" 0x00007f264dd87ad3 in ?? ()
101 LWP 3142 "rpc worker-3142" 0x00007f264dd87ad3 in ?? ()
102 LWP 3143 "rpc worker-3143" 0x00007f264dd87ad3 in ?? ()
103 LWP 3144 "rpc worker-3144" 0x00007f264dd87ad3 in ?? ()
104 LWP 3145 "rpc worker-3145" 0x00007f264dd87ad3 in ?? ()
105 LWP 3146 "rpc worker-3146" 0x00007f264dd87ad3 in ?? ()
106 LWP 3147 "rpc worker-3147" 0x00007f264dd87ad3 in ?? ()
107 LWP 3148 "rpc worker-3148" 0x00007f264dd87ad3 in ?? ()
108 LWP 3149 "rpc worker-3149" 0x00007f264dd87ad3 in ?? ()
109 LWP 3151 "rpc worker-3151" 0x00007f264dd87ad3 in ?? ()
110 LWP 3152 "rpc worker-3152" 0x00007f264dd87ad3 in ?? ()
111 LWP 3153 "rpc worker-3153" 0x00007f264dd87ad3 in ?? ()
112 LWP 3154 "rpc worker-3154" 0x00007f264dd87ad3 in ?? ()
113 LWP 3155 "rpc worker-3155" 0x00007f264dd87ad3 in ?? ()
114 LWP 3156 "rpc worker-3156" 0x00007f264dd87ad3 in ?? ()
115 LWP 3157 "rpc worker-3157" 0x00007f264dd87ad3 in ?? ()
116 LWP 3158 "rpc worker-3158" 0x00007f264dd87ad3 in ?? ()
117 LWP 3159 "diag-logger-315" 0x00007f264dd87fb9 in ?? ()
118 LWP 3160 "result-tracker-" 0x00007f264dd87fb9 in ?? ()
119 LWP 3161 "excess-log-dele" 0x00007f264dd87fb9 in ?? ()
120 LWP 3162 "acceptor-3162" 0x00007f2646abc0c7 in ?? ()
121 LWP 3163 "heartbeat-3163" 0x00007f264dd87fb9 in ?? ()
122 LWP 3164 "maintenance_sch" 0x00007f264dd87fb9 in ?? ()
Thread 122 (LWP 3164):
#0 0x00007f264dd87fb9 in ?? ()
#1 0x00007f260362f6e0 in ?? ()
#2 0x0000000000000326 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f260362f730 in ?? ()
#6 0x000000000000064c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 3163):
#0 0x00007f264dd87fb9 in ?? ()
#1 0x26b5f4687947e954 in ?? ()
#2 0x00000000000000c9 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019e80 in ?? ()
#5 0x00007f2603e475f0 in ?? ()
#6 0x0000000000000192 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 3162):
#0 0x00007f2646abc0c7 in ?? ()
#1 0x00006140000216a8 in ?? ()
#2 0x00007f260466ed90 in ?? ()
#3 0x00007f260466edc0 in ?? ()
#4 0x00007f260466eec0 in ?? ()
#5 0x00007f260466edb0 in ?? ()
#6 0x00007f260466ee20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x00007f264a942960 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 3161):
#0 0x00007f264dd87fb9 in ?? ()
#1 0x00007f2604e88f40 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffccf07aad0 in ?? ()
#5 0x00007f2604e88f30 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 3160):
#0 0x00007f264dd87fb9 in ?? ()
#1 0x00007f26056a1100 in ?? ()
#2 0x00000000000000c9 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f26056a10f0 in ?? ()
#6 0x0000000000000192 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 3159):
#0 0x00007f264dd87fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 3158):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007f26066d0e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f26066d0eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 3157):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000028 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002a6858 in ?? ()
#4 0x00007f2606ee8e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2606ee8eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 3156):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 3155):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 3154):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 3153):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 3152):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 3151):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 3149):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 3148):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 3147):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 3146):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 3145):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 3144):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 3143):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 3142):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 3141):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 3140):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 3139):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 3138):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 3137):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 3136):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 3135):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 3134):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 3133):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 3132):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 3131):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 3130):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 3129):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 3128):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 3127):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 3126):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 3125):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 3124):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 3123):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 3122):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 3121):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 3120):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 3119):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 3118):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 3117):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a6888 in ?? ()
#4 0x00007f261aaa1e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f261aaa1eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 3116):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 3115):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 3114):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 3113):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 3112):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 3111):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 3110):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 3109):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 3108):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 3105):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 3104):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 3103):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 3102):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 3101):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 3100):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 3099):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 3098):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 3097):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 3096):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 3095):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000120038 in ?? ()
#4 0x00007f262446ae90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f262446aeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 3094):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 3093):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 3092):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 3091):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 3090):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 3089):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 3088):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 3087):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 3086):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 3085):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 3084):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 3083):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 3082):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 3081):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 3080):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 3079):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 3078):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 3077):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 3076):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 3075):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00009ffe8 in ?? ()
#4 0x00007f262e64be90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f262e64beb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 3074):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 3073):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 3072):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 3071):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 3070):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 3069):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 3068):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 3067):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 3066):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 3065):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 3064):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 3063):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 3062):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 3061):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 3060):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 3059):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 3058):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 3057):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 3056):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 3055):
#0 0x00007f264dd87fb9 in ?? ()
#1 0x00007f263884bcc0 in ?? ()
#2 0x0000000000000014 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a720 in ?? ()
#5 0x00007f263884bcb0 in ?? ()
#6 0x0000000000000028 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 3054):
#0 0x00007f264dd87fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 3053):
#0 0x00007f264dd87fb9 in ?? ()
#1 0x00007f2639889240 in ?? ()
#2 0x0000000000000014 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007f2639889230 in ?? ()
#6 0x0000000000000028 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 3052):
#0 0x00007f264dd87ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 3051):
#0 0x00007f2646abaa47 in ?? ()
#1 0x00007f263a8b2320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f263a8b2310 in ?? ()
#4 0x00007f263a8b2520 in ?? ()
#5 0x00007f263a8b2360 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007f263a8b23e0 in ?? ()
#8 0x00007f264912d25d in ?? ()
#9 0x3fb97d9b3acb3000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x00000000f4000000 in ?? ()
#14 0x00007f2600000000 in ?? ()
#15 0x41da17a869707644 in ?? ()
#16 0x00000fe54750e47c in ?? ()
#17 0x00007f263a8b23c0 in ?? ()
#18 0x00007f2649131ba3 in ?? ()
#19 0x00007f263a8b2390 in ?? ()
#20 0x3fb97d9b3acb3000 in ?? ()
#21 0x000000003a8b23e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb97d9b3acb3000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 3050):
#0 0x00007f2646abaa47 in ?? ()
#1 0x00007f263b0b3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f263b0b3310 in ?? ()
#4 0x00007f263b0b3520 in ?? ()
#5 0x00007f263b0b3360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f263b0b33e0 in ?? ()
#8 0x00007f264912d25d in ?? ()
#9 0x3fb94f9b8dac4000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x0000000062000000 in ?? ()
#14 0x00007f2600000000 in ?? ()
#15 0x41da17a869707644 in ?? ()
#16 0x00000fe54760e67c in ?? ()
#17 0x00007f263b0b33c0 in ?? ()
#18 0x00007f2649131ba3 in ?? ()
#19 0x00007f263b0b3390 in ?? ()
#20 0x3fb94f9b8dac4000 in ?? ()
#21 0x000000003b0b33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb94f9b8dac4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 3049):
#0 0x00007f2646abaa47 in ?? ()
#1 0x00007f263b8b4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f263b8b4310 in ?? ()
#4 0x00007f263b8b4520 in ?? ()
#5 0x00007f263b8b4360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f263b8b43e0 in ?? ()
#8 0x00007f264912d25d in ?? ()
#9 0x3fb95b7dd545d000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x00000000f6800000 in ?? ()
#14 0x00007f2600000000 in ?? ()
#15 0x41da17a869707646 in ?? ()
#16 0x00000fe54770e87c in ?? ()
#17 0x00007f263b8b43c0 in ?? ()
#18 0x00007f2649131ba3 in ?? ()
#19 0x00007f263b8b4390 in ?? ()
#20 0x3fb95b7dd545d000 in ?? ()
#21 0x000000003b8b43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb95b7dd545d000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 3048):
#0 0x00007f2646abaa47 in ?? ()
#1 0x00007f263d8ad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f263d8ad310 in ?? ()
#4 0x00007f263d8ad520 in ?? ()
#5 0x00007f263d8ad360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f263d8ad3e0 in ?? ()
#8 0x00007f264912d25d in ?? ()
#9 0x3fb978f5d361d000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x0000000079800000 in ?? ()
#14 0x00007f2600000000 in ?? ()
#15 0x41da17a869707644 in ?? ()
#16 0x00000fe547b0da7c in ?? ()
#17 0x00007f263d8ad3c0 in ?? ()
#18 0x00007f2649131ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 3044):
#0 0x00007f2646aadcb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f263f2b77c8 in ?? ()
#3 0x000060200001ea10 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 3043):
#0 0x00007f264dd87fb9 in ?? ()
#1 0x00007f263eab60c0 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001cf40 in ?? ()
#5 0x00007f263eab60b0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 3042):
#0 0x00007f264dd8b9e2 in ?? ()
#1 0x00007f263e2b3ba0 in ?? ()
#2 0x00007f263e2b3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f263e2b3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007f263e2b3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f263e2b3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007f265369eedc in ?? ()
#12 0x00007f265368e8d9 in ?? ()
#13 0x00007f2653692ec6 in ?? ()
#14 0x00007f2653697900 in ?? ()
#15 0x00007f2653697729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007f264a9634de in ?? ()
#18 0x00007f264dd816db in ?? ()
#19 0x00000fe547c4e684 in ?? ()
#20 0x00007f263e2b3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 3036):
#0 0x00007f264dd87fb9 in ?? ()
#1 0x00007f26402b8c80 in ?? ()
#2 0x00000000000003f5 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001e5a8 in ?? ()
#5 0x00007f26402b8c70 in ?? ()
#6 0x00000000000007ea in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 3035):
#0 0x00007f264dd87fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 3034):
#0 0x00007f264dd87fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 3033):
#0 0x00007f264dd87fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f2641abc450 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 3029):
#0 0x00007f264dd8bd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 13:57:53.042795 1586 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 6 with UUID 373a9aec2682487a8ff2a4f0d1ee87ab and pid 4441
************************ BEGIN STACKS **************************
[New LWP 4444]
[New LWP 4445]
[New LWP 4446]
[New LWP 4447]
[New LWP 4453]
[New LWP 4454]
[New LWP 4455]
[New LWP 4458]
[New LWP 4459]
[New LWP 4460]
[New LWP 4461]
[New LWP 4462]
[New LWP 4463]
[New LWP 4465]
[New LWP 4466]
[New LWP 4467]
[New LWP 4468]
[New LWP 4469]
[New LWP 4470]
[New LWP 4471]
[New LWP 4472]
[New LWP 4473]
[New LWP 4474]
[New LWP 4475]
[New LWP 4476]
[New LWP 4477]
[New LWP 4478]
[New LWP 4479]
[New LWP 4480]
[New LWP 4481]
[New LWP 4482]
[New LWP 4483]
[New LWP 4484]
[New LWP 4485]
[New LWP 4487]
[New LWP 4488]
[New LWP 4491]
[New LWP 4492]
[New LWP 4493]
[New LWP 4494]
[New LWP 4495]
[New LWP 4496]
[New LWP 4497]
[New LWP 4498]
[New LWP 4499]
[New LWP 4500]
[New LWP 4501]
[New LWP 4502]
[New LWP 4503]
[New LWP 4504]
[New LWP 4505]
[New LWP 4506]
[New LWP 4507]
[New LWP 4508]
[New LWP 4509]
[New LWP 4510]
[New LWP 4511]
[New LWP 4512]
[New LWP 4513]
[New LWP 4514]
[New LWP 4515]
[New LWP 4516]
[New LWP 4517]
[New LWP 4518]
[New LWP 4519]
[New LWP 4520]
[New LWP 4521]
[New LWP 4522]
[New LWP 4523]
[New LWP 4524]
[New LWP 4525]
[New LWP 4526]
[New LWP 4528]
[New LWP 4529]
[New LWP 4531]
[New LWP 4532]
[New LWP 4533]
[New LWP 4534]
[New LWP 4535]
[New LWP 4536]
[New LWP 4537]
[New LWP 4538]
[New LWP 4539]
[New LWP 4540]
[New LWP 4541]
[New LWP 4542]
[New LWP 4543]
[New LWP 4544]
[New LWP 4545]
[New LWP 4546]
[New LWP 4547]
[New LWP 4548]
[New LWP 4549]
[New LWP 4550]
[New LWP 4551]
[New LWP 4552]
[New LWP 4553]
[New LWP 4554]
[New LWP 4555]
[New LWP 4556]
[New LWP 4557]
[New LWP 4558]
[New LWP 4559]
[New LWP 4561]
[New LWP 4562]
[New LWP 4563]
[New LWP 4564]
[New LWP 4565]
[New LWP 4566]
[New LWP 4567]
[New LWP 4568]
[New LWP 4569]
[New LWP 4570]
[New LWP 4571]
[New LWP 4572]
[New LWP 4574]
[New LWP 4575]
[New LWP 4576]
[New LWP 4577]
[New LWP 4578]
[New LWP 4581]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007febfc553d50 in ?? ()
Id Target Id Frame
* 1 LWP 4441 "kudu" 0x00007febfc553d50 in ?? ()
2 LWP 4444 "kudu" 0x00007febfc54ffb9 in ?? ()
3 LWP 4445 "kudu" 0x00007febfc54ffb9 in ?? ()
4 LWP 4446 "kudu" 0x00007febfc54ffb9 in ?? ()
5 LWP 4447 "kernel-watcher-" 0x00007febfc54ffb9 in ?? ()
6 LWP 4453 "ntp client-4453" 0x00007febfc5539e2 in ?? ()
7 LWP 4454 "file cache-evic" 0x00007febfc54ffb9 in ?? ()
8 LWP 4455 "sq_acceptor" 0x00007febf5275cb9 in ?? ()
9 LWP 4458 "rpc reactor-445" 0x00007febf5282a47 in ?? ()
10 LWP 4459 "rpc reactor-445" 0x00007febf5282a47 in ?? ()
11 LWP 4460 "rpc reactor-446" 0x00007febf5282a47 in ?? ()
12 LWP 4461 "rpc reactor-446" 0x00007febf5282a47 in ?? ()
13 LWP 4462 "MaintenanceMgr " 0x00007febfc54fad3 in ?? ()
14 LWP 4463 "txn-status-mana" 0x00007febfc54ffb9 in ?? ()
15 LWP 4465 "collect_and_rem" 0x00007febfc54ffb9 in ?? ()
16 LWP 4466 "tc-session-exp-" 0x00007febfc54ffb9 in ?? ()
17 LWP 4467 "rpc worker-4467" 0x00007febfc54fad3 in ?? ()
18 LWP 4468 "rpc worker-4468" 0x00007febfc54fad3 in ?? ()
19 LWP 4469 "rpc worker-4469" 0x00007febfc54fad3 in ?? ()
20 LWP 4470 "rpc worker-4470" 0x00007febfc54fad3 in ?? ()
21 LWP 4471 "rpc worker-4471" 0x00007febfc54fad3 in ?? ()
22 LWP 4472 "rpc worker-4472" 0x00007febfc54fad3 in ?? ()
23 LWP 4473 "rpc worker-4473" 0x00007febfc54fad3 in ?? ()
24 LWP 4474 "rpc worker-4474" 0x00007febfc54fad3 in ?? ()
25 LWP 4475 "rpc worker-4475" 0x00007febfc54fad3 in ?? ()
26 LWP 4476 "rpc worker-4476" 0x00007febfc54fad3 in ?? ()
27 LWP 4477 "rpc worker-4477" 0x00007febfc54fad3 in ?? ()
28 LWP 4478 "rpc worker-4478" 0x00007febfc54fad3 in ?? ()
29 LWP 4479 "rpc worker-4479" 0x00007febfc54fad3 in ?? ()
30 LWP 4480 "rpc worker-4480" 0x00007febfc54fad3 in ?? ()
31 LWP 4481 "rpc worker-4481" 0x00007febfc54fad3 in ?? ()
32 LWP 4482 "rpc worker-4482" 0x00007febfc54fad3 in ?? ()
33 LWP 4483 "rpc worker-4483" 0x00007febfc54fad3 in ?? ()
34 LWP 4484 "rpc worker-4484" 0x00007febfc54fad3 in ?? ()
35 LWP 4485 "rpc worker-4485" 0x00007febfc54fad3 in ?? ()
36 LWP 4487 "rpc worker-4487" 0x00007febfc54fad3 in ?? ()
37 LWP 4488 "rpc worker-4488" 0x00007febfc54fad3 in ?? ()
38 LWP 4491 "rpc worker-4491" 0x00007febfc54fad3 in ?? ()
39 LWP 4492 "rpc worker-4492" 0x00007febfc54fad3 in ?? ()
40 LWP 4493 "rpc worker-4493" 0x00007febfc54fad3 in ?? ()
41 LWP 4494 "rpc worker-4494" 0x00007febfc54fad3 in ?? ()
42 LWP 4495 "rpc worker-4495" 0x00007febfc54fad3 in ?? ()
43 LWP 4496 "rpc worker-4496" 0x00007febfc54fad3 in ?? ()
44 LWP 4497 "rpc worker-4497" 0x00007febfc54fad3 in ?? ()
45 LWP 4498 "rpc worker-4498" 0x00007febfc54fad3 in ?? ()
46 LWP 4499 "rpc worker-4499" 0x00007febfc54fad3 in ?? ()
47 LWP 4500 "rpc worker-4500" 0x00007febfc54fad3 in ?? ()
48 LWP 4501 "rpc worker-4501" 0x00007febfc54fad3 in ?? ()
49 LWP 4502 "rpc worker-4502" 0x00007febfc54fad3 in ?? ()
50 LWP 4503 "rpc worker-4503" 0x00007febfc54fad3 in ?? ()
51 LWP 4504 "rpc worker-4504" 0x00007febfc54fad3 in ?? ()
52 LWP 4505 "rpc worker-4505" 0x00007febfc54fad3 in ?? ()
53 LWP 4506 "rpc worker-4506" 0x00007febfc54fad3 in ?? ()
54 LWP 4507 "rpc worker-4507" 0x00007febfc54fad3 in ?? ()
55 LWP 4508 "rpc worker-4508" 0x00007febfc54fad3 in ?? ()
56 LWP 4509 "rpc worker-4509" 0x00007febfc54fad3 in ?? ()
57 LWP 4510 "rpc worker-4510" 0x00007febfc54fad3 in ?? ()
58 LWP 4511 "rpc worker-4511" 0x00007febfc54fad3 in ?? ()
59 LWP 4512 "rpc worker-4512" 0x00007febfc54fad3 in ?? ()
60 LWP 4513 "rpc worker-4513" 0x00007febfc54fad3 in ?? ()
61 LWP 4514 "rpc worker-4514" 0x00007febfc54fad3 in ?? ()
62 LWP 4515 "rpc worker-4515" 0x00007febfc54fad3 in ?? ()
63 LWP 4516 "rpc worker-4516" 0x00007febfc54fad3 in ?? ()
64 LWP 4517 "rpc worker-4517" 0x00007febfc54fad3 in ?? ()
65 LWP 4518 "rpc worker-4518" 0x00007febfc54fad3 in ?? ()
66 LWP 4519 "rpc worker-4519" 0x00007febfc54fad3 in ?? ()
67 LWP 4520 "rpc worker-4520" 0x00007febfc54fad3 in ?? ()
68 LWP 4521 "rpc worker-4521" 0x00007febfc54fad3 in ?? ()
69 LWP 4522 "rpc worker-4522" 0x00007febfc54fad3 in ?? ()
70 LWP 4523 "rpc worker-4523" 0x00007febfc54fad3 in ?? ()
71 LWP 4524 "rpc worker-4524" 0x00007febfc54fad3 in ?? ()
72 LWP 4525 "rpc worker-4525" 0x00007febfc54fad3 in ?? ()
73 LWP 4526 "rpc worker-4526" 0x00007febfc54fad3 in ?? ()
74 LWP 4528 "rpc worker-4528" 0x00007febfc54fad3 in ?? ()
75 LWP 4529 "rpc worker-4529" 0x00007febfc54fad3 in ?? ()
76 LWP 4531 "rpc worker-4531" 0x00007febfc54fad3 in ?? ()
77 LWP 4532 "rpc worker-4532" 0x00007febfc54fad3 in ?? ()
78 LWP 4533 "rpc worker-4533" 0x00007febfc54fad3 in ?? ()
79 LWP 4534 "rpc worker-4534" 0x00007febfc54fad3 in ?? ()
80 LWP 4535 "rpc worker-4535" 0x00007febfc54fad3 in ?? ()
81 LWP 4536 "rpc worker-4536" 0x00007febfc54fad3 in ?? ()
82 LWP 4537 "rpc worker-4537" 0x00007febfc54fad3 in ?? ()
83 LWP 4538 "rpc worker-4538" 0x00007febfc54fad3 in ?? ()
84 LWP 4539 "rpc worker-4539" 0x00007febfc54fad3 in ?? ()
85 LWP 4540 "rpc worker-4540" 0x00007febfc54fad3 in ?? ()
86 LWP 4541 "rpc worker-4541" 0x00007febfc54fad3 in ?? ()
87 LWP 4542 "rpc worker-4542" 0x00007febfc54fad3 in ?? ()
88 LWP 4543 "rpc worker-4543" 0x00007febfc54fad3 in ?? ()
89 LWP 4544 "rpc worker-4544" 0x00007febfc54fad3 in ?? ()
90 LWP 4545 "rpc worker-4545" 0x00007febfc54fad3 in ?? ()
91 LWP 4546 "rpc worker-4546" 0x00007febfc54fad3 in ?? ()
92 LWP 4547 "rpc worker-4547" 0x00007febfc54fad3 in ?? ()
93 LWP 4548 "rpc worker-4548" 0x00007febfc54fad3 in ?? ()
94 LWP 4549 "rpc worker-4549" 0x00007febfc54fad3 in ?? ()
95 LWP 4550 "rpc worker-4550" 0x00007febfc54fad3 in ?? ()
96 LWP 4551 "rpc worker-4551" 0x00007febfc54fad3 in ?? ()
97 LWP 4552 "rpc worker-4552" 0x00007febfc54fad3 in ?? ()
98 LWP 4553 "rpc worker-4553" 0x00007febfc54fad3 in ?? ()
99 LWP 4554 "rpc worker-4554" 0x00007febfc54fad3 in ?? ()
100 LWP 4555 "rpc worker-4555" 0x00007febfc54fad3 in ?? ()
101 LWP 4556 "rpc worker-4556" 0x00007febfc54fad3 in ?? ()
102 LWP 4557 "rpc worker-4557" 0x00007febfc54fad3 in ?? ()
103 LWP 4558 "rpc worker-4558" 0x00007febfc54fad3 in ?? ()
104 LWP 4559 "rpc worker-4559" 0x00007febfc54fad3 in ?? ()
105 LWP 4561 "rpc worker-4561" 0x00007febfc54fad3 in ?? ()
106 LWP 4562 "rpc worker-4562" 0x00007febfc54fad3 in ?? ()
107 LWP 4563 "rpc worker-4563" 0x00007febfc54fad3 in ?? ()
108 LWP 4564 "rpc worker-4564" 0x00007febfc54fad3 in ?? ()
109 LWP 4565 "rpc worker-4565" 0x00007febfc54fad3 in ?? ()
110 LWP 4566 "rpc worker-4566" 0x00007febfc54fad3 in ?? ()
111 LWP 4567 "rpc worker-4567" 0x00007febfc54fad3 in ?? ()
112 LWP 4568 "rpc worker-4568" 0x00007febfc54fad3 in ?? ()
113 LWP 4569 "rpc worker-4569" 0x00007febfc54fad3 in ?? ()
114 LWP 4570 "rpc worker-4570" 0x00007febfc54fad3 in ?? ()
115 LWP 4571 "rpc worker-4571" 0x00007febfc54fad3 in ?? ()
116 LWP 4572 "rpc worker-4572" 0x00007febfc54fad3 in ?? ()
117 LWP 4574 "diag-logger-457" 0x00007febfc54ffb9 in ?? ()
118 LWP 4575 "result-tracker-" 0x00007febfc54ffb9 in ?? ()
119 LWP 4576 "excess-log-dele" 0x00007febfc54ffb9 in ?? ()
120 LWP 4577 "acceptor-4577" 0x00007febf52840c7 in ?? ()
121 LWP 4578 "heartbeat-4578" 0x00007febfc54ffb9 in ?? ()
122 LWP 4581 "maintenance_sch" 0x00007febfc54ffb9 in ?? ()
Thread 122 (LWP 4581):
#0 0x00007febfc54ffb9 in ?? ()
#1 0x00007febb04466e0 in ?? ()
#2 0x000000000000023a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007febb0446730 in ?? ()
#6 0x0000000000000474 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 4578):
#0 0x00007febfc54ffb9 in ?? ()
#1 0x26b5f4687947e954 in ?? ()
#2 0x000000000000008f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019e80 in ?? ()
#5 0x00007febb147d5f0 in ?? ()
#6 0x000000000000011e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 4577):
#0 0x00007febf52840c7 in ?? ()
#1 0x00006140000212a8 in ?? ()
#2 0x00007febb1c94d90 in ?? ()
#3 0x00007febb1c94dc0 in ?? ()
#4 0x00007febb1c94ec0 in ?? ()
#5 0x00007febb1c94db0 in ?? ()
#6 0x00007febb1c94e20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x00007febf910a960 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 4576):
#0 0x00007febfc54ffb9 in ?? ()
#1 0x00007febb24adf40 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff917ad4b0 in ?? ()
#5 0x00007febb24adf30 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 4575):
#0 0x00007febfc54ffb9 in ?? ()
#1 0x00007febb2cc6100 in ?? ()
#2 0x000000000000008e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007febb2cc60f0 in ?? ()
#6 0x000000000000011c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 4574):
#0 0x00007febfc54ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 4572):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000014 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002e6818 in ?? ()
#4 0x00007febb3d06e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007febb3d06eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 4571):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000020 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002e0018 in ?? ()
#4 0x00007febb451ee90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007febb451eeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 4570):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 4569):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 4568):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 4567):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 4566):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 4565):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 4564):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 4563):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 4562):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 4561):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 4559):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 4558):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 4557):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 4556):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 4555):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 4554):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 4553):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 4552):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 4551):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 4550):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 4549):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 4548):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 4547):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 4546):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 4545):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 4544):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 4543):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 4542):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 4541):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 4540):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 4539):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 4538):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 4537):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 4536):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 4535):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 4534):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 4533):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 4532):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 4531):
#0 0x00007febfc54fad3 in ?? ()
#1 0x00000000000005f4 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007febc80e7e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007febc80e7eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 4529):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000001cb9 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a688c in ?? ()
#4 0x00007febc9117e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007febc9117eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a6840 in ?? ()
#9 0x00007febfc54f770 in ?? ()
#10 0x00007febc9117eb0 in ?? ()
#11 0x00000ffd79222fd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 74 (LWP 4528):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000001277 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a008c in ?? ()
#4 0x00007febc992fe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007febc992feb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a0040 in ?? ()
#9 0x00007febfc54f770 in ?? ()
#10 0x00007febc992feb0 in ?? ()
#11 0x00000ffd79325fd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 73 (LWP 4526):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 4525):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 4524):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 4523):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 4522):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 4521):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 4520):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 4519):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 4518):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 4517):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 4516):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 4515):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 4514):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 4513):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 4512):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 4511):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 4510):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 4509):
#0 0x00007febfc54fad3 in ?? ()
#1 0x000000000000000d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007febd2bf2e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007febd2bf2eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007febfc54f770 in ?? ()
#10 0x00007febd2bf2eb0 in ?? ()
#11 0x00000ffd7a57e5d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 4508):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 4507):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 4506):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 4505):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 4504):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 4503):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 4502):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 4501):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 4500):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 4499):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 4498):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 4497):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 4496):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 4495):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 4494):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 4493):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 4492):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 4491):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 4488):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 4487):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000002f14 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007febdc5bbe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007febdc5bbeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 4485):
#0 0x00007febfc54fad3 in ?? ()
#1 0x000000000000272e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00009ffe8 in ?? ()
#4 0x00007febdcdd3e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007febdcdd3eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 34 (LWP 4484):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000002da0 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000967f8 in ?? ()
#4 0x00007febdd5ebe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007febdd5ebeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 33 (LWP 4483):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000002763 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00008fffc in ?? ()
#4 0x00007febdde03e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007febdde03eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00008ffb0 in ?? ()
#9 0x00007febfc54f770 in ?? ()
#10 0x00007febdde03eb0 in ?? ()
#11 0x00000ffd7bbc07d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 32 (LWP 4482):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000001920 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00008d008 in ?? ()
#4 0x00007febde61be90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007febde61beb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 31 (LWP 4481):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000002268 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000086808 in ?? ()
#4 0x00007febdee33e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007febdee33eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 30 (LWP 4480):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 4479):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 4478):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 4477):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 4476):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 4475):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 4474):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 4473):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 4472):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 4471):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 4470):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 4469):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 4468):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 4467):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 4466):
#0 0x00007febfc54ffb9 in ?? ()
#1 0x00007febe67e0cc0 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ae20 in ?? ()
#5 0x00007febe67e0cb0 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 4465):
#0 0x00007febfc54ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 4463):
#0 0x00007febfc54ffb9 in ?? ()
#1 0x00007febe8055240 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007febe8055230 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 4462):
#0 0x00007febfc54fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 4461):
#0 0x00007febf5282a47 in ?? ()
#1 0x00007febe90a4320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007febe90a4310 in ?? ()
#4 0x00007febe90a4520 in ?? ()
#5 0x00007febe90a4360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007febe90a43e0 in ?? ()
#8 0x00007febf78f525d in ?? ()
#9 0x3fb98291e6fd2000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x00000000c0800000 in ?? ()
#14 0x00007feb00000000 in ?? ()
#15 0x41da17a869707646 in ?? ()
#16 0x00000ffdfd20c87c in ?? ()
#17 0x00007febe90a43c0 in ?? ()
#18 0x00007febf78f9ba3 in ?? ()
#19 0x00007febe90a4390 in ?? ()
#20 0x3fb98291e6fd2000 in ?? ()
#21 0x00000000e90a43e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb98291e6fd2000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 4460):
#0 0x00007febf5282a47 in ?? ()
#1 0x00007febe98b3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007febe98b3310 in ?? ()
#4 0x00007febe98b3520 in ?? ()
#5 0x00007febe98b3360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007febe98b33e0 in ?? ()
#8 0x00007febf78f525d in ?? ()
#9 0x3fb98a6857033000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x00000000be000000 in ?? ()
#14 0x00007feb00000000 in ?? ()
#15 0x41da17a869707646 in ?? ()
#16 0x00000ffdfd30e67c in ?? ()
#17 0x00007febe98b33c0 in ?? ()
#18 0x00007febf78f9ba3 in ?? ()
#19 0x00007febe98b3390 in ?? ()
#20 0x3fb98a6857033000 in ?? ()
#21 0x00000000e98b33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb98a6857033000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 4459):
#0 0x00007febf5282a47 in ?? ()
#1 0x00007febea0b4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007febea0b4310 in ?? ()
#4 0x00007febea0b4520 in ?? ()
#5 0x00007febea0b4360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007febea0b43e0 in ?? ()
#8 0x00007febf78f525d in ?? ()
#9 0x3fae943f22e6c000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x00000000f4000000 in ?? ()
#14 0x00007feb00000000 in ?? ()
#15 0x41da17a869707643 in ?? ()
#16 0x00000ffdfd40e87c in ?? ()
#17 0x00007febea0b43c0 in ?? ()
#18 0x00007febf78f9ba3 in ?? ()
#19 0x00007febea0b4390 in ?? ()
#20 0x3fae943f22e6c000 in ?? ()
#21 0x00000000ea0b43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fae943f22e6c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 4458):
#0 0x00007febf5282a47 in ?? ()
#1 0x00007febec0ad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007febec0ad310 in ?? ()
#4 0x00007febec0ad520 in ?? ()
#5 0x00007febec0ad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007febec0ad3e0 in ?? ()
#8 0x00007febf78f525d in ?? ()
#9 0x3fb954ffc3b5b000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x0000000014000000 in ?? ()
#14 0x00007feb00000000 in ?? ()
#15 0x41da17a869707648 in ?? ()
#16 0x00000ffdfd80da7c in ?? ()
#17 0x00007febec0ad3c0 in ?? ()
#18 0x00007febf78f9ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 4455):
#0 0x00007febf5275cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007febedab77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 4454):
#0 0x00007febfc54ffb9 in ?? ()
#1 0x00007febed2b60c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001cf40 in ?? ()
#5 0x00007febed2b60b0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 4453):
#0 0x00007febfc5539e2 in ?? ()
#1 0x00007febecab3ba0 in ?? ()
#2 0x00007febecab3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007febecab3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007febecab3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007febecab3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007fec01e66edc in ?? ()
#12 0x00007fec01e568d9 in ?? ()
#13 0x00007fec01e5aec6 in ?? ()
#14 0x00007fec01e5f900 in ?? ()
#15 0x00007fec01e5f729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007febf912b4de in ?? ()
#18 0x00007febfc5496db in ?? ()
#19 0x00000ffdfd94e684 in ?? ()
#20 0x00007febecab3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 4447):
#0 0x00007febfc54ffb9 in ?? ()
#1 0x00007febeeab8c80 in ?? ()
#2 0x00000000000002d1 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001e5a8 in ?? ()
#5 0x00007febeeab8c70 in ?? ()
#6 0x00000000000005a2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 4446):
#0 0x00007febfc54ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 4445):
#0 0x00007febfc54ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 4444):
#0 0x00007febfc54ffb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007febf02bc450 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 4441):
#0 0x00007febfc553d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 13:57:53.757709 1586 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 7 with UUID f9d895c7b77646baaff4f9a677e8c69d and pid 5289
************************ BEGIN STACKS **************************
[New LWP 5290]
[New LWP 5291]
[New LWP 5292]
[New LWP 5293]
[New LWP 5299]
[New LWP 5300]
[New LWP 5301]
[New LWP 5306]
[New LWP 5307]
[New LWP 5308]
[New LWP 5309]
[New LWP 5310]
[New LWP 5311]
[New LWP 5312]
[New LWP 5313]
[New LWP 5314]
[New LWP 5315]
[New LWP 5316]
[New LWP 5317]
[New LWP 5318]
[New LWP 5319]
[New LWP 5320]
[New LWP 5321]
[New LWP 5322]
[New LWP 5323]
[New LWP 5324]
[New LWP 5325]
[New LWP 5326]
[New LWP 5327]
[New LWP 5328]
[New LWP 5329]
[New LWP 5330]
[New LWP 5331]
[New LWP 5332]
[New LWP 5333]
[New LWP 5334]
[New LWP 5335]
[New LWP 5336]
[New LWP 5337]
[New LWP 5338]
[New LWP 5339]
[New LWP 5340]
[New LWP 5341]
[New LWP 5342]
[New LWP 5343]
[New LWP 5344]
[New LWP 5346]
[New LWP 5347]
[New LWP 5348]
[New LWP 5349]
[New LWP 5350]
[New LWP 5351]
[New LWP 5352]
[New LWP 5353]
[New LWP 5354]
[New LWP 5355]
[New LWP 5356]
[New LWP 5357]
[New LWP 5358]
[New LWP 5359]
[New LWP 5360]
[New LWP 5361]
[New LWP 5362]
[New LWP 5363]
[New LWP 5364]
[New LWP 5365]
[New LWP 5366]
[New LWP 5367]
[New LWP 5368]
[New LWP 5369]
[New LWP 5370]
[New LWP 5371]
[New LWP 5372]
[New LWP 5373]
[New LWP 5374]
[New LWP 5375]
[New LWP 5376]
[New LWP 5377]
[New LWP 5378]
[New LWP 5379]
[New LWP 5380]
[New LWP 5381]
[New LWP 5382]
[New LWP 5383]
[New LWP 5384]
[New LWP 5385]
[New LWP 5386]
[New LWP 5387]
[New LWP 5388]
[New LWP 5389]
[New LWP 5390]
[New LWP 5391]
[New LWP 5392]
[New LWP 5393]
[New LWP 5394]
[New LWP 5395]
[New LWP 5396]
[New LWP 5397]
[New LWP 5398]
[New LWP 5399]
[New LWP 5400]
[New LWP 5402]
[New LWP 5403]
[New LWP 5404]
[New LWP 5405]
[New LWP 5406]
[New LWP 5407]
[New LWP 5408]
[New LWP 5409]
[New LWP 5410]
[New LWP 5411]
[New LWP 5412]
[New LWP 5413]
[New LWP 5414]
[New LWP 5415]
[New LWP 5416]
[New LWP 5417]
[New LWP 5418]
[New LWP 5419]
[New LWP 5420]
[New LWP 5423]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007f9ec12e8d50 in ?? ()
Id Target Id Frame
* 1 LWP 5289 "kudu" 0x00007f9ec12e8d50 in ?? ()
2 LWP 5290 "kudu" 0x00007f9ec12e4fb9 in ?? ()
3 LWP 5291 "kudu" 0x00007f9ec12e4fb9 in ?? ()
4 LWP 5292 "kudu" 0x00007f9ec12e4fb9 in ?? ()
5 LWP 5293 "kernel-watcher-" 0x00007f9ec12e4fb9 in ?? ()
6 LWP 5299 "ntp client-5299" 0x00007f9ec12e89e2 in ?? ()
7 LWP 5300 "file cache-evic" 0x00007f9ec12e4fb9 in ?? ()
8 LWP 5301 "sq_acceptor" 0x00007f9eba00acb9 in ?? ()
9 LWP 5306 "rpc reactor-530" 0x00007f9eba017a47 in ?? ()
10 LWP 5307 "rpc reactor-530" 0x00007f9eba017a47 in ?? ()
11 LWP 5308 "rpc reactor-530" 0x00007f9eba017a47 in ?? ()
12 LWP 5309 "rpc reactor-530" 0x00007f9eba017a47 in ?? ()
13 LWP 5310 "MaintenanceMgr " 0x00007f9ec12e4ad3 in ?? ()
14 LWP 5311 "txn-status-mana" 0x00007f9ec12e4fb9 in ?? ()
15 LWP 5312 "collect_and_rem" 0x00007f9ec12e4fb9 in ?? ()
16 LWP 5313 "tc-session-exp-" 0x00007f9ec12e4fb9 in ?? ()
17 LWP 5314 "rpc worker-5314" 0x00007f9ec12e4ad3 in ?? ()
18 LWP 5315 "rpc worker-5315" 0x00007f9ec12e4ad3 in ?? ()
19 LWP 5316 "rpc worker-5316" 0x00007f9ec12e4ad3 in ?? ()
20 LWP 5317 "rpc worker-5317" 0x00007f9ec12e4ad3 in ?? ()
21 LWP 5318 "rpc worker-5318" 0x00007f9ec12e4ad3 in ?? ()
22 LWP 5319 "rpc worker-5319" 0x00007f9ec12e4ad3 in ?? ()
23 LWP 5320 "rpc worker-5320" 0x00007f9ec12e4ad3 in ?? ()
24 LWP 5321 "rpc worker-5321" 0x00007f9ec12e4ad3 in ?? ()
25 LWP 5322 "rpc worker-5322" 0x00007f9ec12e4ad3 in ?? ()
26 LWP 5323 "rpc worker-5323" 0x00007f9ec12e4ad3 in ?? ()
27 LWP 5324 "rpc worker-5324" 0x00007f9ec12e4ad3 in ?? ()
28 LWP 5325 "rpc worker-5325" 0x00007f9ec12e4ad3 in ?? ()
29 LWP 5326 "rpc worker-5326" 0x00007f9ec12e4ad3 in ?? ()
30 LWP 5327 "rpc worker-5327" 0x00007f9ec12e4ad3 in ?? ()
31 LWP 5328 "rpc worker-5328" 0x00007f9ec12e4ad3 in ?? ()
32 LWP 5329 "rpc worker-5329" 0x00007f9ec12e4ad3 in ?? ()
33 LWP 5330 "rpc worker-5330" 0x00007f9ec12e4ad3 in ?? ()
34 LWP 5331 "rpc worker-5331" 0x00007f9ec12e4ad3 in ?? ()
35 LWP 5332 "rpc worker-5332" 0x00007f9ec12e4ad3 in ?? ()
36 LWP 5333 "rpc worker-5333" 0x00007f9ec12e4ad3 in ?? ()
37 LWP 5334 "rpc worker-5334" 0x00007f9ec12e4ad3 in ?? ()
38 LWP 5335 "rpc worker-5335" 0x00007f9ec12e4ad3 in ?? ()
39 LWP 5336 "rpc worker-5336" 0x00007f9ec12e4ad3 in ?? ()
40 LWP 5337 "rpc worker-5337" 0x00007f9ec12e4ad3 in ?? ()
41 LWP 5338 "rpc worker-5338" 0x00007f9ec12e4ad3 in ?? ()
42 LWP 5339 "rpc worker-5339" 0x00007f9ec12e4ad3 in ?? ()
43 LWP 5340 "rpc worker-5340" 0x00007f9ec12e4ad3 in ?? ()
44 LWP 5341 "rpc worker-5341" 0x00007f9ec12e4ad3 in ?? ()
45 LWP 5342 "rpc worker-5342" 0x00007f9ec12e4ad3 in ?? ()
46 LWP 5343 "rpc worker-5343" 0x00007f9ec12e4ad3 in ?? ()
47 LWP 5344 "rpc worker-5344" 0x00007f9ec12e4ad3 in ?? ()
48 LWP 5346 "rpc worker-5346" 0x00007f9ec12e4ad3 in ?? ()
49 LWP 5347 "rpc worker-5347" 0x00007f9ec12e4ad3 in ?? ()
50 LWP 5348 "rpc worker-5348" 0x00007f9ec12e4ad3 in ?? ()
51 LWP 5349 "rpc worker-5349" 0x00007f9ec12e4ad3 in ?? ()
52 LWP 5350 "rpc worker-5350" 0x00007f9ec12e4ad3 in ?? ()
53 LWP 5351 "rpc worker-5351" 0x00007f9ec12e4ad3 in ?? ()
54 LWP 5352 "rpc worker-5352" 0x00007f9ec12e4ad3 in ?? ()
55 LWP 5353 "rpc worker-5353" 0x00007f9ec12e4ad3 in ?? ()
56 LWP 5354 "rpc worker-5354" 0x00007f9ec12e4ad3 in ?? ()
57 LWP 5355 "rpc worker-5355" 0x00007f9ec12e4ad3 in ?? ()
58 LWP 5356 "rpc worker-5356" 0x00007f9ec12e4ad3 in ?? ()
59 LWP 5357 "rpc worker-5357" 0x00007f9ec12e4ad3 in ?? ()
60 LWP 5358 "rpc worker-5358" 0x00007f9ec12e4ad3 in ?? ()
61 LWP 5359 "rpc worker-5359" 0x00007f9ec12e4ad3 in ?? ()
62 LWP 5360 "rpc worker-5360" 0x00007f9ec12e4ad3 in ?? ()
63 LWP 5361 "rpc worker-5361" 0x00007f9ec12e4ad3 in ?? ()
64 LWP 5362 "rpc worker-5362" 0x00007f9ec12e4ad3 in ?? ()
65 LWP 5363 "rpc worker-5363" 0x00007f9ec12e4ad3 in ?? ()
66 LWP 5364 "rpc worker-5364" 0x00007f9ec12e4ad3 in ?? ()
67 LWP 5365 "rpc worker-5365" 0x00007f9ec12e4ad3 in ?? ()
68 LWP 5366 "rpc worker-5366" 0x00007f9ec12e4ad3 in ?? ()
69 LWP 5367 "rpc worker-5367" 0x00007f9ec12e4ad3 in ?? ()
70 LWP 5368 "rpc worker-5368" 0x00007f9ec12e4ad3 in ?? ()
71 LWP 5369 "rpc worker-5369" 0x00007f9ec12e4ad3 in ?? ()
72 LWP 5370 "rpc worker-5370" 0x00007f9ec12e4ad3 in ?? ()
73 LWP 5371 "rpc worker-5371" 0x00007f9ec12e4ad3 in ?? ()
74 LWP 5372 "rpc worker-5372" 0x00007f9ec12e4ad3 in ?? ()
75 LWP 5373 "rpc worker-5373" 0x00007f9ec12e4ad3 in ?? ()
76 LWP 5374 "rpc worker-5374" 0x00007f9ec12e4ad3 in ?? ()
77 LWP 5375 "rpc worker-5375" 0x00007f9ec12e4ad3 in ?? ()
78 LWP 5376 "rpc worker-5376" 0x00007f9ec12e4ad3 in ?? ()
79 LWP 5377 "rpc worker-5377" 0x00007f9ec12e4ad3 in ?? ()
80 LWP 5378 "rpc worker-5378" 0x00007f9ec12e4ad3 in ?? ()
81 LWP 5379 "rpc worker-5379" 0x00007f9ec12e4ad3 in ?? ()
82 LWP 5380 "rpc worker-5380" 0x00007f9ec12e4ad3 in ?? ()
83 LWP 5381 "rpc worker-5381" 0x00007f9ec12e4ad3 in ?? ()
84 LWP 5382 "rpc worker-5382" 0x00007f9ec12e4ad3 in ?? ()
85 LWP 5383 "rpc worker-5383" 0x00007f9ec12e4ad3 in ?? ()
86 LWP 5384 "rpc worker-5384" 0x00007f9ec12e4ad3 in ?? ()
87 LWP 5385 "rpc worker-5385" 0x00007f9ec12e4ad3 in ?? ()
88 LWP 5386 "rpc worker-5386" 0x00007f9ec12e4ad3 in ?? ()
89 LWP 5387 "rpc worker-5387" 0x00007f9ec12e4ad3 in ?? ()
90 LWP 5388 "rpc worker-5388" 0x00007f9ec12e4ad3 in ?? ()
91 LWP 5389 "rpc worker-5389" 0x00007f9ec12e4ad3 in ?? ()
92 LWP 5390 "rpc worker-5390" 0x00007f9ec12e4ad3 in ?? ()
93 LWP 5391 "rpc worker-5391" 0x00007f9ec12e4ad3 in ?? ()
94 LWP 5392 "rpc worker-5392" 0x00007f9ec12e4ad3 in ?? ()
95 LWP 5393 "rpc worker-5393" 0x00007f9ec12e4ad3 in ?? ()
96 LWP 5394 "rpc worker-5394" 0x00007f9ec12e4ad3 in ?? ()
97 LWP 5395 "rpc worker-5395" 0x00007f9ec12e4ad3 in ?? ()
98 LWP 5396 "rpc worker-5396" 0x00007f9ec12e4ad3 in ?? ()
99 LWP 5397 "rpc worker-5397" 0x00007f9ec12e4ad3 in ?? ()
100 LWP 5398 "rpc worker-5398" 0x00007f9ec12e4ad3 in ?? ()
101 LWP 5399 "rpc worker-5399" 0x00007f9ec12e4ad3 in ?? ()
102 LWP 5400 "rpc worker-5400" 0x00007f9ec12e4ad3 in ?? ()
103 LWP 5402 "rpc worker-5402" 0x00007f9ec12e4ad3 in ?? ()
104 LWP 5403 "rpc worker-5403" 0x00007f9ec12e4ad3 in ?? ()
105 LWP 5404 "rpc worker-5404" 0x00007f9ec12e4ad3 in ?? ()
106 LWP 5405 "rpc worker-5405" 0x00007f9ec12e4ad3 in ?? ()
107 LWP 5406 "rpc worker-5406" 0x00007f9ec12e4ad3 in ?? ()
108 LWP 5407 "rpc worker-5407" 0x00007f9ec12e4ad3 in ?? ()
109 LWP 5408 "rpc worker-5408" 0x00007f9ec12e4ad3 in ?? ()
110 LWP 5409 "rpc worker-5409" 0x00007f9ec12e4ad3 in ?? ()
111 LWP 5410 "rpc worker-5410" 0x00007f9ec12e4ad3 in ?? ()
112 LWP 5411 "rpc worker-5411" 0x00007f9ec12e4ad3 in ?? ()
113 LWP 5412 "rpc worker-5412" 0x00007f9ec12e4ad3 in ?? ()
114 LWP 5413 "rpc worker-5413" 0x00007f9ec12e4ad3 in ?? ()
115 LWP 5414 "rpc worker-5414" 0x00007f9ec12e4ad3 in ?? ()
116 LWP 5415 "rpc worker-5415" 0x00007f9ec12e4ad3 in ?? ()
117 LWP 5416 "diag-logger-541" 0x00007f9ec12e4fb9 in ?? ()
118 LWP 5417 "result-tracker-" 0x00007f9ec12e4fb9 in ?? ()
119 LWP 5418 "excess-log-dele" 0x00007f9ec12e4fb9 in ?? ()
120 LWP 5419 "acceptor-5419" 0x00007f9eba0190c7 in ?? ()
121 LWP 5420 "heartbeat-5420" 0x00007f9ec12e4fb9 in ?? ()
122 LWP 5423 "maintenance_sch" 0x00007f9ec12e4fb9 in ?? ()
Thread 122 (LWP 5423):
#0 0x00007f9ec12e4fb9 in ?? ()
#1 0x00007f9e7637d6e0 in ?? ()
#2 0x0000000000000134 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f9e7637d730 in ?? ()
#6 0x0000000000000268 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 5420):
#0 0x00007f9ec12e4fb9 in ?? ()
#1 0x26b5f4687947e954 in ?? ()
#2 0x000000000000004c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019e80 in ?? ()
#5 0x00007f9e773a45f0 in ?? ()
#6 0x0000000000000098 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 5419):
#0 0x00007f9eba0190c7 in ?? ()
#1 0x00006140000216a8 in ?? ()
#2 0x00007f9e77bcbd90 in ?? ()
#3 0x00007f9e77bcbdc0 in ?? ()
#4 0x00007f9e77bcbec0 in ?? ()
#5 0x00007f9e77bcbdb0 in ?? ()
#6 0x00007f9e77bcbe20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 5418):
#0 0x00007f9ec12e4fb9 in ?? ()
#1 0x00007f9e783e5f40 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff2fc22070 in ?? ()
#5 0x00007f9e783e5f30 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 5417):
#0 0x00007f9ec12e4fb9 in ?? ()
#1 0x00007f9e78bfe100 in ?? ()
#2 0x000000000000004d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f9e78bfe0f0 in ?? ()
#6 0x000000000000009a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 5416):
#0 0x00007f9ec12e4fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 5415):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007f9e79c2de90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f9e79c2deb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007f9ec12e4770 in ?? ()
#10 0x00007f9e79c2deb0 in ?? ()
#11 0x00000ff3cf385bd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 5414):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007f9e7a445e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f9e7a445eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007f9ec12e4770 in ?? ()
#10 0x00007f9e7a445eb0 in ?? ()
#11 0x00000ff3cf488bd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 5413):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 5412):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 5411):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 5410):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 5409):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 5408):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 5407):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 5406):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 5405):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 5404):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 5403):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 5402):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 5400):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 5399):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 5398):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 5397):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 5396):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 5395):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 5394):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 5393):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 5392):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 5391):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 5390):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 5389):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 5388):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 5387):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 5386):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 5385):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 5384):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 5383):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 5382):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 5381):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 5380):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 5379):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 5378):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 5377):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 5376):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 5375):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 5374):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a6888 in ?? ()
#4 0x00007f9e8dffee90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f9e8dffeeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 5373):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 5372):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 5371):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 5370):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 5369):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 5368):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 5367):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 5366):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 5365):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 5364):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 5363):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 5362):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 5361):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 5360):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 5359):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 5358):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 5357):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 5356):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 5355):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 5354):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 5353):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007f9e989f7e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f9e989f7eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 54 (LWP 5352):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 5351):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 5350):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 5349):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 5348):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 5347):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 5346):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 5344):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 5343):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 5342):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 5341):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 5340):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 5339):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 5338):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 5337):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 5336):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 5335):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 5334):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 5333):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00009ffe8 in ?? ()
#4 0x00007f9ea1ba4e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f9ea1ba4eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 5332):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 5331):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 5330):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 5329):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 5328):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 5327):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 5326):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 5325):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 5324):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 5323):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 5322):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 5321):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 5320):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 5319):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 5318):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 5317):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 5316):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 5315):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 5314):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 5313):
#0 0x00007f9ec12e4fb9 in ?? ()
#1 0x00007f9eabbc6cc0 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a720 in ?? ()
#5 0x00007f9eabbc6cb0 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 5312):
#0 0x00007f9ec12e4fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 5311):
#0 0x00007f9ec12e4fb9 in ?? ()
#1 0x00007f9eacbc8240 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007f9eacbc8230 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 5310):
#0 0x00007f9ec12e4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 5309):
#0 0x00007f9eba017a47 in ?? ()
#1 0x00007f9eadbcb320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f9eadbcb310 in ?? ()
#4 0x00007f9eadbcb520 in ?? ()
#5 0x00007f9eadbcb360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f9eadbcb3e0 in ?? ()
#8 0x00007f9ebc68a25d in ?? ()
#9 0x3fb95c1e3d5a4000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x000000001a800000 in ?? ()
#14 0x00007f9e00000000 in ?? ()
#15 0x41da17a869707643 in ?? ()
#16 0x00000ff455b7167c in ?? ()
#17 0x00007f9eadbcb3c0 in ?? ()
#18 0x00007f9ebc68eba3 in ?? ()
#19 0x00007f9eadbcb390 in ?? ()
#20 0x3fb95c1e3d5a4000 in ?? ()
#21 0x00000000adbcb3e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb95c1e3d5a4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 5308):
#0 0x00007f9eba017a47 in ?? ()
#1 0x00007f9eae3cc320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f9eae3cc310 in ?? ()
#4 0x00007f9eae3cc520 in ?? ()
#5 0x00007f9eae3cc360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f9eae3cc3e0 in ?? ()
#8 0x00007f9ebc68a25d in ?? ()
#9 0x3fb95819607bb000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x0000000036000000 in ?? ()
#14 0x00007f9e00000000 in ?? ()
#15 0x41da17a869707645 in ?? ()
#16 0x00000ff455c7187c in ?? ()
#17 0x00007f9eae3cc3c0 in ?? ()
#18 0x00007f9ebc68eba3 in ?? ()
#19 0x00007f9eae3cc390 in ?? ()
#20 0x3fb95819607bb000 in ?? ()
#21 0x00000000ae3cc3e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb95819607bb000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 5307):
#0 0x00007f9eba017a47 in ?? ()
#1 0x00007f9eaebcd320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f9eaebcd310 in ?? ()
#4 0x00007f9eaebcd520 in ?? ()
#5 0x00007f9eaebcd360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f9eaebcd3e0 in ?? ()
#8 0x00007f9ebc68a25d in ?? ()
#9 0x3fb9540d03c37000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x0000000027000000 in ?? ()
#14 0x00007f9e00000000 in ?? ()
#15 0x41da17a869707647 in ?? ()
#16 0x00000ff455d71a7c in ?? ()
#17 0x00007f9eaebcd3c0 in ?? ()
#18 0x00007f9ebc68eba3 in ?? ()
#19 0x00007f9eaebcd390 in ?? ()
#20 0x3fb9540d03c37000 in ?? ()
#21 0x00000000aebcd3e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb9540d03c37000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 5306):
#0 0x00007f9eba017a47 in ?? ()
#1 0x00007f9eb07b3320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f9eb07b3310 in ?? ()
#4 0x00007f9eb07b3520 in ?? ()
#5 0x00007f9eb07b3360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007f9eb07b33e0 in ?? ()
#8 0x00007f9ebc68a25d in ?? ()
#9 0x3fb97351cb265000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x0000000086800000 in ?? ()
#14 0x00007f9e00000000 in ?? ()
#15 0x41da17a869707647 in ?? ()
#16 0x00000ff4560ee67c in ?? ()
#17 0x00007f9eb07b33c0 in ?? ()
#18 0x00007f9ebc68eba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 5301):
#0 0x00007f9eba00acb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f9eb27b77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 5300):
#0 0x00007f9ec12e4fb9 in ?? ()
#1 0x00007f9eb1fb60c0 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001cf40 in ?? ()
#5 0x00007f9eb1fb60b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 5299):
#0 0x00007f9ec12e89e2 in ?? ()
#1 0x00007f9eb0fb2ba0 in ?? ()
#2 0x00007f9eb0fb2c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f9eb0fb2bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007f9eb0fb2c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f9eb0fb2c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007f9ec6bfbedc in ?? ()
#12 0x00007f9ec6beb8d9 in ?? ()
#13 0x00007f9ec6befec6 in ?? ()
#14 0x00007f9ec6bf4900 in ?? ()
#15 0x00007f9ec6bf4729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007f9ebdec04de in ?? ()
#18 0x00007f9ec12de6db in ?? ()
#19 0x00000ff4561ee484 in ?? ()
#20 0x00007f9eb0fb2440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 5293):
#0 0x00007f9ec12e4fb9 in ?? ()
#1 0x00007f9eb37b8c80 in ?? ()
#2 0x0000000000000188 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001e5a8 in ?? ()
#5 0x00007f9eb37b8c70 in ?? ()
#6 0x0000000000000310 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 5292):
#0 0x00007f9ec12e4fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 5291):
#0 0x00007f9ec12e4fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 5290):
#0 0x00007f9ec12e4fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000004 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f9eb4fbc450 in ?? ()
#6 0x0000000000000008 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 5289):
#0 0x00007f9ec12e8d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 13:57:54.440605 1586 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 8 with UUID 5f1fab906378491e919a4a3364723428 and pid 5142
************************ BEGIN STACKS **************************
[New LWP 5146]
[New LWP 5147]
[New LWP 5148]
[New LWP 5150]
[New LWP 5156]
[New LWP 5157]
[New LWP 5158]
[New LWP 5164]
[New LWP 5165]
[New LWP 5166]
[New LWP 5167]
[New LWP 5168]
[New LWP 5169]
[New LWP 5171]
[New LWP 5172]
[New LWP 5173]
[New LWP 5174]
[New LWP 5175]
[New LWP 5176]
[New LWP 5179]
[New LWP 5180]
[New LWP 5181]
[New LWP 5182]
[New LWP 5183]
[New LWP 5184]
[New LWP 5185]
[New LWP 5186]
[New LWP 5187]
[New LWP 5188]
[New LWP 5189]
[New LWP 5190]
[New LWP 5191]
[New LWP 5192]
[New LWP 5193]
[New LWP 5194]
[New LWP 5195]
[New LWP 5196]
[New LWP 5197]
[New LWP 5198]
[New LWP 5199]
[New LWP 5200]
[New LWP 5201]
[New LWP 5202]
[New LWP 5203]
[New LWP 5204]
[New LWP 5205]
[New LWP 5206]
[New LWP 5207]
[New LWP 5208]
[New LWP 5209]
[New LWP 5210]
[New LWP 5211]
[New LWP 5212]
[New LWP 5213]
[New LWP 5214]
[New LWP 5215]
[New LWP 5216]
[New LWP 5217]
[New LWP 5218]
[New LWP 5219]
[New LWP 5220]
[New LWP 5221]
[New LWP 5222]
[New LWP 5223]
[New LWP 5224]
[New LWP 5225]
[New LWP 5226]
[New LWP 5227]
[New LWP 5228]
[New LWP 5229]
[New LWP 5230]
[New LWP 5231]
[New LWP 5232]
[New LWP 5233]
[New LWP 5234]
[New LWP 5235]
[New LWP 5236]
[New LWP 5237]
[New LWP 5238]
[New LWP 5239]
[New LWP 5240]
[New LWP 5241]
[New LWP 5242]
[New LWP 5243]
[New LWP 5244]
[New LWP 5245]
[New LWP 5246]
[New LWP 5247]
[New LWP 5248]
[New LWP 5249]
[New LWP 5250]
[New LWP 5251]
[New LWP 5252]
[New LWP 5253]
[New LWP 5254]
[New LWP 5255]
[New LWP 5256]
[New LWP 5257]
[New LWP 5258]
[New LWP 5259]
[New LWP 5260]
[New LWP 5262]
[New LWP 5263]
[New LWP 5264]
[New LWP 5265]
[New LWP 5266]
[New LWP 5267]
[New LWP 5268]
[New LWP 5269]
[New LWP 5270]
[New LWP 5271]
[New LWP 5272]
[New LWP 5273]
[New LWP 5274]
[New LWP 5275]
[New LWP 5276]
[New LWP 5277]
[New LWP 5279]
[New LWP 5280]
[New LWP 5281]
[New LWP 5286]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007f61f8542d50 in ?? ()
Id Target Id Frame
* 1 LWP 5142 "kudu" 0x00007f61f8542d50 in ?? ()
2 LWP 5146 "kudu" 0x00007f61f853efb9 in ?? ()
3 LWP 5147 "kudu" 0x00007f61f853efb9 in ?? ()
4 LWP 5148 "kudu" 0x00007f61f853efb9 in ?? ()
5 LWP 5150 "kernel-watcher-" 0x00007f61f853efb9 in ?? ()
6 LWP 5156 "ntp client-5156" 0x00007f61f85429e2 in ?? ()
7 LWP 5157 "file cache-evic" 0x00007f61f853efb9 in ?? ()
8 LWP 5158 "sq_acceptor" 0x00007f61f1264cb9 in ?? ()
9 LWP 5164 "rpc reactor-516" 0x00007f61f1271a47 in ?? ()
10 LWP 5165 "rpc reactor-516" 0x00007f61f1271a47 in ?? ()
11 LWP 5166 "rpc reactor-516" 0x00007f61f1271a47 in ?? ()
12 LWP 5167 "rpc reactor-516" 0x00007f61f1271a47 in ?? ()
13 LWP 5168 "MaintenanceMgr " 0x00007f61f853ead3 in ?? ()
14 LWP 5169 "txn-status-mana" 0x00007f61f853efb9 in ?? ()
15 LWP 5171 "collect_and_rem" 0x00007f61f853efb9 in ?? ()
16 LWP 5172 "tc-session-exp-" 0x00007f61f853efb9 in ?? ()
17 LWP 5173 "rpc worker-5173" 0x00007f61f853ead3 in ?? ()
18 LWP 5174 "rpc worker-5174" 0x00007f61f853ead3 in ?? ()
19 LWP 5175 "rpc worker-5175" 0x00007f61f853ead3 in ?? ()
20 LWP 5176 "rpc worker-5176" 0x00007f61f853ead3 in ?? ()
21 LWP 5179 "rpc worker-5179" 0x00007f61f853ead3 in ?? ()
22 LWP 5180 "rpc worker-5180" 0x00007f61f853ead3 in ?? ()
23 LWP 5181 "rpc worker-5181" 0x00007f61f853ead3 in ?? ()
24 LWP 5182 "rpc worker-5182" 0x00007f61f853ead3 in ?? ()
25 LWP 5183 "rpc worker-5183" 0x00007f61f853ead3 in ?? ()
26 LWP 5184 "rpc worker-5184" 0x00007f61f853ead3 in ?? ()
27 LWP 5185 "rpc worker-5185" 0x00007f61f853ead3 in ?? ()
28 LWP 5186 "rpc worker-5186" 0x00007f61f853ead3 in ?? ()
29 LWP 5187 "rpc worker-5187" 0x00007f61f853ead3 in ?? ()
30 LWP 5188 "rpc worker-5188" 0x00007f61f853ead3 in ?? ()
31 LWP 5189 "rpc worker-5189" 0x00007f61f853ead3 in ?? ()
32 LWP 5190 "rpc worker-5190" 0x00007f61f853ead3 in ?? ()
33 LWP 5191 "rpc worker-5191" 0x00007f61f853ead3 in ?? ()
34 LWP 5192 "rpc worker-5192" 0x00007f61f853ead3 in ?? ()
35 LWP 5193 "rpc worker-5193" 0x00007f61f853ead3 in ?? ()
36 LWP 5194 "rpc worker-5194" 0x00007f61f853ead3 in ?? ()
37 LWP 5195 "rpc worker-5195" 0x00007f61f853ead3 in ?? ()
38 LWP 5196 "rpc worker-5196" 0x00007f61f853ead3 in ?? ()
39 LWP 5197 "rpc worker-5197" 0x00007f61f853ead3 in ?? ()
40 LWP 5198 "rpc worker-5198" 0x00007f61f853ead3 in ?? ()
41 LWP 5199 "rpc worker-5199" 0x00007f61f853ead3 in ?? ()
42 LWP 5200 "rpc worker-5200" 0x00007f61f853ead3 in ?? ()
43 LWP 5201 "rpc worker-5201" 0x00007f61f853ead3 in ?? ()
44 LWP 5202 "rpc worker-5202" 0x00007f61f853ead3 in ?? ()
45 LWP 5203 "rpc worker-5203" 0x00007f61f853ead3 in ?? ()
46 LWP 5204 "rpc worker-5204" 0x00007f61f853ead3 in ?? ()
47 LWP 5205 "rpc worker-5205" 0x00007f61f853ead3 in ?? ()
48 LWP 5206 "rpc worker-5206" 0x00007f61f853ead3 in ?? ()
49 LWP 5207 "rpc worker-5207" 0x00007f61f853ead3 in ?? ()
50 LWP 5208 "rpc worker-5208" 0x00007f61f853ead3 in ?? ()
51 LWP 5209 "rpc worker-5209" 0x00007f61f853ead3 in ?? ()
52 LWP 5210 "rpc worker-5210" 0x00007f61f853ead3 in ?? ()
53 LWP 5211 "rpc worker-5211" 0x00007f61f853ead3 in ?? ()
54 LWP 5212 "rpc worker-5212" 0x00007f61f853ead3 in ?? ()
55 LWP 5213 "rpc worker-5213" 0x00007f61f853ead3 in ?? ()
56 LWP 5214 "rpc worker-5214" 0x00007f61f853ead3 in ?? ()
57 LWP 5215 "rpc worker-5215" 0x00007f61f853ead3 in ?? ()
58 LWP 5216 "rpc worker-5216" 0x00007f61f853ead3 in ?? ()
59 LWP 5217 "rpc worker-5217" 0x00007f61f853ead3 in ?? ()
60 LWP 5218 "rpc worker-5218" 0x00007f61f853ead3 in ?? ()
61 LWP 5219 "rpc worker-5219" 0x00007f61f853ead3 in ?? ()
62 LWP 5220 "rpc worker-5220" 0x00007f61f853ead3 in ?? ()
63 LWP 5221 "rpc worker-5221" 0x00007f61f853ead3 in ?? ()
64 LWP 5222 "rpc worker-5222" 0x00007f61f853ead3 in ?? ()
65 LWP 5223 "rpc worker-5223" 0x00007f61f853ead3 in ?? ()
66 LWP 5224 "rpc worker-5224" 0x00007f61f853ead3 in ?? ()
67 LWP 5225 "rpc worker-5225" 0x00007f61f853ead3 in ?? ()
68 LWP 5226 "rpc worker-5226" 0x00007f61f853ead3 in ?? ()
69 LWP 5227 "rpc worker-5227" 0x00007f61f853ead3 in ?? ()
70 LWP 5228 "rpc worker-5228" 0x00007f61f853ead3 in ?? ()
71 LWP 5229 "rpc worker-5229" 0x00007f61f853ead3 in ?? ()
72 LWP 5230 "rpc worker-5230" 0x00007f61f853ead3 in ?? ()
73 LWP 5231 "rpc worker-5231" 0x00007f61f853ead3 in ?? ()
74 LWP 5232 "rpc worker-5232" 0x00007f61f853ead3 in ?? ()
75 LWP 5233 "rpc worker-5233" 0x00007f61f853ead3 in ?? ()
76 LWP 5234 "rpc worker-5234" 0x00007f61f853ead3 in ?? ()
77 LWP 5235 "rpc worker-5235" 0x00007f61f853ead3 in ?? ()
78 LWP 5236 "rpc worker-5236" 0x00007f61f853ead3 in ?? ()
79 LWP 5237 "rpc worker-5237" 0x00007f61f853ead3 in ?? ()
80 LWP 5238 "rpc worker-5238" 0x00007f61f853ead3 in ?? ()
81 LWP 5239 "rpc worker-5239" 0x00007f61f853ead3 in ?? ()
82 LWP 5240 "rpc worker-5240" 0x00007f61f853ead3 in ?? ()
83 LWP 5241 "rpc worker-5241" 0x00007f61f853ead3 in ?? ()
84 LWP 5242 "rpc worker-5242" 0x00007f61f853ead3 in ?? ()
85 LWP 5243 "rpc worker-5243" 0x00007f61f853ead3 in ?? ()
86 LWP 5244 "rpc worker-5244" 0x00007f61f853ead3 in ?? ()
87 LWP 5245 "rpc worker-5245" 0x00007f61f853ead3 in ?? ()
88 LWP 5246 "rpc worker-5246" 0x00007f61f853ead3 in ?? ()
89 LWP 5247 "rpc worker-5247" 0x00007f61f853ead3 in ?? ()
90 LWP 5248 "rpc worker-5248" 0x00007f61f853ead3 in ?? ()
91 LWP 5249 "rpc worker-5249" 0x00007f61f853ead3 in ?? ()
92 LWP 5250 "rpc worker-5250" 0x00007f61f853ead3 in ?? ()
93 LWP 5251 "rpc worker-5251" 0x00007f61f853ead3 in ?? ()
94 LWP 5252 "rpc worker-5252" 0x00007f61f853ead3 in ?? ()
95 LWP 5253 "rpc worker-5253" 0x00007f61f853ead3 in ?? ()
96 LWP 5254 "rpc worker-5254" 0x00007f61f853ead3 in ?? ()
97 LWP 5255 "rpc worker-5255" 0x00007f61f853ead3 in ?? ()
98 LWP 5256 "rpc worker-5256" 0x00007f61f853ead3 in ?? ()
99 LWP 5257 "rpc worker-5257" 0x00007f61f853ead3 in ?? ()
100 LWP 5258 "rpc worker-5258" 0x00007f61f853ead3 in ?? ()
101 LWP 5259 "rpc worker-5259" 0x00007f61f853ead3 in ?? ()
102 LWP 5260 "rpc worker-5260" 0x00007f61f853ead3 in ?? ()
103 LWP 5262 "rpc worker-5262" 0x00007f61f853ead3 in ?? ()
104 LWP 5263 "rpc worker-5263" 0x00007f61f853ead3 in ?? ()
105 LWP 5264 "rpc worker-5264" 0x00007f61f853ead3 in ?? ()
106 LWP 5265 "rpc worker-5265" 0x00007f61f853ead3 in ?? ()
107 LWP 5266 "rpc worker-5266" 0x00007f61f853ead3 in ?? ()
108 LWP 5267 "rpc worker-5267" 0x00007f61f853ead3 in ?? ()
109 LWP 5268 "rpc worker-5268" 0x00007f61f853ead3 in ?? ()
110 LWP 5269 "rpc worker-5269" 0x00007f61f853ead3 in ?? ()
111 LWP 5270 "rpc worker-5270" 0x00007f61f853ead3 in ?? ()
112 LWP 5271 "rpc worker-5271" 0x00007f61f853ead3 in ?? ()
113 LWP 5272 "rpc worker-5272" 0x00007f61f853ead3 in ?? ()
114 LWP 5273 "rpc worker-5273" 0x00007f61f853ead3 in ?? ()
115 LWP 5274 "rpc worker-5274" 0x00007f61f853ead3 in ?? ()
116 LWP 5275 "rpc worker-5275" 0x00007f61f853ead3 in ?? ()
117 LWP 5276 "diag-logger-527" 0x00007f61f853efb9 in ?? ()
118 LWP 5277 "result-tracker-" 0x00007f61f853efb9 in ?? ()
119 LWP 5279 "excess-log-dele" 0x00007f61f853efb9 in ?? ()
120 LWP 5280 "acceptor-5280" 0x00007f61f12730c7 in ?? ()
121 LWP 5281 "heartbeat-5281" 0x00007f61f853efb9 in ?? ()
122 LWP 5286 "maintenance_sch" 0x00007f61f853efb9 in ?? ()
Thread 122 (LWP 5286):
#0 0x00007f61f853efb9 in ?? ()
#1 0x00007f61ac4456e0 in ?? ()
#2 0x0000000000000141 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f61ac445730 in ?? ()
#6 0x0000000000000282 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 5281):
#0 0x00007f61f853efb9 in ?? ()
#1 0x26b5f4687947e954 in ?? ()
#2 0x0000000000000052 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x0000613000019e84 in ?? ()
#5 0x00007f61ad46c5f0 in ?? ()
#6 0x00000000000000a5 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f61ad46c610 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007f61ad46c690 in ?? ()
#12 0x00000fec35a8d8d4 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 120 (LWP 5280):
#0 0x00007f61f12730c7 in ?? ()
#1 0x0000614000040ea8 in ?? ()
#2 0x00007f61adc9cd90 in ?? ()
#3 0x00007f61adc9cdc0 in ?? ()
#4 0x00007f61adc9cec0 in ?? ()
#5 0x00007f61adc9cdb0 in ?? ()
#6 0x00007f61adc9ce20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 5279):
#0 0x00007f61f853efb9 in ?? ()
#1 0x00007f61ae4c5f40 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffce7b7a330 in ?? ()
#5 0x00007f61ae4c5f30 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 5277):
#0 0x00007f61f853efb9 in ?? ()
#1 0x00007f61af4ed100 in ?? ()
#2 0x0000000000000050 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f61af4ed0f0 in ?? ()
#6 0x00000000000000a0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 5276):
#0 0x00007f61f853efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 5275):
#0 0x00007f61f853ead3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b684c in ?? ()
#4 0x00007f61b051ce90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f61b051ceb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b6800 in ?? ()
#9 0x00007f61f853e770 in ?? ()
#10 0x00007f61b051ceb0 in ?? ()
#11 0x00000fec360a39d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 5274):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 5273):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007f61b165fe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f61b165feb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007f61f853e770 in ?? ()
#10 0x00007f61b165feb0 in ?? ()
#11 0x00000fec362cbfd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 113 (LWP 5272):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 5271):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 5270):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 5269):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 5268):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 5267):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 5266):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 5265):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 5264):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 5263):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 5262):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 5260):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 5259):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 5258):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 5257):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 5256):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 5255):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 5254):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 5253):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 5252):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 5251):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 5250):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 5249):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 5248):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 5247):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 5246):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 5245):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 5244):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 5243):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 5242):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 5241):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 5240):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 5239):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 5238):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 5237):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 5236):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 5235):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 5234):
#0 0x00007f61f853ead3 in ?? ()
#1 0x00000000000002df in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007f61c49f0e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f61c49f0eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007f61f853e770 in ?? ()
#10 0x00007f61c49f0eb0 in ?? ()
#11 0x00000fec3893e1d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 5233):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000005 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a688c in ?? ()
#4 0x00007f61c5208e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f61c5208eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a6840 in ?? ()
#9 0x00007f61f853e770 in ?? ()
#10 0x00007f61c5208eb0 in ?? ()
#11 0x00000fec38a411d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 74 (LWP 5232):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 5231):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 5230):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 5229):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 5228):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 5227):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 5226):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 5225):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 5224):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 5223):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 5222):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 5221):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 5220):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 5219):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 5218):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 5217):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 5216):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 5215):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 5214):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007f61cebd1e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f61cebd1eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 5213):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 5212):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 5211):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 5210):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 5209):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 5208):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 5207):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 5206):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 5205):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 5204):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 5203):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 5202):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 5201):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 5200):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 5199):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 5198):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 5197):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 5196):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 5195):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 5194):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000acfe8 in ?? ()
#4 0x00007f61d6d52e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f61d6d52eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 5193):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 5192):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 5191):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 5190):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 5189):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 5188):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 5187):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 5186):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 5185):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 5184):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 5183):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 5182):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 5181):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 5180):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 5179):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 5176):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 5175):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 5174):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 5173):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 5172):
#0 0x00007f61f853efb9 in ?? ()
#1 0x00007f61e0ec2cc0 in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ae20 in ?? ()
#5 0x00007f61e0ec2cb0 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 5171):
#0 0x00007f61f853efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 5169):
#0 0x00007f61f853efb9 in ?? ()
#1 0x00007f61e26c5240 in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007f61e26c5230 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 5168):
#0 0x00007f61f853ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 5167):
#0 0x00007f61f1271a47 in ?? ()
#1 0x00007f61e36c8320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f61e36c8310 in ?? ()
#4 0x00007f61e36c8520 in ?? ()
#5 0x00007f61e36c8360 in ?? ()
#6 0x0000614000041498 in ?? ()
#7 0x00007f61e36c83e0 in ?? ()
#8 0x00007f61f38e425d in ?? ()
#9 0x3fabb1115a99a000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x0000000085600000 in ?? ()
#14 0x00007f6100000000 in ?? ()
#15 0x41da17a869707647 in ?? ()
#16 0x00000fecbc6d107c in ?? ()
#17 0x00007f61e36c83c0 in ?? ()
#18 0x00007f61f38e8ba3 in ?? ()
#19 0x00007f61e36c8390 in ?? ()
#20 0x3fabb1115a99a000 in ?? ()
#21 0x00000000e36c83e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000041498 in ?? ()
#24 0x3fabb1115a99a000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 5166):
#0 0x00007f61f1271a47 in ?? ()
#1 0x00007f61e3ec9320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f61e3ec9310 in ?? ()
#4 0x00007f61e3ec9520 in ?? ()
#5 0x00007f61e3ec9360 in ?? ()
#6 0x0000614000041298 in ?? ()
#7 0x00007f61e3ec93e0 in ?? ()
#8 0x00007f61f38e425d in ?? ()
#9 0x3fb95650730d9000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x00000000ee000000 in ?? ()
#14 0x00007f6100000000 in ?? ()
#15 0x41da17a869707644 in ?? ()
#16 0x00000fecbc7d127c in ?? ()
#17 0x00007f61e3ec93c0 in ?? ()
#18 0x00007f61f38e8ba3 in ?? ()
#19 0x00007f61e3ec9390 in ?? ()
#20 0x3fb95650730d9000 in ?? ()
#21 0x00000000e3ec93e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000041298 in ?? ()
#24 0x3fb95650730d9000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 5165):
#0 0x00007f61f1271a47 in ?? ()
#1 0x00007f61e62b0320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f61e62b0310 in ?? ()
#4 0x00007f61e62b0520 in ?? ()
#5 0x00007f61e62b0360 in ?? ()
#6 0x0000614000041098 in ?? ()
#7 0x00007f61e62b03e0 in ?? ()
#8 0x00007f61f38e425d in ?? ()
#9 0x3fb983b01f120000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x00000000f3800000 in ?? ()
#14 0x00007f6100000000 in ?? ()
#15 0x41da17a869707647 in ?? ()
#16 0x00000fecbcc4e07c in ?? ()
#17 0x00007f61e62b03c0 in ?? ()
#18 0x00007f61f38e8ba3 in ?? ()
#19 0x00007f61e62b0390 in ?? ()
#20 0x3fb983b01f120000 in ?? ()
#21 0x00000000e62b03e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000041098 in ?? ()
#24 0x3fb983b01f120000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 5164):
#0 0x00007f61f1271a47 in ?? ()
#1 0x00007f61e46ca320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f61e46ca310 in ?? ()
#4 0x00007f61e46ca520 in ?? ()
#5 0x00007f61e46ca360 in ?? ()
#6 0x0000614000040e98 in ?? ()
#7 0x00007f61e46ca3e0 in ?? ()
#8 0x00007f61f38e425d in ?? ()
#9 0x3fb95af322ae0000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x0000000054000000 in ?? ()
#14 0x00007f6100000000 in ?? ()
#15 0x41da17a869707644 in ?? ()
#16 0x00000fecbc8d147c in ?? ()
#17 0x00007f61e46ca3c0 in ?? ()
#18 0x00007f61f38e8ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 5158):
#0 0x00007f61f1264cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f61e9ab77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 5157):
#0 0x00007f61f853efb9 in ?? ()
#1 0x00007f61e92b60c0 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001cf40 in ?? ()
#5 0x00007f61e92b60b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 5156):
#0 0x00007f61f85429e2 in ?? ()
#1 0x00007f61e82b2ba0 in ?? ()
#2 0x00007f61e82b2c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f61e82b2bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007f61e82b2c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f61e82b2c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007f61fde55edc in ?? ()
#12 0x00007f61fde458d9 in ?? ()
#13 0x00007f61fde49ec6 in ?? ()
#14 0x00007f61fde4e900 in ?? ()
#15 0x00007f61fde4e729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007f61f511a4de in ?? ()
#18 0x00007f61f85386db in ?? ()
#19 0x00000fecbd04e484 in ?? ()
#20 0x00007f61e82b2440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 5150):
#0 0x00007f61f853efb9 in ?? ()
#1 0x00007f61eaab8c80 in ?? ()
#2 0x000000000000019c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001e5a8 in ?? ()
#5 0x00007f61eaab8c70 in ?? ()
#6 0x0000000000000338 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 5148):
#0 0x00007f61f853efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 5147):
#0 0x00007f61f853efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 5146):
#0 0x00007f61f853efb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000004 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f61ec2bc450 in ?? ()
#6 0x0000000000000008 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 5142):
#0 0x00007f61f8542d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250627 13:57:55.142887 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 4008
I20250627 13:57:55.150231 4133 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:55.273057 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 4008
I20250627 13:57:55.296645 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 3719
I20250627 13:57:55.303119 3847 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:55.568248 6029 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0204b303a14a41b096bfddffc4d04290)
I20250627 13:57:55.568444 6029 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:57:55.569315 6029 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5f1fab906378491e919a4a3364723428 (127.1.140.137:40611), 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207)
I20250627 13:57:55.587008 5234 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" candidate_term: 5 candidate_status { last_received { term: 4 index: 16592 } } ignore_live_leader: false dest_uuid: "5f1fab906378491e919a4a3364723428" is_pre_election: true
I20250627 13:57:55.587985 6033 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0204b303a14a41b096bfddffc4d04290)
I20250627 13:57:55.588141 6033 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:57:55.588896 6033 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009), 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207)
I20250627 13:57:55.589313 4529 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "5f1fab906378491e919a4a3364723428" candidate_term: 5 candidate_status { last_received { term: 4 index: 16592 } } ignore_live_leader: false dest_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" is_pre_election: true
I20250627 13:57:55.589520 4529 raft_consensus.cc:2466] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f1fab906378491e919a4a3364723428 in term 4.
I20250627 13:57:55.589892 5167 leader_election.cc:304] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 373a9aec2682487a8ff2a4f0d1ee87ab, 5f1fab906378491e919a4a3364723428; no voters:
I20250627 13:57:55.590142 6033 raft_consensus.cc:2802] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250627 13:57:55.590248 6033 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0204b303a14a41b096bfddffc4d04290)
I20250627 13:57:55.590335 6033 raft_consensus.cc:3058] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 4 FOLLOWER]: Advancing to term 5
I20250627 13:57:55.594822 6033 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:57:55.595476 6033 leader_election.cc:290] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [CANDIDATE]: Term 5 election: Requested vote from peers 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009), 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207)
I20250627 13:57:55.595669 4529 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "5f1fab906378491e919a4a3364723428" candidate_term: 5 candidate_status { last_received { term: 4 index: 16592 } } ignore_live_leader: false dest_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab"
I20250627 13:57:55.595821 4529 raft_consensus.cc:3058] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 4 FOLLOWER]: Advancing to term 5
I20250627 13:57:55.599336 4529 raft_consensus.cc:2466] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f1fab906378491e919a4a3364723428 in term 5.
I20250627 13:57:55.599635 5167 leader_election.cc:304] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 373a9aec2682487a8ff2a4f0d1ee87ab, 5f1fab906378491e919a4a3364723428; no voters:
I20250627 13:57:55.599872 6033 raft_consensus.cc:2802] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 5 FOLLOWER]: Leader election won for term 5
I20250627 13:57:55.600219 6033 raft_consensus.cc:695] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 5 LEADER]: Becoming Leader. State: Replica: 5f1fab906378491e919a4a3364723428, State: Running, Role: LEADER
I20250627 13:57:55.600642 6033 consensus_queue.cc:237] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16592, Committed index: 16592, Last appended: 4.16592, Last appended by leader: 16592, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:57:55.604357 1621 catalog_manager.cc:5582] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 reported cstate change: term changed from 4 to 5, leader changed from 0204b303a14a41b096bfddffc4d04290 (127.1.140.130) to 5f1fab906378491e919a4a3364723428 (127.1.140.137). New cstate: current_term: 5 leader_uuid: "5f1fab906378491e919a4a3364723428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } health_report { overall_health: UNKNOWN } } }
I20250627 13:57:55.670285 3805 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "5f1fab906378491e919a4a3364723428" candidate_term: 5 candidate_status { last_received { term: 4 index: 16592 } } ignore_live_leader: false dest_uuid: "0204b303a14a41b096bfddffc4d04290"
I20250627 13:57:55.671562 3806 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8dca13501f184a25a621c700a1939986" candidate_uuid: "5f1fab906378491e919a4a3364723428" candidate_term: 5 candidate_status { last_received { term: 4 index: 16592 } } ignore_live_leader: false dest_uuid: "0204b303a14a41b096bfddffc4d04290" is_pre_election: true
I20250627 13:57:55.672299 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 3719
I20250627 13:57:55.672405 5234 raft_consensus.cc:1238] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [term 5 LEADER]: Rejecting Update request from peer 0204b303a14a41b096bfddffc4d04290 for earlier term 4. Current term is 5. Ops: []
I20250627 13:57:55.716961 4529 raft_consensus.cc:1273] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 5 FOLLOWER]: Refusing update from remote peer 5f1fab906378491e919a4a3364723428: Log matching property violated. Preceding OpId in replica: term: 4 index: 16592. Preceding OpId from leader: term: 5 index: 16593. (index mismatch)
I20250627 13:57:55.717605 6033 consensus_queue.cc:1035] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16593, Last known committed idx: 16592, Time since last communication: 0.000s
W20250627 13:57:55.729442 5166 connection.cc:537] client connection to 127.1.140.130:42207 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:57:55.729672 5166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:57:55.729681 5167 connection.cc:537] server connection from 127.1.140.130:37925 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:57:55.730214 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 3863
W20250627 13:57:55.730456 6030 negotiation.cc:337] Failed RPC negotiation. Trace:
0627 13:57:55.569544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.140.130:42207 (local address 127.1.140.135:32841)
0627 13:57:55.581923 (+ 12379us) negotiation.cc:107] Waiting for socket to connect
0627 13:57:55.581938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0627 13:57:55.582029 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0627 13:57:55.729546 (+147517us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.140.130:42207: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":12279,"thread_start_us":11780,"threads_started":1}
W20250627 13:57:55.730731 4460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.140.130:42207: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:57:55.732594 4460 leader_election.cc:336] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Network error: Client connection negotiation failed: client connection to 127.1.140.130:42207: connect: Connection refused (error 111)
W20250627 13:57:55.732731 5166 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.130:42207: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:57:55.732743 4460 leader_election.cc:304] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 373a9aec2682487a8ff2a4f0d1ee87ab; no voters: 0204b303a14a41b096bfddffc4d04290, 5f1fab906378491e919a4a3364723428
I20250627 13:57:55.732998 6029 raft_consensus.cc:2747] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250627 13:57:55.737865 3991 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:55.855321 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 3863
I20250627 13:57:55.878365 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 3314
I20250627 13:57:55.884490 3441 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:56.001255 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 3314
I20250627 13:57:56.024214 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 3171
I20250627 13:57:56.030833 3301 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:56.135213 4581 maintenance_manager.cc:419] P 373a9aec2682487a8ff2a4f0d1ee87ab: Scheduling FlushMRSOp(8dca13501f184a25a621c700a1939986): perf score=0.084259
W20250627 13:57:56.146828 5166 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.130:42207: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250627 13:57:56.153869 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 3171
I20250627 13:57:56.187669 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 3029
I20250627 13:57:56.196982 3158 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:56.322887 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 3029
I20250627 13:57:56.357071 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 4441
I20250627 13:57:56.367774 4571 generic_service.cc:196] Checking for leaks (request via RPC)
W20250627 13:57:56.687258 5166 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 -> Peer 0204b303a14a41b096bfddffc4d04290 (127.1.140.130:42207): Couldn't send request to peer 0204b303a14a41b096bfddffc4d04290. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.130:42207: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250627 13:57:56.822106 6057 raft_consensus.cc:491] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5f1fab906378491e919a4a3364723428)
I20250627 13:57:56.822358 6057 raft_consensus.cc:513] T 8dca13501f184a25a621c700a1939986 P 373a9aec2682487a8ff2a4f0d1ee87ab [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1fab906378491e919a4a3364723428" member_type: VOTER last_known_addr { host: "127.1.140.137" port: 40611 } } peers { permanent_uuid: "373a9aec2682487a8ff2a4f0d1ee87ab" member_type: VOTER last_known_addr { host: "127.1.140.135" port: 34009 } } peers { permanent_uuid: "0204b303a14a41b096bfddffc4d04290" member_type: VOTER last_known_addr { host: "127.1.140.130" port: 42207 } }
I20250627 13:57:56.822638 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 4441
W20250627 13:57:56.884290 5167 connection.cc:537] client connection to 127.1.140.135:34009 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250627 13:57:56.884500 1610 connection.cc:537] server connection from 127.1.140.135:34237 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250627 13:57:56.884720 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 5289
W20250627 13:57:56.885449 5167 consensus_peers.cc:487] T 8dca13501f184a25a621c700a1939986 P 5f1fab906378491e919a4a3364723428 -> Peer 373a9aec2682487a8ff2a4f0d1ee87ab (127.1.140.135:34009): Couldn't send request to peer 373a9aec2682487a8ff2a4f0d1ee87ab. Status: Network error: Client connection negotiation failed: client connection to 127.1.140.135:34009: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250627 13:57:56.891430 5414 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:56.988304 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 5289
I20250627 13:57:57.010414 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 5142
I20250627 13:57:57.016180 5273 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:57.434132 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 5142
I20250627 13:57:57.492813 1586 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu pid 1594
I20250627 13:57:57.499189 1655 generic_service.cc:196] Checking for leaks (request via RPC)
I20250627 13:57:57.677265 1586 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskbElLHZ/build/asan/bin/kudu with pid 1594
2025-06-27T13:57:57Z chronyd exiting
I20250627 13:57:57.726533 1586 test_util.cc:183] -----------------------------------------------
I20250627 13:57:57.726616 1586 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskbElLHZ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1751032457422240-1586-0
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3, where GetParam() = 800-byte object <01-00 00-00 01-61 00-00 09-00 00-00 BE-BE BE-BE A0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-1F 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 03-00 00-00 03-00 00-00 00-BE BE-BE BE-BE BE-BE> (220246 ms)
[----------] 1 test from RollingRestartArgs/RollingRestartITest (220246 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (220246 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3, where GetParam() = 800-byte object <01-00 00-00 01-61 00-00 09-00 00-00 BE-BE BE-BE A0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-1F 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 03-00 00-00 03-00 00-00 00-BE BE-BE BE-BE BE-BE>
1 FAILED TEST