Note: This is test shard 8 of 8.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RollingRestartArgs/RollingRestartITest
[ RUN ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3
2025-01-16T02:54:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-16T02:54:26Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250116 02:54:26.846482 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.228.126:43731
--webserver_interface=127.7.228.126
--webserver_port=0
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.7.228.126:43731
--location_mapping_cmd=/tmp/dist-test-task8SxoCU/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/location-assignment.state --map /L0:3 --map /L1:4 --map /L2:4
--master_client_location_assignment_enabled=false with env {}
W20250116 02:54:27.133515 8089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:54:27.133790 8089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:54:27.133882 8089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:54:27.140575 8089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250116 02:54:27.140668 8089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:54:27.140718 8089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250116 02:54:27.140767 8089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250116 02:54:27.148903 8089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/wal
--location_mapping_cmd=/tmp/dist-test-task8SxoCU/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/location-assignment.state --map /L0:3 --map /L1:4 --map /L2:4
--ipki_ca_key_size=768
--master_addresses=127.7.228.126:43731
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.7.228.126:43731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.7.228.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:54:27.150287 8089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:54:27.151786 8089 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:54:27.160998 8095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:27.161592 8097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:27.162644 8094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:54:27.163676 8089 server_base.cc:1034] running on GCE node
I20250116 02:54:27.164283 8089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:54:27.165335 8089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:54:27.166523 8089 hybrid_clock.cc:648] HybridClock initialized: now 1736996067166454 us; error 46 us; skew 500 ppm
I20250116 02:54:27.166797 8089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:54:27.172371 8089 webserver.cc:458] Webserver started at http://127.7.228.126:42985/ using document root <none> and password file <none>
I20250116 02:54:27.172966 8089 fs_manager.cc:362] Metadata directory not provided
I20250116 02:54:27.173058 8089 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:54:27.173295 8089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:54:27.175606 8089 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/data/instance:
uuid: "56ddececa4d0424dae4d75d6a003686b"
format_stamp: "Formatted at 2025-01-16 02:54:27 on dist-test-slave-ldbs"
I20250116 02:54:27.176306 8089 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/wal/instance:
uuid: "56ddececa4d0424dae4d75d6a003686b"
format_stamp: "Formatted at 2025-01-16 02:54:27 on dist-test-slave-ldbs"
I20250116 02:54:27.180815 8089 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250116 02:54:27.183646 8103 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:27.184796 8089 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250116 02:54:27.184937 8089 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/wal
uuid: "56ddececa4d0424dae4d75d6a003686b"
format_stamp: "Formatted at 2025-01-16 02:54:27 on dist-test-slave-ldbs"
I20250116 02:54:27.185070 8089 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:54:27.208303 8089 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:54:27.208943 8089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:54:27.209182 8089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:54:27.225886 8089 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.126:43731
I20250116 02:54:27.225910 8154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.126:43731 every 8 connection(s)
I20250116 02:54:27.227293 8089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/data/info.pb
I20250116 02:54:27.228695 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 8089
I20250116 02:54:27.228938 8081 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/wal/instance
I20250116 02:54:27.231595 8155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:54:27.240597 8155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b: Bootstrap starting.
I20250116 02:54:27.243517 8155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b: Neither blocks nor log segments found. Creating new log.
I20250116 02:54:27.244448 8155 log.cc:826] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b: Log is configured to *not* fsync() on all Append() calls
I20250116 02:54:27.246876 8155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b: No bootstrap required, opened a new log
I20250116 02:54:27.254151 8155 raft_consensus.cc:357] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ddececa4d0424dae4d75d6a003686b" member_type: VOTER last_known_addr { host: "127.7.228.126" port: 43731 } }
I20250116 02:54:27.254464 8155 raft_consensus.cc:383] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:54:27.254554 8155 raft_consensus.cc:738] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56ddececa4d0424dae4d75d6a003686b, State: Initialized, Role: FOLLOWER
I20250116 02:54:27.255077 8155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ddececa4d0424dae4d75d6a003686b" member_type: VOTER last_known_addr { host: "127.7.228.126" port: 43731 } }
I20250116 02:54:27.255275 8155 raft_consensus.cc:397] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250116 02:54:27.255385 8155 raft_consensus.cc:491] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250116 02:54:27.255519 8155 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:54:27.257226 8155 raft_consensus.cc:513] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ddececa4d0424dae4d75d6a003686b" member_type: VOTER last_known_addr { host: "127.7.228.126" port: 43731 } }
I20250116 02:54:27.257676 8155 leader_election.cc:304] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56ddececa4d0424dae4d75d6a003686b; no voters:
I20250116 02:54:27.258168 8155 leader_election.cc:290] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250116 02:54:27.258328 8160 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:54:27.259305 8160 raft_consensus.cc:695] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [term 1 LEADER]: Becoming Leader. State: Replica: 56ddececa4d0424dae4d75d6a003686b, State: Running, Role: LEADER
I20250116 02:54:27.259812 8160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ddececa4d0424dae4d75d6a003686b" member_type: VOTER last_known_addr { host: "127.7.228.126" port: 43731 } }
I20250116 02:54:27.259979 8155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [sys.catalog]: configured and running, proceeding with master startup.
I20250116 02:54:27.262446 8162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56ddececa4d0424dae4d75d6a003686b. Latest consensus state: current_term: 1 leader_uuid: "56ddececa4d0424dae4d75d6a003686b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ddececa4d0424dae4d75d6a003686b" member_type: VOTER last_known_addr { host: "127.7.228.126" port: 43731 } } }
I20250116 02:54:27.262720 8162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [sys.catalog]: This master's current role is: LEADER
I20250116 02:54:27.263234 8161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56ddececa4d0424dae4d75d6a003686b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56ddececa4d0424dae4d75d6a003686b" member_type: VOTER last_known_addr { host: "127.7.228.126" port: 43731 } } }
I20250116 02:54:27.263535 8161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b [sys.catalog]: This master's current role is: LEADER
I20250116 02:54:27.264835 8169 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250116 02:54:27.268899 8169 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250116 02:54:27.275942 8169 catalog_manager.cc:1348] Generated new cluster ID: bdbc049ea87847fdac6c563bf0cd9aa4
I20250116 02:54:27.276036 8169 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250116 02:54:27.285053 8169 catalog_manager.cc:1371] Generated new certificate authority record
I20250116 02:54:27.286044 8169 catalog_manager.cc:1505] Loading token signing keys...
I20250116 02:54:27.294752 8169 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 56ddececa4d0424dae4d75d6a003686b: Generated new TSK 0
I20250116 02:54:27.295333 8169 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250116 02:54:27.303617 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.65:0
--local_ip_for_outbound_sockets=127.7.228.65
--webserver_interface=127.7.228.65
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250116 02:54:27.596345 8179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:54:27.596602 8179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:54:27.596688 8179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:54:27.603868 8179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:54:27.604029 8179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.65
I20250116 02:54:27.612555 8179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.228.65
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.65
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:54:27.613852 8179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:54:27.615324 8179 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:54:27.625304 8184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:27.625806 8185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:27.626780 8187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:54:27.627120 8179 server_base.cc:1034] running on GCE node
I20250116 02:54:27.627633 8179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:54:27.628439 8179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:54:27.629616 8179 hybrid_clock.cc:648] HybridClock initialized: now 1736996067629553 us; error 43 us; skew 500 ppm
I20250116 02:54:27.629894 8179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:54:27.631815 8179 webserver.cc:458] Webserver started at http://127.7.228.65:45657/ using document root <none> and password file <none>
I20250116 02:54:27.632429 8179 fs_manager.cc:362] Metadata directory not provided
I20250116 02:54:27.632539 8179 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:54:27.632792 8179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:54:27.635021 8179 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data/instance:
uuid: "1361eef60904481394d5230d1bc84711"
format_stamp: "Formatted at 2025-01-16 02:54:27 on dist-test-slave-ldbs"
I20250116 02:54:27.635620 8179 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/wal/instance:
uuid: "1361eef60904481394d5230d1bc84711"
format_stamp: "Formatted at 2025-01-16 02:54:27 on dist-test-slave-ldbs"
I20250116 02:54:27.639918 8179 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250116 02:54:27.642742 8193 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:27.643759 8179 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250116 02:54:27.643887 8179 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/wal
uuid: "1361eef60904481394d5230d1bc84711"
format_stamp: "Formatted at 2025-01-16 02:54:27 on dist-test-slave-ldbs"
I20250116 02:54:27.644053 8179 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:54:27.663964 8179 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:54:27.664608 8179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:54:27.664845 8179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:54:27.665642 8179 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:54:27.667047 8179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:54:27.667124 8179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:27.667243 8179 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:54:27.667300 8179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:27.696169 8179 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.65:40373
I20250116 02:54:27.696204 8305 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.65:40373 every 8 connection(s)
I20250116 02:54:27.697310 8179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data/info.pb
I20250116 02:54:27.706988 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 8179
I20250116 02:54:27.707175 8081 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/wal/instance
I20250116 02:54:27.710777 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.66:0
--local_ip_for_outbound_sockets=127.7.228.66
--webserver_interface=127.7.228.66
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250116 02:54:27.710835 8306 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:54:27.711136 8306 heartbeater.cc:463] Registering TS with master...
I20250116 02:54:27.711849 8306 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:27.764993 8120 ts_manager.cc:194] Registered new tserver with Master: 1361eef60904481394d5230d1bc84711 (127.7.228.65:40373)
I20250116 02:54:27.766746 8120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.65:53789
W20250116 02:54:28.013572 8309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:54:28.013818 8309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:54:28.013944 8309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:54:28.020785 8309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:54:28.021013 8309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.66
I20250116 02:54:28.028954 8309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.228.66
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.66
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:54:28.030375 8309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:54:28.031873 8309 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:54:28.042014 8315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:28.043038 8316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:28.043666 8318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:54:28.043931 8309 server_base.cc:1034] running on GCE node
I20250116 02:54:28.044368 8309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:54:28.045070 8309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:54:28.046278 8309 hybrid_clock.cc:648] HybridClock initialized: now 1736996068046222 us; error 31 us; skew 500 ppm
I20250116 02:54:28.046550 8309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:54:28.048424 8309 webserver.cc:458] Webserver started at http://127.7.228.66:46195/ using document root <none> and password file <none>
I20250116 02:54:28.049096 8309 fs_manager.cc:362] Metadata directory not provided
I20250116 02:54:28.049206 8309 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:54:28.049453 8309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:54:28.051787 8309 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data/instance:
uuid: "429bf4133bf54797b117ef8fad632920"
format_stamp: "Formatted at 2025-01-16 02:54:28 on dist-test-slave-ldbs"
I20250116 02:54:28.052430 8309 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal/instance:
uuid: "429bf4133bf54797b117ef8fad632920"
format_stamp: "Formatted at 2025-01-16 02:54:28 on dist-test-slave-ldbs"
I20250116 02:54:28.056742 8309 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250116 02:54:28.059448 8324 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:28.060518 8309 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250116 02:54:28.060664 8309 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal
uuid: "429bf4133bf54797b117ef8fad632920"
format_stamp: "Formatted at 2025-01-16 02:54:28 on dist-test-slave-ldbs"
I20250116 02:54:28.060832 8309 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:54:28.080775 8309 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:54:28.081454 8309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:54:28.081733 8309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:54:28.082544 8309 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:54:28.083717 8309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:54:28.083786 8309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:28.083899 8309 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:54:28.083947 8309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:28.110994 8309 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.66:41805
I20250116 02:54:28.111033 8436 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.66:41805 every 8 connection(s)
I20250116 02:54:28.112043 8309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data/info.pb
I20250116 02:54:28.117676 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 8309
I20250116 02:54:28.117841 8081 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal/instance
I20250116 02:54:28.121014 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.67:0
--local_ip_for_outbound_sockets=127.7.228.67
--webserver_interface=127.7.228.67
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250116 02:54:28.123798 8437 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:54:28.124063 8437 heartbeater.cc:463] Registering TS with master...
I20250116 02:54:28.124697 8437 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:28.174886 8120 ts_manager.cc:194] Registered new tserver with Master: 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805)
I20250116 02:54:28.175726 8120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.66:48123
W20250116 02:54:28.420373 8440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:54:28.420639 8440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:54:28.420786 8440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:54:28.427498 8440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:54:28.427708 8440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.67
I20250116 02:54:28.435840 8440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.228.67
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.67
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:54:28.437192 8440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:54:28.438800 8440 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:54:28.449702 8447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:28.449851 8446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:28.450579 8449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:54:28.451576 8440 server_base.cc:1034] running on GCE node
I20250116 02:54:28.452057 8440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:54:28.452770 8440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:54:28.453959 8440 hybrid_clock.cc:648] HybridClock initialized: now 1736996068453913 us; error 36 us; skew 500 ppm
I20250116 02:54:28.454233 8440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:54:28.456125 8440 webserver.cc:458] Webserver started at http://127.7.228.67:44901/ using document root <none> and password file <none>
I20250116 02:54:28.456722 8440 fs_manager.cc:362] Metadata directory not provided
I20250116 02:54:28.456815 8440 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:54:28.457095 8440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:54:28.459309 8440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data/instance:
uuid: "660967712270485c81d6a5f1239c89b1"
format_stamp: "Formatted at 2025-01-16 02:54:28 on dist-test-slave-ldbs"
I20250116 02:54:28.461830 8440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/wal/instance:
uuid: "660967712270485c81d6a5f1239c89b1"
format_stamp: "Formatted at 2025-01-16 02:54:28 on dist-test-slave-ldbs"
I20250116 02:54:28.466399 8440 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250116 02:54:28.469280 8455 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:28.470412 8440 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250116 02:54:28.470549 8440 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/wal
uuid: "660967712270485c81d6a5f1239c89b1"
format_stamp: "Formatted at 2025-01-16 02:54:28 on dist-test-slave-ldbs"
I20250116 02:54:28.470719 8440 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:54:28.488054 8440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:54:28.488667 8440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:54:28.488896 8440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:54:28.489758 8440 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:54:28.491073 8440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:54:28.491158 8440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:28.491287 8440 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:54:28.491333 8440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:28.518716 8440 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.67:33499
I20250116 02:54:28.518772 8567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.67:33499 every 8 connection(s)
I20250116 02:54:28.519850 8440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data/info.pb
I20250116 02:54:28.524914 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 8440
I20250116 02:54:28.525099 8081 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/wal/instance
I20250116 02:54:28.528056 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.68:0
--local_ip_for_outbound_sockets=127.7.228.68
--webserver_interface=127.7.228.68
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250116 02:54:28.531880 8568 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:54:28.532160 8568 heartbeater.cc:463] Registering TS with master...
I20250116 02:54:28.532815 8568 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:28.572413 8120 ts_manager.cc:194] Registered new tserver with Master: 660967712270485c81d6a5f1239c89b1 (127.7.228.67:33499)
I20250116 02:54:28.573324 8120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.67:50615
I20250116 02:54:28.769874 8306 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
W20250116 02:54:28.815339 8571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:54:28.815562 8571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:54:28.815680 8571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:54:28.822465 8571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:54:28.822659 8571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.68
I20250116 02:54:28.830477 8571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.7.228.68
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.68
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:54:28.831812 8571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:54:28.833205 8571 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:54:28.843367 8577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:28.844383 8578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:28.844872 8580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:54:28.845753 8571 server_base.cc:1034] running on GCE node
I20250116 02:54:28.846217 8571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:54:28.846921 8571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:54:28.848109 8571 hybrid_clock.cc:648] HybridClock initialized: now 1736996068848061 us; error 28 us; skew 500 ppm
I20250116 02:54:28.848337 8571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:54:28.850271 8571 webserver.cc:458] Webserver started at http://127.7.228.68:44091/ using document root <none> and password file <none>
I20250116 02:54:28.850900 8571 fs_manager.cc:362] Metadata directory not provided
I20250116 02:54:28.851017 8571 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:54:28.851306 8571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:54:28.853444 8571 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data/instance:
uuid: "0791364971ba49f3b2813bdc36c11eda"
format_stamp: "Formatted at 2025-01-16 02:54:28 on dist-test-slave-ldbs"
I20250116 02:54:28.854122 8571 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/wal/instance:
uuid: "0791364971ba49f3b2813bdc36c11eda"
format_stamp: "Formatted at 2025-01-16 02:54:28 on dist-test-slave-ldbs"
I20250116 02:54:28.858345 8571 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250116 02:54:28.861069 8586 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:28.862231 8571 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250116 02:54:28.862371 8571 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/wal
uuid: "0791364971ba49f3b2813bdc36c11eda"
format_stamp: "Formatted at 2025-01-16 02:54:28 on dist-test-slave-ldbs"
I20250116 02:54:28.862531 8571 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:54:28.881879 8571 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:54:28.882565 8571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:54:28.882795 8571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:54:28.883610 8571 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:54:28.884862 8571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:54:28.884933 8571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:28.885071 8571 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:54:28.885124 8571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:28.913338 8571 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.68:41593
I20250116 02:54:28.913383 8698 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.68:41593 every 8 connection(s)
I20250116 02:54:28.914399 8571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data/info.pb
I20250116 02:54:28.922616 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 8571
I20250116 02:54:28.922745 8081 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/wal/instance
I20250116 02:54:28.925568 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.69:0
--local_ip_for_outbound_sockets=127.7.228.69
--webserver_interface=127.7.228.69
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250116 02:54:28.925799 8699 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:54:28.926036 8699 heartbeater.cc:463] Registering TS with master...
I20250116 02:54:28.926677 8699 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:28.964265 8120 ts_manager.cc:194] Registered new tserver with Master: 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:54:28.965271 8120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.68:39551
I20250116 02:54:29.178004 8437 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
W20250116 02:54:29.215764 8702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:54:29.216012 8702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:54:29.216145 8702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:54:29.222754 8702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:54:29.222935 8702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.69
I20250116 02:54:29.230751 8702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data/info.pb
--webserver_interface=127.7.228.69
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.69
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:54:29.232074 8702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:54:29.233492 8702 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:54:29.244356 8708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:29.245076 8711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:29.246186 8709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:54:29.246466 8702 server_base.cc:1034] running on GCE node
I20250116 02:54:29.246901 8702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:54:29.247645 8702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:54:29.248842 8702 hybrid_clock.cc:648] HybridClock initialized: now 1736996069248789 us; error 27 us; skew 500 ppm
I20250116 02:54:29.249095 8702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:54:29.251099 8702 webserver.cc:458] Webserver started at http://127.7.228.69:40473/ using document root <none> and password file <none>
I20250116 02:54:29.251749 8702 fs_manager.cc:362] Metadata directory not provided
I20250116 02:54:29.251849 8702 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:54:29.252127 8702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:54:29.254304 8702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data/instance:
uuid: "ff972e037b0e40bba36f7896effdb71f"
format_stamp: "Formatted at 2025-01-16 02:54:29 on dist-test-slave-ldbs"
I20250116 02:54:29.254940 8702 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/wal/instance:
uuid: "ff972e037b0e40bba36f7896effdb71f"
format_stamp: "Formatted at 2025-01-16 02:54:29 on dist-test-slave-ldbs"
I20250116 02:54:29.259172 8702 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250116 02:54:29.261920 8717 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:29.263016 8702 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250116 02:54:29.263151 8702 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/wal
uuid: "ff972e037b0e40bba36f7896effdb71f"
format_stamp: "Formatted at 2025-01-16 02:54:29 on dist-test-slave-ldbs"
I20250116 02:54:29.263295 8702 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:54:29.279207 8702 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:54:29.279826 8702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:54:29.280064 8702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:54:29.280867 8702 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:54:29.282167 8702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:54:29.282243 8702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:29.282383 8702 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:54:29.282433 8702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:29.308782 8702 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.69:46881
I20250116 02:54:29.308825 8829 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.69:46881 every 8 connection(s)
I20250116 02:54:29.309870 8702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data/info.pb
I20250116 02:54:29.318470 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 8702
I20250116 02:54:29.318634 8081 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/wal/instance
I20250116 02:54:29.320292 8830 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:54:29.320571 8830 heartbeater.cc:463] Registering TS with master...
I20250116 02:54:29.321406 8830 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:29.322126 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.70:0
--local_ip_for_outbound_sockets=127.7.228.70
--webserver_interface=127.7.228.70
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250116 02:54:29.359920 8119 ts_manager.cc:194] Registered new tserver with Master: ff972e037b0e40bba36f7896effdb71f (127.7.228.69:46881)
I20250116 02:54:29.361068 8119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.69:50855
I20250116 02:54:29.575827 8568 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
W20250116 02:54:29.607594 8833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:54:29.607828 8833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:54:29.607918 8833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:54:29.614480 8833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:54:29.614632 8833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.70
I20250116 02:54:29.622256 8833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data/info.pb
--webserver_interface=127.7.228.70
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.70
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:54:29.623435 8833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:54:29.624884 8833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:54:29.635445 8840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:29.635773 8839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:29.636329 8842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:54:29.637313 8833 server_base.cc:1034] running on GCE node
I20250116 02:54:29.637792 8833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:54:29.638525 8833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:54:29.639732 8833 hybrid_clock.cc:648] HybridClock initialized: now 1736996069639671 us; error 25 us; skew 500 ppm
I20250116 02:54:29.639966 8833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:54:29.642098 8833 webserver.cc:458] Webserver started at http://127.7.228.70:46821/ using document root <none> and password file <none>
I20250116 02:54:29.642724 8833 fs_manager.cc:362] Metadata directory not provided
I20250116 02:54:29.642832 8833 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:54:29.643075 8833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:54:29.645275 8833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data/instance:
uuid: "d5ba0801ecba49269379dd93eedde2f8"
format_stamp: "Formatted at 2025-01-16 02:54:29 on dist-test-slave-ldbs"
I20250116 02:54:29.645893 8833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/wal/instance:
uuid: "d5ba0801ecba49269379dd93eedde2f8"
format_stamp: "Formatted at 2025-01-16 02:54:29 on dist-test-slave-ldbs"
I20250116 02:54:29.650046 8833 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250116 02:54:29.652803 8848 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:29.653910 8833 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250116 02:54:29.654032 8833 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/wal
uuid: "d5ba0801ecba49269379dd93eedde2f8"
format_stamp: "Formatted at 2025-01-16 02:54:29 on dist-test-slave-ldbs"
I20250116 02:54:29.654179 8833 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:54:29.700544 8833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:54:29.701287 8833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:54:29.701582 8833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:54:29.702431 8833 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:54:29.703751 8833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:54:29.703822 8833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:29.703938 8833 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:54:29.703985 8833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:29.731626 8833 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.70:42173
I20250116 02:54:29.731660 8960 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.70:42173 every 8 connection(s)
I20250116 02:54:29.732721 8833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data/info.pb
I20250116 02:54:29.736654 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 8833
I20250116 02:54:29.736795 8081 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/wal/instance
I20250116 02:54:29.739910 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.71:0
--local_ip_for_outbound_sockets=127.7.228.71
--webserver_interface=127.7.228.71
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250116 02:54:29.744480 8961 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:54:29.744709 8961 heartbeater.cc:463] Registering TS with master...
I20250116 02:54:29.745502 8961 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:29.783355 8119 ts_manager.cc:194] Registered new tserver with Master: d5ba0801ecba49269379dd93eedde2f8 (127.7.228.70:42173)
I20250116 02:54:29.784219 8119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.70:55517
I20250116 02:54:29.967875 8699 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
W20250116 02:54:30.026336 8964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:54:30.026589 8964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:54:30.026721 8964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:54:30.033926 8964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:54:30.034126 8964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.71
I20250116 02:54:30.042997 8964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data/info.pb
--webserver_interface=127.7.228.71
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.71
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:54:30.044401 8964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:54:30.045953 8964 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:54:30.056915 8973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:30.057809 8971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:30.058079 8970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:54:30.059090 8964 server_base.cc:1034] running on GCE node
I20250116 02:54:30.059582 8964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:54:30.060367 8964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:54:30.061604 8964 hybrid_clock.cc:648] HybridClock initialized: now 1736996070061476 us; error 98 us; skew 500 ppm
I20250116 02:54:30.061879 8964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:54:30.063932 8964 webserver.cc:458] Webserver started at http://127.7.228.71:34529/ using document root <none> and password file <none>
I20250116 02:54:30.064590 8964 fs_manager.cc:362] Metadata directory not provided
I20250116 02:54:30.064700 8964 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:54:30.064996 8964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:54:30.067564 8964 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data/instance:
uuid: "dfb2cd26d0bb42c99dbd98c0c1a2f731"
format_stamp: "Formatted at 2025-01-16 02:54:30 on dist-test-slave-ldbs"
I20250116 02:54:30.068253 8964 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/wal/instance:
uuid: "dfb2cd26d0bb42c99dbd98c0c1a2f731"
format_stamp: "Formatted at 2025-01-16 02:54:30 on dist-test-slave-ldbs"
I20250116 02:54:30.072839 8964 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20250116 02:54:30.075707 8979 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:30.076821 8964 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250116 02:54:30.076949 8964 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/wal
uuid: "dfb2cd26d0bb42c99dbd98c0c1a2f731"
format_stamp: "Formatted at 2025-01-16 02:54:30 on dist-test-slave-ldbs"
I20250116 02:54:30.077121 8964 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:54:30.117290 8964 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:54:30.118094 8964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:54:30.118371 8964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:54:30.119261 8964 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:54:30.120656 8964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:54:30.120733 8964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:30.120862 8964 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:54:30.120918 8964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:30.149276 8964 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.71:44289
I20250116 02:54:30.149338 9091 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.71:44289 every 8 connection(s)
I20250116 02:54:30.150409 8964 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data/info.pb
I20250116 02:54:30.155139 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 8964
I20250116 02:54:30.155313 8081 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/wal/instance
I20250116 02:54:30.158205 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.72:0
--local_ip_for_outbound_sockets=127.7.228.72
--webserver_interface=127.7.228.72
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250116 02:54:30.162642 9092 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:54:30.162900 9092 heartbeater.cc:463] Registering TS with master...
I20250116 02:54:30.163563 9092 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:30.216507 8119 ts_manager.cc:194] Registered new tserver with Master: dfb2cd26d0bb42c99dbd98c0c1a2f731 (127.7.228.71:44289)
I20250116 02:54:30.217314 8119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.71:48993
I20250116 02:54:30.364328 8830 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
W20250116 02:54:30.449221 9095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:54:30.449496 9095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:54:30.449651 9095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:54:30.456418 9095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:54:30.456637 9095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.72
I20250116 02:54:30.464758 9095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data/info.pb
--webserver_interface=127.7.228.72
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.72
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:54:30.466020 9095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:54:30.467780 9095 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:54:30.477979 9101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:30.478600 9104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:30.479461 9102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:54:30.480536 9095 server_base.cc:1034] running on GCE node
I20250116 02:54:30.481019 9095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:54:30.481884 9095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:54:30.483073 9095 hybrid_clock.cc:648] HybridClock initialized: now 1736996070483019 us; error 41 us; skew 500 ppm
I20250116 02:54:30.483325 9095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:54:30.485116 9095 webserver.cc:458] Webserver started at http://127.7.228.72:33809/ using document root <none> and password file <none>
I20250116 02:54:30.485822 9095 fs_manager.cc:362] Metadata directory not provided
I20250116 02:54:30.485929 9095 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:54:30.486202 9095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:54:30.488355 9095 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data/instance:
uuid: "9b5741c935d64e23af0c52c50e78420c"
format_stamp: "Formatted at 2025-01-16 02:54:30 on dist-test-slave-ldbs"
I20250116 02:54:30.489006 9095 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/wal/instance:
uuid: "9b5741c935d64e23af0c52c50e78420c"
format_stamp: "Formatted at 2025-01-16 02:54:30 on dist-test-slave-ldbs"
I20250116 02:54:30.493427 9095 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20250116 02:54:30.496248 9110 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:30.497336 9095 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250116 02:54:30.497478 9095 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/wal
uuid: "9b5741c935d64e23af0c52c50e78420c"
format_stamp: "Formatted at 2025-01-16 02:54:30 on dist-test-slave-ldbs"
I20250116 02:54:30.497684 9095 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:54:30.514117 9095 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:54:30.514766 9095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:54:30.514995 9095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:54:30.515763 9095 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:54:30.516983 9095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:54:30.517055 9095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:30.517167 9095 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:54:30.517218 9095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:30.545614 9095 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.72:45211
I20250116 02:54:30.545673 9222 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.72:45211 every 8 connection(s)
I20250116 02:54:30.546697 9095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data/info.pb
I20250116 02:54:30.553238 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 9095
I20250116 02:54:30.553407 8081 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/wal/instance
I20250116 02:54:30.557444 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.73:0
--local_ip_for_outbound_sockets=127.7.228.73
--webserver_interface=127.7.228.73
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250116 02:54:30.560966 9223 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:54:30.561261 9223 heartbeater.cc:463] Registering TS with master...
I20250116 02:54:30.562004 9223 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:30.601380 8119 ts_manager.cc:194] Registered new tserver with Master: 9b5741c935d64e23af0c52c50e78420c (127.7.228.72:45211)
I20250116 02:54:30.602387 8119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.72:51049
I20250116 02:54:30.786733 8961 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
W20250116 02:54:30.858409 9226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:54:30.858661 9226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:54:30.858811 9226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:54:30.865566 9226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:54:30.865760 9226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.73
I20250116 02:54:30.874178 9226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.73:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data/info.pb
--webserver_interface=127.7.228.73
--webserver_port=0
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.73
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:54:30.875511 9226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:54:30.876909 9226 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:54:30.887055 9232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:30.887923 9233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:30.888810 9235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:54:30.889154 9226 server_base.cc:1034] running on GCE node
I20250116 02:54:30.889695 9226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:54:30.890453 9226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:54:30.891680 9226 hybrid_clock.cc:648] HybridClock initialized: now 1736996070891615 us; error 41 us; skew 500 ppm
I20250116 02:54:30.891921 9226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:54:30.893802 9226 webserver.cc:458] Webserver started at http://127.7.228.73:39105/ using document root <none> and password file <none>
I20250116 02:54:30.894385 9226 fs_manager.cc:362] Metadata directory not provided
I20250116 02:54:30.894495 9226 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:54:30.894747 9226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250116 02:54:30.896957 9226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data/instance:
uuid: "497640d49e7242aaa63256055e91414e"
format_stamp: "Formatted at 2025-01-16 02:54:30 on dist-test-slave-ldbs"
I20250116 02:54:30.897665 9226 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/wal/instance:
uuid: "497640d49e7242aaa63256055e91414e"
format_stamp: "Formatted at 2025-01-16 02:54:30 on dist-test-slave-ldbs"
I20250116 02:54:30.902148 9226 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20250116 02:54:30.904888 9241 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:30.906098 9226 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250116 02:54:30.906234 9226 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/wal
uuid: "497640d49e7242aaa63256055e91414e"
format_stamp: "Formatted at 2025-01-16 02:54:30 on dist-test-slave-ldbs"
I20250116 02:54:30.906397 9226 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:54:30.925318 9226 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:54:30.925956 9226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:54:30.926174 9226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:54:30.926895 9226 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:54:30.928109 9226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:54:30.928182 9226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:30.928300 9226 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:54:30.928354 9226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:30.960300 9226 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.73:42313
I20250116 02:54:30.960340 9353 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.73:42313 every 8 connection(s)
I20250116 02:54:30.961515 9226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data/info.pb
I20250116 02:54:30.963124 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 9226
I20250116 02:54:30.963346 8081 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/wal/instance
I20250116 02:54:30.974117 9354 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:54:30.974824 9354 heartbeater.cc:463] Registering TS with master...
I20250116 02:54:30.975528 9354 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:31.015029 8119 ts_manager.cc:194] Registered new tserver with Master: 497640d49e7242aaa63256055e91414e (127.7.228.73:42313)
I20250116 02:54:31.015918 8119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.73:53813
I20250116 02:54:31.019407 8081 external_mini_cluster.cc:934] 9 TS(s) registered with all masters
I20250116 02:54:31.038291 8081 test_util.cc:274] Using random seed: -372613838
I20250116 02:54:31.056798 8119 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54042:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250116 02:54:31.086848 8372 tablet_service.cc:1467] Processing CreateTablet for tablet 29c42185024a48cda0403d61962cf176 (DEFAULT_TABLE table=test-workload [id=6273b42324004cbfbc020e5da4f7a620]), partition=RANGE (key) PARTITION UNBOUNDED
I20250116 02:54:31.086848 9289 tablet_service.cc:1467] Processing CreateTablet for tablet 29c42185024a48cda0403d61962cf176 (DEFAULT_TABLE table=test-workload [id=6273b42324004cbfbc020e5da4f7a620]), partition=RANGE (key) PARTITION UNBOUNDED
I20250116 02:54:31.087143 8634 tablet_service.cc:1467] Processing CreateTablet for tablet 29c42185024a48cda0403d61962cf176 (DEFAULT_TABLE table=test-workload [id=6273b42324004cbfbc020e5da4f7a620]), partition=RANGE (key) PARTITION UNBOUNDED
I20250116 02:54:31.088137 8372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c42185024a48cda0403d61962cf176. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:54:31.088163 9289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c42185024a48cda0403d61962cf176. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:54:31.088513 8634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c42185024a48cda0403d61962cf176. 1 dirs total, 0 dirs full, 0 dirs failed
I20250116 02:54:31.097375 9378 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap starting.
I20250116 02:54:31.100584 9380 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Bootstrap starting.
I20250116 02:54:31.101135 9378 tablet_bootstrap.cc:654] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Neither blocks nor log segments found. Creating new log.
I20250116 02:54:31.102464 9379 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Bootstrap starting.
I20250116 02:54:31.102732 9378 log.cc:826] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Log is configured to *not* fsync() on all Append() calls
I20250116 02:54:31.104871 9378 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: No bootstrap required, opened a new log
I20250116 02:54:31.105105 9378 ts_tablet_manager.cc:1397] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20250116 02:54:31.105749 9379 tablet_bootstrap.cc:654] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Neither blocks nor log segments found. Creating new log.
I20250116 02:54:31.106518 9380 tablet_bootstrap.cc:654] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Neither blocks nor log segments found. Creating new log.
I20250116 02:54:31.106915 9379 log.cc:826] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Log is configured to *not* fsync() on all Append() calls
I20250116 02:54:31.107789 9380 log.cc:826] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Log is configured to *not* fsync() on all Append() calls
I20250116 02:54:31.109839 9379 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: No bootstrap required, opened a new log
I20250116 02:54:31.110095 9379 ts_tablet_manager.cc:1397] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20250116 02:54:31.110230 9380 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: No bootstrap required, opened a new log
I20250116 02:54:31.110478 9380 ts_tablet_manager.cc:1397] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20250116 02:54:31.112381 9378 raft_consensus.cc:357] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:54:31.112690 9378 raft_consensus.cc:383] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:54:31.112779 9378 raft_consensus.cc:738] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0791364971ba49f3b2813bdc36c11eda, State: Initialized, Role: FOLLOWER
I20250116 02:54:31.113317 9378 consensus_queue.cc:260] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:54:31.115703 9378 ts_tablet_manager.cc:1428] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20250116 02:54:31.120543 9379 raft_consensus.cc:357] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:54:31.120925 9379 raft_consensus.cc:383] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:54:31.121021 9379 raft_consensus.cc:738] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429bf4133bf54797b117ef8fad632920, State: Initialized, Role: FOLLOWER
I20250116 02:54:31.121618 9379 consensus_queue.cc:260] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:54:31.123039 9379 ts_tablet_manager.cc:1428] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Time spent starting tablet: real 0.013s user 0.004s sys 0.007s
I20250116 02:54:31.126000 9380 raft_consensus.cc:357] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:54:31.126389 9380 raft_consensus.cc:383] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250116 02:54:31.126492 9380 raft_consensus.cc:738] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 497640d49e7242aaa63256055e91414e, State: Initialized, Role: FOLLOWER
I20250116 02:54:31.127050 9380 consensus_queue.cc:260] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:54:31.127872 9354 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
I20250116 02:54:31.128154 9380 ts_tablet_manager.cc:1428] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Time spent starting tablet: real 0.011s user 0.003s sys 0.006s
I20250116 02:54:31.136657 9384 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250116 02:54:31.136832 9384 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:54:31.138307 9384 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 497640d49e7242aaa63256055e91414e (127.7.228.73:42313), 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805)
I20250116 02:54:31.146869 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "0791364971ba49f3b2813bdc36c11eda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "497640d49e7242aaa63256055e91414e" is_pre_election: true
I20250116 02:54:31.146869 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "0791364971ba49f3b2813bdc36c11eda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429bf4133bf54797b117ef8fad632920" is_pre_election: true
I20250116 02:54:31.147343 9309 raft_consensus.cc:2463] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0791364971ba49f3b2813bdc36c11eda in term 0.
I20250116 02:54:31.147363 8392 raft_consensus.cc:2463] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0791364971ba49f3b2813bdc36c11eda in term 0.
I20250116 02:54:31.147992 8587 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0791364971ba49f3b2813bdc36c11eda, 429bf4133bf54797b117ef8fad632920, 497640d49e7242aaa63256055e91414e; no voters:
I20250116 02:54:31.148355 9384 raft_consensus.cc:2798] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250116 02:54:31.148494 9384 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250116 02:54:31.148593 9384 raft_consensus.cc:3054] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:54:31.150599 9384 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:54:31.151094 9384 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [CANDIDATE]: Term 1 election: Requested vote from peers 497640d49e7242aaa63256055e91414e (127.7.228.73:42313), 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805)
I20250116 02:54:31.151492 9309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "0791364971ba49f3b2813bdc36c11eda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "497640d49e7242aaa63256055e91414e"
I20250116 02:54:31.151566 8392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "0791364971ba49f3b2813bdc36c11eda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "429bf4133bf54797b117ef8fad632920"
I20250116 02:54:31.151654 9309 raft_consensus.cc:3054] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:54:31.151767 8392 raft_consensus.cc:3054] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 0 FOLLOWER]: Advancing to term 1
I20250116 02:54:31.153450 9309 raft_consensus.cc:2463] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0791364971ba49f3b2813bdc36c11eda in term 1.
I20250116 02:54:31.154095 8588 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0791364971ba49f3b2813bdc36c11eda, 497640d49e7242aaa63256055e91414e; no voters:
I20250116 02:54:31.154328 9384 raft_consensus.cc:2798] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 1 FOLLOWER]: Leader election won for term 1
I20250116 02:54:31.154671 9384 raft_consensus.cc:695] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 1 LEADER]: Becoming Leader. State: Replica: 0791364971ba49f3b2813bdc36c11eda, State: Running, Role: LEADER
I20250116 02:54:31.154934 8392 raft_consensus.cc:2463] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0791364971ba49f3b2813bdc36c11eda in term 1.
I20250116 02:54:31.155093 9384 consensus_queue.cc:237] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:54:31.159036 8119 catalog_manager.cc:5526] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda reported cstate change: term changed from 0 to 1, leader changed from <none> to 0791364971ba49f3b2813bdc36c11eda (127.7.228.68). New cstate: current_term: 1 leader_uuid: "0791364971ba49f3b2813bdc36c11eda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } health_report { overall_health: HEALTHY } } }
W20250116 02:54:31.167115 8700 tablet.cc:2367] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250116 02:54:31.172883 8081 maintenance_mode-itest.cc:745] Restarting batch of 3 tservers: ff972e037b0e40bba36f7896effdb71f,d5ba0801ecba49269379dd93eedde2f8,dfb2cd26d0bb42c99dbd98c0c1a2f731
W20250116 02:54:31.212922 9355 tablet.cc:2367] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250116 02:54:31.219496 9092 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
I20250116 02:54:31.253198 8392 raft_consensus.cc:1270] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 1 FOLLOWER]: Refusing update from remote peer 0791364971ba49f3b2813bdc36c11eda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:54:31.254221 9384 consensus_queue.cc:1035] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Connected to new peer: Peer: permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:54:31.255846 9309 raft_consensus.cc:1270] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Refusing update from remote peer 0791364971ba49f3b2813bdc36c11eda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250116 02:54:31.257095 9389 consensus_queue.cc:1035] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Connected to new peer: Peer: permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250116 02:54:31.286092 9404 mvcc.cc:204] Tried to move back new op lower bound from 7114735907841662976 to 7114735907454607360. Current Snapshot: MvccSnapshot[applied={T|T < 7114735907841662976}]
I20250116 02:54:31.298532 9406 mvcc.cc:204] Tried to move back new op lower bound from 7114735907841662976 to 7114735907454607360. Current Snapshot: MvccSnapshot[applied={T|T < 7114735907841662976}]
I20250116 02:54:31.303145 9405 mvcc.cc:204] Tried to move back new op lower bound from 7114735907841662976 to 7114735907454607360. Current Snapshot: MvccSnapshot[applied={T|T < 7114735907841662976}]
I20250116 02:54:31.605221 9223 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
I20250116 02:54:32.131299 8119 ts_manager.cc:292] Set tserver state for ff972e037b0e40bba36f7896effdb71f to MAINTENANCE_MODE
I20250116 02:54:32.504703 8119 ts_manager.cc:292] Set tserver state for d5ba0801ecba49269379dd93eedde2f8 to MAINTENANCE_MODE
I20250116 02:54:32.530637 8119 ts_manager.cc:292] Set tserver state for dfb2cd26d0bb42c99dbd98c0c1a2f731 to MAINTENANCE_MODE
I20250116 02:54:34.855717 8765 tablet_service.cc:1422] Tablet server ff972e037b0e40bba36f7896effdb71f set to quiescing
I20250116 02:54:34.855958 8765 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:54:35.457029 9027 tablet_service.cc:1422] Tablet server dfb2cd26d0bb42c99dbd98c0c1a2f731 set to quiescing
I20250116 02:54:35.457268 9027 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:54:36.022728 9393 meta_cache.cc:1261] Time spent looking up entry by key: real 0.053s user 0.000s sys 0.001s
I20250116 02:54:37.068893 8896 tablet_service.cc:1422] Tablet server d5ba0801ecba49269379dd93eedde2f8 set to quiescing
I20250116 02:54:37.069213 8896 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:54:38.681244 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 8702
I20250116 02:54:38.748129 8825 generic_service.cc:196] Checking for leaks (request via RPC)
I20250116 02:54:38.930151 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 8702
I20250116 02:54:38.991816 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.69:46881
--local_ip_for_outbound_sockets=127.7.228.69
--tserver_master_addrs=127.7.228.126:43731
--webserver_port=40473
--webserver_interface=127.7.228.69
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250116 02:54:39.635869 9520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:54:39.636426 9520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:54:39.636665 9520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:54:39.661844 9520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:54:39.663802 9520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.69
I20250116 02:54:39.697400 9520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.69:46881
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data/info.pb
--webserver_interface=127.7.228.69
--webserver_port=40473
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.69
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:54:39.699901 9520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:54:39.703708 9520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:54:39.773950 9526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:39.777307 9527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:39.850899 9529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:54:39.890172 9520 server_base.cc:1034] running on GCE node
I20250116 02:54:39.891094 9520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:54:39.896697 9520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:54:39.905009 9520 hybrid_clock.cc:648] HybridClock initialized: now 1736996079904886 us; error 82 us; skew 500 ppm
I20250116 02:54:39.905589 9520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:54:39.941891 9520 webserver.cc:458] Webserver started at http://127.7.228.69:40473/ using document root <none> and password file <none>
I20250116 02:54:39.942767 9520 fs_manager.cc:362] Metadata directory not provided
I20250116 02:54:39.942975 9520 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:54:39.991163 9520 fs_manager.cc:714] Time spent opening directory manager: real 0.046s user 0.005s sys 0.000s
I20250116 02:54:40.011682 9535 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:40.015614 9520 fs_manager.cc:730] Time spent opening block manager: real 0.022s user 0.005s sys 0.000s
I20250116 02:54:40.015901 9520 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/wal
uuid: "ff972e037b0e40bba36f7896effdb71f"
format_stamp: "Formatted at 2025-01-16 02:54:29 on dist-test-slave-ldbs"
I20250116 02:54:40.017125 9520 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:54:40.095646 9520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:54:40.096554 9520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:54:40.097049 9520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:54:40.104574 9520 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:54:40.110620 9520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:54:40.110886 9520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:40.111075 9520 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:54:40.111192 9520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:40.607076 9520 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.69:46881
I20250116 02:54:40.607131 9650 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.69:46881 every 8 connection(s)
I20250116 02:54:40.608951 9520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data/info.pb
I20250116 02:54:40.614869 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 9520
I20250116 02:54:40.615095 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 8833
I20250116 02:54:40.652917 8956 generic_service.cc:196] Checking for leaks (request via RPC)
I20250116 02:54:40.785975 9651 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:54:40.786267 9651 heartbeater.cc:463] Registering TS with master...
I20250116 02:54:40.787113 9651 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:40.789219 8119 ts_manager.cc:194] Re-registered known tserver with Master: ff972e037b0e40bba36f7896effdb71f (127.7.228.69:46881)
I20250116 02:54:40.790215 8119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.69:50663
I20250116 02:54:40.858461 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 8833
W20250116 02:54:40.906009 8105 connection.cc:537] server connection from 127.7.228.70:55517 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250116 02:54:40.907212 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.70:42173
--local_ip_for_outbound_sockets=127.7.228.70
--tserver_master_addrs=127.7.228.126:43731
--webserver_port=46821
--webserver_interface=127.7.228.70
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250116 02:54:41.508481 9659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:54:41.515890 9659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:54:41.516045 9659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:54:41.533125 9659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:54:41.534377 9659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.70
I20250116 02:54:41.549934 9659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.70:42173
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data/info.pb
--webserver_interface=127.7.228.70
--webserver_port=46821
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.70
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:54:41.551908 9659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:54:41.556522 9659 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:54:41.631810 9664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:41.634339 9665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:54:41.647632 9659 server_base.cc:1034] running on GCE node
W20250116 02:54:41.649690 9668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:54:41.651908 9659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:54:41.657488 9659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:54:41.660826 9659 hybrid_clock.cc:648] HybridClock initialized: now 1736996081660630 us; error 163 us; skew 500 ppm
I20250116 02:54:41.661201 9659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:54:41.698246 9659 webserver.cc:458] Webserver started at http://127.7.228.70:46821/ using document root <none> and password file <none>
I20250116 02:54:41.699589 9659 fs_manager.cc:362] Metadata directory not provided
I20250116 02:54:41.699852 9659 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:54:41.717212 9659 fs_manager.cc:714] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20250116 02:54:41.728654 9673 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:41.735114 9659 fs_manager.cc:730] Time spent opening block manager: real 0.015s user 0.001s sys 0.006s
I20250116 02:54:41.735345 9659 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/wal
uuid: "d5ba0801ecba49269379dd93eedde2f8"
format_stamp: "Formatted at 2025-01-16 02:54:29 on dist-test-slave-ldbs"
I20250116 02:54:41.737380 9659 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:54:41.795878 9651 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
I20250116 02:54:41.934702 9659 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:54:41.942143 9659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:54:41.942483 9659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:54:41.986413 9659 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:54:42.046033 9659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:54:42.046231 9659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250116 02:54:42.046383 9659 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:54:42.046496 9659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:43.027833 9786 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.70:42173 every 8 connection(s)
I20250116 02:54:43.028128 9659 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.70:42173
I20250116 02:54:43.029927 9659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data/info.pb
I20250116 02:54:43.037770 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 9659
I20250116 02:54:43.037968 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 8964
I20250116 02:54:43.135437 9787 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:54:43.135766 9787 heartbeater.cc:463] Registering TS with master...
I20250116 02:54:43.136859 9787 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:43.138952 8120 ts_manager.cc:194] Re-registered known tserver with Master: d5ba0801ecba49269379dd93eedde2f8 (127.7.228.70:42173)
I20250116 02:54:43.139945 8120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.70:58441
I20250116 02:54:43.140697 9087 generic_service.cc:196] Checking for leaks (request via RPC)
I20250116 02:54:43.315316 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 8964
I20250116 02:54:43.343745 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.71:44289
--local_ip_for_outbound_sockets=127.7.228.71
--tserver_master_addrs=127.7.228.126:43731
--webserver_port=34529
--webserver_interface=127.7.228.71
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250116 02:54:44.078646 9795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:54:44.079129 9795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:54:44.079342 9795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:54:44.090528 9795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:54:44.090832 9795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.71
I20250116 02:54:44.103914 9795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.71:44289
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data/info.pb
--webserver_interface=127.7.228.71
--webserver_port=34529
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.71
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:54:44.106060 9795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:54:44.111601 9795 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250116 02:54:44.143013 9787 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
W20250116 02:54:44.144568 9801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:44.147986 9802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:54:44.160004 9805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:54:44.168617 9795 server_base.cc:1034] running on GCE node
I20250116 02:54:44.169469 9795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:54:44.185388 9795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:54:44.189704 9795 hybrid_clock.cc:648] HybridClock initialized: now 1736996084189599 us; error 78 us; skew 500 ppm
I20250116 02:54:44.190184 9795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:54:44.208168 9795 webserver.cc:458] Webserver started at http://127.7.228.71:34529/ using document root <none> and password file <none>
I20250116 02:54:44.209208 9795 fs_manager.cc:362] Metadata directory not provided
I20250116 02:54:44.209359 9795 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:54:44.258471 9795 fs_manager.cc:714] Time spent opening directory manager: real 0.047s user 0.017s sys 0.028s
I20250116 02:54:44.268298 9810 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:44.270097 9795 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.006s sys 0.003s
I20250116 02:54:44.270296 9795 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/wal
uuid: "dfb2cd26d0bb42c99dbd98c0c1a2f731"
format_stamp: "Formatted at 2025-01-16 02:54:30 on dist-test-slave-ldbs"
I20250116 02:54:44.271396 9795 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:54:44.320740 9795 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:54:44.321796 9795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:54:44.322132 9795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:54:44.329900 9795 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:54:44.337872 9795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:54:44.338006 9795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:44.338143 9795 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:54:44.338244 9795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:54:45.204289 9795 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.71:44289
I20250116 02:54:45.204355 9924 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.71:44289 every 8 connection(s)
I20250116 02:54:45.206095 9795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data/info.pb
I20250116 02:54:45.212877 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 9795
I20250116 02:54:45.301103 9925 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:54:45.301425 9925 heartbeater.cc:463] Registering TS with master...
I20250116 02:54:45.302762 9925 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:45.304404 8120 ts_manager.cc:194] Re-registered known tserver with Master: dfb2cd26d0bb42c99dbd98c0c1a2f731 (127.7.228.71:44289)
I20250116 02:54:45.305379 8120 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.71:53225
I20250116 02:54:46.308192 9925 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
I20250116 02:54:46.445348 8634 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250116 02:54:46.500943 9289 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:54:46.525720 8503 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:54:46.552033 9158 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:54:46.553946 8241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:54:46.647380 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:54:46.649292 8372 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250116 02:54:46.686816 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:54:46.687489 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:54:48.363629 8120 ts_manager.cc:281] Unset tserver state for ff972e037b0e40bba36f7896effdb71f from MAINTENANCE_MODE
I20250116 02:54:48.404949 9925 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:48.405830 8699 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:48.406376 9354 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:48.406872 8437 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:48.617091 8568 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:48.644522 9223 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:48.809062 8120 ts_manager.cc:281] Unset tserver state for d5ba0801ecba49269379dd93eedde2f8 from MAINTENANCE_MODE
I20250116 02:54:48.812549 8306 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:48.817651 9651 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:48.826901 8120 ts_manager.cc:281] Unset tserver state for dfb2cd26d0bb42c99dbd98c0c1a2f731 from MAINTENANCE_MODE
I20250116 02:54:49.156960 9787 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:49.413465 9925 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:49.413633 9354 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:49.415930 8699 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:49.423518 8437 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:49.621425 8568 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:49.648236 9223 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:49.816076 8306 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:49.820591 9651 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:54:51.704836 8081 maintenance_mode-itest.cc:745] Restarting batch of 3 tservers: 1361eef60904481394d5230d1bc84711,429bf4133bf54797b117ef8fad632920,660967712270485c81d6a5f1239c89b1
I20250116 02:54:52.525419 8120 ts_manager.cc:292] Set tserver state for 660967712270485c81d6a5f1239c89b1 to MAINTENANCE_MODE
I20250116 02:54:52.949369 8120 ts_manager.cc:292] Set tserver state for 1361eef60904481394d5230d1bc84711 to MAINTENANCE_MODE
I20250116 02:54:54.101402 8120 ts_manager.cc:292] Set tserver state for 429bf4133bf54797b117ef8fad632920 to MAINTENANCE_MODE
I20250116 02:54:55.620452 8503 tablet_service.cc:1422] Tablet server 660967712270485c81d6a5f1239c89b1 set to quiescing
I20250116 02:54:55.620704 8503 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:54:55.645418 8241 tablet_service.cc:1422] Tablet server 1361eef60904481394d5230d1bc84711 set to quiescing
I20250116 02:54:55.645727 8241 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:54:57.862293 8372 tablet_service.cc:1422] Tablet server 429bf4133bf54797b117ef8fad632920 set to quiescing
I20250116 02:54:57.862480 8372 tablet_service.cc:1429] Tablet server has 0 leaders and 2 scanners
I20250116 02:55:02.368795 8372 tablet_service.cc:1422] Tablet server 429bf4133bf54797b117ef8fad632920 set to quiescing
I20250116 02:55:02.369038 8372 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:03.731702 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 8179
I20250116 02:55:03.767320 8300 generic_service.cc:196] Checking for leaks (request via RPC)
I20250116 02:55:04.072049 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 8179
I20250116 02:55:04.094966 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.65:40373
--local_ip_for_outbound_sockets=127.7.228.65
--tserver_master_addrs=127.7.228.126:43731
--webserver_port=45657
--webserver_interface=127.7.228.65
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250116 02:55:04.651615 10185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:55:04.652223 10185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:55:04.652470 10185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:55:04.667047 10185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:55:04.667306 10185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.65
I20250116 02:55:04.688076 10185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.65:40373
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.7.228.65
--webserver_port=45657
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.65
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:55:04.690493 10185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:55:04.692778 10185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:55:04.759397 10192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:55:04.780129 10191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:55:04.783344 10194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:55:04.784060 10185 server_base.cc:1034] running on GCE node
I20250116 02:55:04.785319 10185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:55:04.798029 10185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:55:04.804272 10185 hybrid_clock.cc:648] HybridClock initialized: now 1736996104804170 us; error 67 us; skew 500 ppm
I20250116 02:55:04.804755 10185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:55:04.981917 10185 webserver.cc:458] Webserver started at http://127.7.228.65:45657/ using document root <none> and password file <none>
I20250116 02:55:04.983109 10185 fs_manager.cc:362] Metadata directory not provided
I20250116 02:55:04.983338 10185 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:55:04.996891 10185 fs_manager.cc:714] Time spent opening directory manager: real 0.011s user 0.005s sys 0.004s
I20250116 02:55:05.007413 10200 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:55:05.009104 10185 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.003s sys 0.006s
I20250116 02:55:05.009307 10185 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/wal
uuid: "1361eef60904481394d5230d1bc84711"
format_stamp: "Formatted at 2025-01-16 02:54:27 on dist-test-slave-ldbs"
I20250116 02:55:05.010383 10185 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:55:05.257730 10185 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:55:05.258816 10185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:55:05.259086 10185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:55:05.301899 10185 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:55:05.310060 10185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:55:05.310205 10185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250116 02:55:05.310336 10185 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:55:05.310439 10185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:55:05.998344 10314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.65:40373 every 8 connection(s)
I20250116 02:55:06.000761 10185 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.65:40373
I20250116 02:55:06.002632 10185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data/info.pb
I20250116 02:55:06.004877 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 10185
I20250116 02:55:06.005031 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 8309
I20250116 02:55:06.074877 8432 generic_service.cc:196] Checking for leaks (request via RPC)
I20250116 02:55:06.124363 10316 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:55:06.124668 10316 heartbeater.cc:463] Registering TS with master...
I20250116 02:55:06.125561 10316 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:06.128860 8118 ts_manager.cc:194] Re-registered known tserver with Master: 1361eef60904481394d5230d1bc84711 (127.7.228.65:40373)
I20250116 02:55:06.129992 8118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.65:36283
W20250116 02:55:06.429816 10324 raft_consensus.cc:668] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: failed to trigger leader election: Illegal state: leader elections are disabled
I20250116 02:55:06.430176 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 8309
W20250116 02:55:06.498860 8587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.228.66:41805 (error 108)
I20250116 02:55:06.499758 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.66:41805
--local_ip_for_outbound_sockets=127.7.228.66
--tserver_master_addrs=127.7.228.126:43731
--webserver_port=46195
--webserver_interface=127.7.228.66
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250116 02:55:06.519439 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.66:41805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250116 02:55:07.002049 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.66:41805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250116 02:55:07.135020 10316 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
I20250116 02:55:07.167179 9396 meta_cache.cc:1510] marking tablet server 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805) as failed
W20250116 02:55:07.167434 9396 meta_cache.cc:302] tablet 29c42185024a48cda0403d61962cf176: replica 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.7.228.66:41805: connect: Connection refused (error 111)
W20250116 02:55:07.167837 10327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:55:07.168273 10327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:55:07.168490 10327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:55:07.190508 10327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:55:07.190723 10327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.66
I20250116 02:55:07.211385 10327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.66:41805
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.7.228.66
--webserver_port=46195
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.66
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:55:07.213058 10327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:55:07.215122 10327 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:55:07.248282 10336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:55:07.249599 10337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:55:07.256605 10339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:55:07.261580 10327 server_base.cc:1034] running on GCE node
I20250116 02:55:07.262276 10327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:55:07.263556 10327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:55:07.266389 10327 hybrid_clock.cc:648] HybridClock initialized: now 1736996107266260 us; error 90 us; skew 500 ppm
I20250116 02:55:07.266808 10327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:55:07.287068 10327 webserver.cc:458] Webserver started at http://127.7.228.66:46195/ using document root <none> and password file <none>
I20250116 02:55:07.288697 10327 fs_manager.cc:362] Metadata directory not provided
I20250116 02:55:07.288930 10327 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:55:07.304822 10327 fs_manager.cc:714] Time spent opening directory manager: real 0.013s user 0.006s sys 0.001s
W20250116 02:55:07.305621 9366 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.7.228.68:41593, user_credentials={real_user=slave}} blocked reactor thread for 52608.2us
I20250116 02:55:07.313066 10345 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:55:07.314998 10327 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20250116 02:55:07.315290 10327 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal
uuid: "429bf4133bf54797b117ef8fad632920"
format_stamp: "Formatted at 2025-01-16 02:54:28 on dist-test-slave-ldbs"
I20250116 02:55:07.316488 10327 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:55:07.398088 10327 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:55:07.399307 10327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:55:07.399653 10327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:55:07.404904 10327 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:55:07.407509 10352 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250116 02:55:07.421967 10327 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250116 02:55:07.422297 10327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.001s
I20250116 02:55:07.422516 10327 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250116 02:55:07.427029 10327 ts_tablet_manager.cc:610] Registered 1 tablets
I20250116 02:55:07.427175 10327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.001s
I20250116 02:55:07.427429 10352 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Bootstrap starting.
W20250116 02:55:07.567512 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.66:41805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250116 02:55:08.094873 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.66:41805: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250116 02:55:08.582805 10460 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.66:41805 every 8 connection(s)
I20250116 02:55:08.583168 10327 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.66:41805
I20250116 02:55:08.585505 10327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data/info.pb
I20250116 02:55:08.588917 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 10327
I20250116 02:55:08.589087 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 8440
I20250116 02:55:08.678123 8562 generic_service.cc:196] Checking for leaks (request via RPC)
W20250116 02:55:08.689905 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
I20250116 02:55:08.694854 10461 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:55:08.695096 10461 heartbeater.cc:463] Registering TS with master...
I20250116 02:55:08.696141 10461 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:08.698973 8118 ts_manager.cc:194] Re-registered known tserver with Master: 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805)
I20250116 02:55:08.700474 8118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.66:43881
I20250116 02:55:08.986888 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 8440
I20250116 02:55:09.032581 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.67:33499
--local_ip_for_outbound_sockets=127.7.228.67
--tserver_master_addrs=127.7.228.126:43731
--webserver_port=44901
--webserver_interface=127.7.228.67
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250116 02:55:09.190572 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
I20250116 02:55:09.240350 10352 log.cc:826] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Log is configured to *not* fsync() on all Append() calls
W20250116 02:55:09.555133 10469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:55:09.555596 10469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:55:09.555809 10469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:55:09.567063 10469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:55:09.567454 10469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.67
I20250116 02:55:09.581347 10469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.67:33499
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.7.228.67
--webserver_port=44901
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.67
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:55:09.583515 10469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:55:09.585664 10469 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:55:09.616508 10477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:55:09.623842 10480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:55:09.633809 10476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:55:09.645491 10469 server_base.cc:1034] running on GCE node
I20250116 02:55:09.646363 10469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:55:09.688059 10469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:55:09.705832 10461 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
I20250116 02:55:09.710134 10469 hybrid_clock.cc:648] HybridClock initialized: now 1736996109710067 us; error 57 us; skew 500 ppm
I20250116 02:55:09.712589 10469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250116 02:55:09.730337 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250116 02:55:09.738715 10469 webserver.cc:458] Webserver started at http://127.7.228.67:44901/ using document root <none> and password file <none>
I20250116 02:55:09.746690 10469 fs_manager.cc:362] Metadata directory not provided
I20250116 02:55:09.746942 10469 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:55:09.812085 10469 fs_manager.cc:714] Time spent opening directory manager: real 0.059s user 0.014s sys 0.013s
I20250116 02:55:09.817386 10486 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:55:09.836202 10469 fs_manager.cc:730] Time spent opening block manager: real 0.023s user 0.005s sys 0.000s
I20250116 02:55:09.838122 10469 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/wal
uuid: "660967712270485c81d6a5f1239c89b1"
format_stamp: "Formatted at 2025-01-16 02:54:28 on dist-test-slave-ldbs"
I20250116 02:55:09.841291 10469 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:55:09.901943 10469 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:55:09.902958 10469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:55:09.903260 10469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:55:09.904563 10469 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:55:09.913566 10469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:55:09.913723 10469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250116 02:55:09.913862 10469 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:55:09.913969 10469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20250116 02:55:10.245527 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20250116 02:55:10.703979 10453 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 429bf4133bf54797b117ef8fad632920 is lagging by at least 6 ops behind the committed index
W20250116 02:55:10.895018 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
I20250116 02:55:11.015165 10600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.67:33499 every 8 connection(s)
I20250116 02:55:11.015439 10469 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.67:33499
I20250116 02:55:11.019194 10469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data/info.pb
I20250116 02:55:11.030205 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 10469
I20250116 02:55:11.170086 10601 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:55:11.170519 10601 heartbeater.cc:463] Registering TS with master...
I20250116 02:55:11.171502 10601 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:11.173103 8118 ts_manager.cc:194] Re-registered known tserver with Master: 660967712270485c81d6a5f1239c89b1 (127.7.228.67:33499)
I20250116 02:55:11.174165 8118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.67:60253
W20250116 02:55:11.372787 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250116 02:55:11.961611 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
I20250116 02:55:12.177886 10601 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
I20250116 02:55:12.196210 8634 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250116 02:55:12.331810 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:12.341675 10394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:12.345152 9289 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250116 02:55:12.365257 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:12.430774 9158 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:12.454919 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:12.492012 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:12.526834 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:55:12.532374 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
W20250116 02:55:13.081724 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
W20250116 02:55:13.559545 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
I20250116 02:55:13.751209 10453 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 429bf4133bf54797b117ef8fad632920 is lagging by at least 310 ops behind the committed index [suppressed 27 similar messages]
W20250116 02:55:14.111151 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
I20250116 02:55:14.251209 10352 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Bootstrap replaying log segment 1/1 (1.46M/4.96M this segment, stats: ops{read=864 overwritten=0 applied=864 ignored=0} inserts{seen=43150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:55:14.623123 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
W20250116 02:55:15.129606 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250116 02:55:15.646301 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
W20250116 02:55:16.120463 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
W20250116 02:55:16.683307 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
I20250116 02:55:16.752823 10330 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 429bf4133bf54797b117ef8fad632920 is lagging by at least 669 ops behind the committed index [suppressed 28 similar messages]
W20250116 02:55:17.214890 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
W20250116 02:55:17.719336 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20250116 02:55:18.303220 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
W20250116 02:55:18.843456 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
I20250116 02:55:19.253950 10352 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Bootstrap replaying log segment 1/1 (3.54M/4.96M this segment, stats: ops{read=2083 overwritten=0 applied=2083 ignored=0} inserts{seen=104100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:55:19.276208 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
I20250116 02:55:19.439591 8634 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250116 02:55:19.506270 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:19.531944 10394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:19.534116 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:19.551366 9289 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250116 02:55:19.571601 9158 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:19.603247 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:19.686559 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:19.694911 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:19.753000 10653 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 429bf4133bf54797b117ef8fad632920 is lagging by at least 950 ops behind the committed index [suppressed 27 similar messages]
W20250116 02:55:19.915871 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
W20250116 02:55:20.417371 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
W20250116 02:55:20.935918 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
W20250116 02:55:21.464905 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
W20250116 02:55:21.987015 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
W20250116 02:55:22.569353 8587 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
I20250116 02:55:22.799914 10352 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Bootstrap replayed 1/1 log segments. Stats: ops{read=2919 overwritten=0 applied=2918 ignored=0} inserts{seen=145850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250116 02:55:22.800505 10352 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Bootstrap complete.
I20250116 02:55:22.802549 10352 ts_tablet_manager.cc:1397] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Time spent bootstrapping tablet: real 15.375s user 11.464s sys 0.575s
I20250116 02:55:22.810235 10352 raft_consensus.cc:357] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:55:22.823901 10352 raft_consensus.cc:738] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429bf4133bf54797b117ef8fad632920, State: Initialized, Role: FOLLOWER
I20250116 02:55:22.824779 10352 consensus_queue.cc:260] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2918, Last appended: 1.2919, Last appended by leader: 2919, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:55:22.828648 10352 ts_tablet_manager.cc:1428] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Time spent starting tablet: real 0.026s user 0.013s sys 0.004s
I20250116 02:55:22.847481 10662 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 429bf4133bf54797b117ef8fad632920 is lagging by at least 1313 ops behind the committed index [suppressed 28 similar messages]
I20250116 02:55:22.897958 10413 raft_consensus.cc:1212] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2918->[1.2919-1.3343] Dedup: 1.2919->[1.2920-1.3343]
I20250116 02:55:23.200538 10718 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0791364971ba49f3b2813bdc36c11eda)
I20250116 02:55:23.200781 10718 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:55:23.202728 10718 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 497640d49e7242aaa63256055e91414e (127.7.228.73:42313), 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:55:23.267434 9307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "429bf4133bf54797b117ef8fad632920" candidate_term: 2 candidate_status { last_received { term: 1 index: 3343 } } ignore_live_leader: false dest_uuid: "497640d49e7242aaa63256055e91414e" is_pre_election: true
I20250116 02:55:23.271471 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "429bf4133bf54797b117ef8fad632920" candidate_term: 2 candidate_status { last_received { term: 1 index: 3343 } } ignore_live_leader: false dest_uuid: "0791364971ba49f3b2813bdc36c11eda" is_pre_election: true
I20250116 02:55:23.273797 10349 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 429bf4133bf54797b117ef8fad632920; no voters: 0791364971ba49f3b2813bdc36c11eda, 497640d49e7242aaa63256055e91414e
I20250116 02:55:23.712193 10718 raft_consensus.cc:2743] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:55:27.146482 8634 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250116 02:55:27.146502 10394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:27.179787 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:27.180402 9289 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250116 02:55:27.214270 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:27.245620 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:27.257014 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:27.291460 9158 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:27.316215 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:29.276454 8118 ts_manager.cc:281] Unset tserver state for 660967712270485c81d6a5f1239c89b1 from MAINTENANCE_MODE
I20250116 02:55:29.277671 8699 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:29.278317 10601 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:29.279417 10316 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:29.288466 9787 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:29.336902 10461 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:29.520169 9925 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:29.522266 9354 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:29.728144 9223 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:29.917043 9651 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:30.193410 8116 ts_manager.cc:281] Unset tserver state for 429bf4133bf54797b117ef8fad632920 from MAINTENANCE_MODE
I20250116 02:55:30.284412 8699 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:30.289119 10316 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:30.291699 9787 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:30.292850 10601 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:30.343502 10461 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:30.498459 8116 ts_manager.cc:281] Unset tserver state for 1361eef60904481394d5230d1bc84711 from MAINTENANCE_MODE
I20250116 02:55:30.524856 9925 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:30.528090 9354 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:30.731515 9223 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:30.920096 9651 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:31.294319 10316 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:31.297590 8699 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:31.297760 10601 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:31.299432 9787 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:31.347373 10461 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:33.485832 8081 maintenance_mode-itest.cc:745] Restarting batch of 2 tservers: 9b5741c935d64e23af0c52c50e78420c,497640d49e7242aaa63256055e91414e
I20250116 02:55:34.836697 8117 ts_manager.cc:292] Set tserver state for 497640d49e7242aaa63256055e91414e to MAINTENANCE_MODE
I20250116 02:55:34.841321 8116 ts_manager.cc:292] Set tserver state for 9b5741c935d64e23af0c52c50e78420c to MAINTENANCE_MODE
I20250116 02:55:37.867362 9158 tablet_service.cc:1422] Tablet server 9b5741c935d64e23af0c52c50e78420c set to quiescing
I20250116 02:55:37.867599 9158 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:38.132890 9289 tablet_service.cc:1422] Tablet server 497640d49e7242aaa63256055e91414e set to quiescing
I20250116 02:55:38.133179 9289 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
W20250116 02:55:39.650696 9366 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.7.228.68:41593, user_credentials={real_user=slave}} blocked reactor thread for 61326.7us
I20250116 02:55:40.423720 9289 tablet_service.cc:1422] Tablet server 497640d49e7242aaa63256055e91414e set to quiescing
I20250116 02:55:40.423934 9289 tablet_service.cc:1429] Tablet server has 0 leaders and 2 scanners
I20250116 02:55:43.906122 9289 tablet_service.cc:1422] Tablet server 497640d49e7242aaa63256055e91414e set to quiescing
I20250116 02:55:43.906366 9289 tablet_service.cc:1429] Tablet server has 0 leaders and 1 scanners
I20250116 02:55:47.428053 9289 tablet_service.cc:1422] Tablet server 497640d49e7242aaa63256055e91414e set to quiescing
I20250116 02:55:47.428290 9289 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:47.738128 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 9095
I20250116 02:55:47.777117 9218 generic_service.cc:196] Checking for leaks (request via RPC)
I20250116 02:55:47.999794 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 9095
I20250116 02:55:48.030725 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.72:45211
--local_ip_for_outbound_sockets=127.7.228.72
--tserver_master_addrs=127.7.228.126:43731
--webserver_port=33809
--webserver_interface=127.7.228.72
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250116 02:55:48.621479 11004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:55:48.621959 11004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:55:48.622172 11004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:55:48.632810 11004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:55:48.633103 11004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.72
I20250116 02:55:48.646261 11004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.72:45211
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data/info.pb
--webserver_interface=127.7.228.72
--webserver_port=33809
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.72
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:55:48.648341 11004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:55:48.650514 11004 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:55:48.680120 11010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:55:48.683758 11011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:55:48.699393 11013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:55:48.700829 11004 server_base.cc:1034] running on GCE node
I20250116 02:55:48.702015 11004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:55:48.705751 11004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:55:48.707057 11004 hybrid_clock.cc:648] HybridClock initialized: now 1736996148706960 us; error 68 us; skew 500 ppm
I20250116 02:55:48.707809 11004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:55:48.716038 11004 webserver.cc:458] Webserver started at http://127.7.228.72:33809/ using document root <none> and password file <none>
I20250116 02:55:48.717168 11004 fs_manager.cc:362] Metadata directory not provided
I20250116 02:55:48.717370 11004 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:55:48.795873 11004 fs_manager.cc:714] Time spent opening directory manager: real 0.076s user 0.026s sys 0.047s
I20250116 02:55:48.799317 11019 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:55:48.802630 11004 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20250116 02:55:48.802810 11004 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/wal
uuid: "9b5741c935d64e23af0c52c50e78420c"
format_stamp: "Formatted at 2025-01-16 02:54:30 on dist-test-slave-ldbs"
I20250116 02:55:48.803874 11004 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:55:48.917624 11004 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:55:48.918556 11004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:55:48.918901 11004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:55:48.920413 11004 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:55:48.976286 11004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250116 02:55:48.976526 11004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250116 02:55:48.976919 11004 ts_tablet_manager.cc:610] Registered 0 tablets
I20250116 02:55:48.977080 11004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250116 02:55:49.806232 11004 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.72:45211
I20250116 02:55:49.806308 11134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.72:45211 every 8 connection(s)
I20250116 02:55:49.810106 11004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data/info.pb
I20250116 02:55:49.811712 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 11004
I20250116 02:55:49.811904 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 9226
I20250116 02:55:49.943341 9348 generic_service.cc:196] Checking for leaks (request via RPC)
I20250116 02:55:49.947278 11135 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:55:49.947546 11135 heartbeater.cc:463] Registering TS with master...
I20250116 02:55:49.948531 11135 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:49.950299 8119 ts_manager.cc:194] Re-registered known tserver with Master: 9b5741c935d64e23af0c52c50e78420c (127.7.228.72:45211)
I20250116 02:55:49.951314 8119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.72:60549
I20250116 02:55:50.421623 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 9226
W20250116 02:55:50.520030 8588 connection.cc:537] client connection to 127.7.228.73:42313 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250116 02:55:50.520182 8588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 19 similar messages]
I20250116 02:55:50.520735 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.73:42313
--local_ip_for_outbound_sockets=127.7.228.73
--tserver_master_addrs=127.7.228.126:43731
--webserver_port=39105
--webserver_interface=127.7.228.73
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250116 02:55:50.536592 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.73:42313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250116 02:55:50.957262 11135 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
W20250116 02:55:51.049641 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.73:42313: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250116 02:55:51.269765 11145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:55:51.270272 11145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:55:51.270457 11145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:55:51.280428 11145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:55:51.280802 11145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.73
I20250116 02:55:51.298081 11145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.73:42313
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data/info.pb
--webserver_interface=127.7.228.73
--webserver_port=39105
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.73
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:55:51.300284 11145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:55:51.304693 11145 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:55:51.340806 11153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:55:51.341042 11154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:55:51.371353 11145 server_base.cc:1034] running on GCE node
W20250116 02:55:51.375844 11156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:55:51.376881 11145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:55:51.379618 11145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:55:51.380924 11145 hybrid_clock.cc:648] HybridClock initialized: now 1736996151380842 us; error 62 us; skew 500 ppm
I20250116 02:55:51.381484 11145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:55:51.399471 11145 webserver.cc:458] Webserver started at http://127.7.228.73:39105/ using document root <none> and password file <none>
I20250116 02:55:51.400844 11145 fs_manager.cc:362] Metadata directory not provided
I20250116 02:55:51.401094 11145 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:55:51.408701 11145 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250116 02:55:51.420882 11162 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:55:51.422781 11145 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.008s sys 0.001s
I20250116 02:55:51.422992 11145 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/wal
uuid: "497640d49e7242aaa63256055e91414e"
format_stamp: "Formatted at 2025-01-16 02:54:30 on dist-test-slave-ldbs"
I20250116 02:55:51.424249 11145 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:55:51.494715 11145 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:55:51.495893 11145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:55:51.496275 11145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:55:51.503559 11145 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20250116 02:55:51.512113 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.73:42313: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250116 02:55:51.512194 11169 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250116 02:55:51.520215 11145 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250116 02:55:51.520375 11145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.003s sys 0.004s
I20250116 02:55:51.520519 11145 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250116 02:55:51.524305 11145 ts_tablet_manager.cc:610] Registered 1 tablets
I20250116 02:55:51.524415 11145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.004s
I20250116 02:55:51.528605 11169 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Bootstrap starting.
W20250116 02:55:52.058329 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.73:42313: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250116 02:55:52.325775 11145 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.73:42313
I20250116 02:55:52.326092 11278 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.73:42313 every 8 connection(s)
I20250116 02:55:52.329965 11145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data/info.pb
I20250116 02:55:52.337903 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 11145
I20250116 02:55:52.376096 11279 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:55:52.377611 11279 heartbeater.cc:463] Registering TS with master...
I20250116 02:55:52.378672 11279 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:55:52.397794 8116 ts_manager.cc:194] Re-registered known tserver with Master: 497640d49e7242aaa63256055e91414e (127.7.228.73:42313)
I20250116 02:55:52.399520 8116 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.73:43871
W20250116 02:55:52.562568 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
W20250116 02:55:53.192535 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
I20250116 02:55:53.222718 11169 log.cc:826] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Log is configured to *not* fsync() on all Append() calls
I20250116 02:55:53.416910 11279 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
W20250116 02:55:53.747246 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250116 02:55:53.885517 11196 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 497640d49e7242aaa63256055e91414e is lagging by at least 3 ops behind the committed index
I20250116 02:55:53.917938 8634 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250116 02:55:53.956120 11214 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:54.032764 10394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:54.035249 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:54.082304 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:54.181046 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:54.181761 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:54.182925 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:55:54.207190 11070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:55:54.405407 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
W20250116 02:55:54.939257 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20250116 02:55:55.470175 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250116 02:55:55.977520 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
W20250116 02:55:56.475766 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
I20250116 02:55:56.969652 11332 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 497640d49e7242aaa63256055e91414e is lagging by at least 335 ops behind the committed index [suppressed 27 similar messages]
W20250116 02:55:57.019680 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
W20250116 02:55:57.618899 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
W20250116 02:55:58.132360 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
I20250116 02:55:58.231369 11169 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Bootstrap replaying log segment 1/2 (1.71M/8.00M this segment, stats: ops{read=1009 overwritten=0 applied=1008 ignored=0} inserts{seen=50350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:55:58.758913 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
W20250116 02:55:59.276085 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250116 02:55:59.777026 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
I20250116 02:56:00.024627 11344 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 497640d49e7242aaa63256055e91414e is lagging by at least 692 ops behind the committed index [suppressed 27 similar messages]
W20250116 02:56:00.287323 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
W20250116 02:56:00.994187 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
I20250116 02:56:01.182332 8634 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250116 02:56:01.230069 10394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:01.239481 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:01.275764 11214 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:01.295307 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:01.409287 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:01.410715 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:01.443984 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:01.529349 11070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:56:01.615958 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
W20250116 02:56:02.109225 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20250116 02:56:02.630357 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
I20250116 02:56:03.071326 11333 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 497640d49e7242aaa63256055e91414e is lagging by at least 1002 ops behind the committed index [suppressed 25 similar messages]
W20250116 02:56:03.223287 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
I20250116 02:56:03.232985 11169 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Bootstrap replaying log segment 1/2 (3.43M/8.00M this segment, stats: ops{read=2018 overwritten=0 applied=2017 ignored=0} inserts{seen=100800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:56:03.706804 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
W20250116 02:56:04.286904 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
W20250116 02:56:04.798025 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
W20250116 02:56:05.365715 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
W20250116 02:56:05.898005 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
I20250116 02:56:06.092152 11403 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 497640d49e7242aaa63256055e91414e is lagging by at least 1343 ops behind the committed index [suppressed 27 similar messages]
W20250116 02:56:06.519716 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
W20250116 02:56:06.994463 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
W20250116 02:56:07.526434 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 156: this message will repeat every 5th retry.
W20250116 02:56:08.017781 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 161: this message will repeat every 5th retry.
I20250116 02:56:08.233366 11169 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Bootstrap replaying log segment 1/2 (5.60M/8.00M this segment, stats: ops{read=3295 overwritten=0 applied=3295 ignored=0} inserts{seen=164700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:56:08.640797 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 166: this message will repeat every 5th retry.
I20250116 02:56:08.671399 10394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:08.672129 8634 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250116 02:56:08.679054 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:08.765830 11214 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:08.789798 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:08.828069 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:08.846014 11070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:08.856628 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:08.899252 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:56:08.913599 9392 meta_cache.cc:1261] Time spent looking up entry by key: real 0.055s user 0.000s sys 0.000s
I20250116 02:56:09.184841 11346 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 497640d49e7242aaa63256055e91414e is lagging by at least 1613 ops behind the committed index [suppressed 26 similar messages]
W20250116 02:56:09.351864 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 171: this message will repeat every 5th retry.
W20250116 02:56:09.883505 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 176: this message will repeat every 5th retry.
W20250116 02:56:10.397814 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 181: this message will repeat every 5th retry.
W20250116 02:56:10.931588 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 186: this message will repeat every 5th retry.
W20250116 02:56:11.434513 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 191: this message will repeat every 5th retry.
W20250116 02:56:11.969045 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 196: this message will repeat every 5th retry.
I20250116 02:56:12.259151 11403 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 497640d49e7242aaa63256055e91414e is lagging by at least 1983 ops behind the committed index [suppressed 28 similar messages]
W20250116 02:56:12.469587 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 201: this message will repeat every 5th retry.
W20250116 02:56:13.061059 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 206: this message will repeat every 5th retry.
I20250116 02:56:13.236071 11169 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Bootstrap replaying log segment 1/2 (7.31M/8.00M this segment, stats: ops{read=4296 overwritten=0 applied=4295 ignored=0} inserts{seen=214700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:56:13.560902 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 211: this message will repeat every 5th retry.
W20250116 02:56:14.121754 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 216: this message will repeat every 5th retry.
W20250116 02:56:14.689136 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 221: this message will repeat every 5th retry.
I20250116 02:56:14.878904 11169 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Bootstrap replayed 1/2 log segments. Stats: ops{read=4699 overwritten=0 applied=4698 ignored=0} inserts{seen=234850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20250116 02:56:15.222219 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 226: this message will repeat every 5th retry.
I20250116 02:56:15.262457 11465 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 497640d49e7242aaa63256055e91414e is lagging by at least 2298 ops behind the committed index [suppressed 27 similar messages]
W20250116 02:56:15.804828 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 231: this message will repeat every 5th retry.
I20250116 02:56:15.965241 8634 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250116 02:56:15.996336 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:16.054942 11214 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:16.064291 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:16.072336 10394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:16.098309 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:16.101866 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:16.113102 11070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:16.132541 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:56:16.368716 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 236: this message will repeat every 5th retry.
W20250116 02:56:16.826603 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 241: this message will repeat every 5th retry.
W20250116 02:56:17.352389 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 246: this message will repeat every 5th retry.
W20250116 02:56:17.811725 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 251: this message will repeat every 5th retry.
I20250116 02:56:18.238582 11169 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Bootstrap replaying log segment 2/2 (1.14M/3.63M this segment, stats: ops{read=5367 overwritten=0 applied=5367 ignored=0} inserts{seen=268300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250116 02:56:18.322540 11471 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 497640d49e7242aaa63256055e91414e is lagging by at least 2608 ops behind the committed index [suppressed 28 similar messages]
W20250116 02:56:18.352792 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 256: this message will repeat every 5th retry.
W20250116 02:56:18.917389 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 261: this message will repeat every 5th retry.
W20250116 02:56:19.449940 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 266: this message will repeat every 5th retry.
W20250116 02:56:19.995981 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 271: this message will repeat every 5th retry.
W20250116 02:56:20.589260 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 276: this message will repeat every 5th retry.
W20250116 02:56:21.072945 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 281: this message will repeat every 5th retry.
I20250116 02:56:21.344201 11523 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 497640d49e7242aaa63256055e91414e is lagging by at least 2951 ops behind the committed index [suppressed 27 similar messages]
W20250116 02:56:21.674535 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 286: this message will repeat every 5th retry.
W20250116 02:56:22.195488 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 291: this message will repeat every 5th retry.
W20250116 02:56:22.753436 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 296: this message will repeat every 5th retry.
I20250116 02:56:22.921939 8634 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250116 02:56:22.962162 10394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:22.976035 11070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:22.985697 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:22.987329 11214 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:23.009508 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:23.012339 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:23.014070 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:23.067484 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:23.246441 11169 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Bootstrap replaying log segment 2/2 (2.88M/3.63M this segment, stats: ops{read=6388 overwritten=0 applied=6388 ignored=0} inserts{seen=319350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:56:23.408574 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 301: this message will repeat every 5th retry.
W20250116 02:56:23.993629 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 306: this message will repeat every 5th retry.
I20250116 02:56:24.408869 11471 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 497640d49e7242aaa63256055e91414e is lagging by at least 3265 ops behind the committed index [suppressed 26 similar messages]
W20250116 02:56:24.441210 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 311: this message will repeat every 5th retry.
W20250116 02:56:24.981107 8588 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda -> Peer 497640d49e7242aaa63256055e91414e (127.7.228.73:42313): Couldn't send request to peer 497640d49e7242aaa63256055e91414e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 316: this message will repeat every 5th retry.
I20250116 02:56:25.370110 11169 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Bootstrap replayed 2/2 log segments. Stats: ops{read=6827 overwritten=0 applied=6827 ignored=0} inserts{seen=341300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250116 02:56:25.370730 11169 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Bootstrap complete.
I20250116 02:56:25.374358 11169 ts_tablet_manager.cc:1397] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Time spent bootstrapping tablet: real 33.846s user 25.331s sys 0.816s
I20250116 02:56:25.380414 11169 raft_consensus.cc:357] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:25.381548 11169 raft_consensus.cc:738] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 497640d49e7242aaa63256055e91414e, State: Initialized, Role: FOLLOWER
I20250116 02:56:25.382308 11169 consensus_queue.cc:260] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6827, Last appended: 1.6827, Last appended by leader: 6827, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:25.389012 11169 ts_tablet_manager.cc:1428] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20250116 02:56:25.808485 11586 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0791364971ba49f3b2813bdc36c11eda)
I20250116 02:56:25.808735 11586 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:25.811136 11586 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805), 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:56:25.865895 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 7252 } } ignore_live_leader: false dest_uuid: "0791364971ba49f3b2813bdc36c11eda" is_pre_election: true
I20250116 02:56:25.876724 10413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 7252 } } ignore_live_leader: false dest_uuid: "429bf4133bf54797b117ef8fad632920" is_pre_election: true
I20250116 02:56:25.877581 11163 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 497640d49e7242aaa63256055e91414e; no voters: 0791364971ba49f3b2813bdc36c11eda, 429bf4133bf54797b117ef8fad632920
I20250116 02:56:26.305147 11586 raft_consensus.cc:2743] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:56:26.793088 11586 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0791364971ba49f3b2813bdc36c11eda)
I20250116 02:56:26.793288 11586 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:26.794095 11586 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805), 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:56:26.795084 10413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8102 } } ignore_live_leader: false dest_uuid: "429bf4133bf54797b117ef8fad632920" is_pre_election: true
I20250116 02:56:26.796021 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8102 } } ignore_live_leader: false dest_uuid: "0791364971ba49f3b2813bdc36c11eda" is_pre_election: true
I20250116 02:56:26.796504 11166 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 497640d49e7242aaa63256055e91414e; no voters: 0791364971ba49f3b2813bdc36c11eda, 429bf4133bf54797b117ef8fad632920
I20250116 02:56:26.796895 11586 raft_consensus.cc:2743] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:56:27.179775 11586 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0791364971ba49f3b2813bdc36c11eda)
I20250116 02:56:27.179944 11586 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:27.180627 11586 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805), 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:56:27.181257 10413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8527 } } ignore_live_leader: false dest_uuid: "429bf4133bf54797b117ef8fad632920" is_pre_election: true
I20250116 02:56:27.181744 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8527 } } ignore_live_leader: false dest_uuid: "0791364971ba49f3b2813bdc36c11eda" is_pre_election: true
I20250116 02:56:27.182346 11166 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 497640d49e7242aaa63256055e91414e; no voters: 0791364971ba49f3b2813bdc36c11eda, 429bf4133bf54797b117ef8fad632920
I20250116 02:56:27.186717 11586 raft_consensus.cc:2743] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:56:27.464753 8109 maintenance_manager.cc:418] P 56ddececa4d0424dae4d75d6a003686b: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033390
I20250116 02:56:27.473634 11600 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [LEADER]: Peer 497640d49e7242aaa63256055e91414e is lagging by at least 1453 ops behind the committed index [suppressed 13 similar messages]
I20250116 02:56:27.511735 11586 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0791364971ba49f3b2813bdc36c11eda)
I20250116 02:56:27.512014 11586 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:27.512787 11586 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805), 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:56:27.513690 10413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8952 } } ignore_live_leader: false dest_uuid: "429bf4133bf54797b117ef8fad632920" is_pre_election: true
I20250116 02:56:27.517845 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 8952 } } ignore_live_leader: false dest_uuid: "0791364971ba49f3b2813bdc36c11eda" is_pre_election: true
I20250116 02:56:27.518399 11166 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 497640d49e7242aaa63256055e91414e; no voters: 0791364971ba49f3b2813bdc36c11eda, 429bf4133bf54797b117ef8fad632920
I20250116 02:56:27.588181 8108 maintenance_manager.cc:642] P 56ddececa4d0424dae4d75d6a003686b: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.122s user 0.028s sys 0.046s Metrics: {"bytes_written":7241,"cfile_init":1,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4160,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":13,"thread_start_us":58593,"threads_started":1,"wal-append.queue_time_us":64440}
I20250116 02:56:27.594462 8109 maintenance_manager.cc:418] P 56ddececa4d0424dae4d75d6a003686b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 583 bytes on disk
I20250116 02:56:27.599834 8108 maintenance_manager.cc:642] P 56ddececa4d0424dae4d75d6a003686b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20250116 02:56:27.742419 11586 raft_consensus.cc:2743] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:56:28.225626 11586 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0791364971ba49f3b2813bdc36c11eda)
I20250116 02:56:28.225821 11586 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:28.226591 11586 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805), 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:56:28.226899 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 9802 } } ignore_live_leader: false dest_uuid: "0791364971ba49f3b2813bdc36c11eda" is_pre_election: true
I20250116 02:56:28.227753 10414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 9802 } } ignore_live_leader: false dest_uuid: "429bf4133bf54797b117ef8fad632920" is_pre_election: true
I20250116 02:56:28.228295 11163 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 497640d49e7242aaa63256055e91414e; no voters: 0791364971ba49f3b2813bdc36c11eda, 429bf4133bf54797b117ef8fad632920
I20250116 02:56:28.228755 11586 raft_consensus.cc:2743] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:56:28.880517 11586 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0791364971ba49f3b2813bdc36c11eda)
I20250116 02:56:28.880692 11586 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:28.882450 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 10227 } } ignore_live_leader: false dest_uuid: "0791364971ba49f3b2813bdc36c11eda" is_pre_election: true
I20250116 02:56:28.881503 11586 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805), 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:56:28.891294 10414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 10227 } } ignore_live_leader: false dest_uuid: "429bf4133bf54797b117ef8fad632920" is_pre_election: true
I20250116 02:56:28.892325 11163 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 497640d49e7242aaa63256055e91414e; no voters: 0791364971ba49f3b2813bdc36c11eda, 429bf4133bf54797b117ef8fad632920
I20250116 02:56:28.892782 11586 raft_consensus.cc:2743] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:56:29.653138 11586 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0791364971ba49f3b2813bdc36c11eda)
I20250116 02:56:29.653317 11586 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:29.654490 11586 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805), 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:56:29.665689 10413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 10652 } } ignore_live_leader: false dest_uuid: "429bf4133bf54797b117ef8fad632920" is_pre_election: true
I20250116 02:56:29.671384 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 10652 } } ignore_live_leader: false dest_uuid: "0791364971ba49f3b2813bdc36c11eda" is_pre_election: true
I20250116 02:56:29.672194 11166 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 497640d49e7242aaa63256055e91414e; no voters: 0791364971ba49f3b2813bdc36c11eda, 429bf4133bf54797b117ef8fad632920
I20250116 02:56:29.672499 11586 raft_consensus.cc:2743] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:56:30.272675 11586 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0791364971ba49f3b2813bdc36c11eda)
I20250116 02:56:30.272881 11586 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:30.273686 11586 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805), 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:56:30.274266 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 11001 } } ignore_live_leader: false dest_uuid: "0791364971ba49f3b2813bdc36c11eda" is_pre_election: true
I20250116 02:56:30.274613 10413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 11001 } } ignore_live_leader: false dest_uuid: "429bf4133bf54797b117ef8fad632920" is_pre_election: true
I20250116 02:56:30.276264 11163 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 497640d49e7242aaa63256055e91414e; no voters: 0791364971ba49f3b2813bdc36c11eda, 429bf4133bf54797b117ef8fad632920
I20250116 02:56:30.276800 11586 raft_consensus.cc:2743] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250116 02:56:30.854696 8634 tablet_service.cc:1429] Tablet server has 1 leaders and 2 scanners
I20250116 02:56:30.885406 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:30.929023 10394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:30.957037 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:30.969985 11070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:30.976670 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:30.995478 11214 tablet_service.cc:1429] Tablet server has 0 leaders and 1 scanners
I20250116 02:56:31.017342 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:31.044997 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:31.184587 8700 maintenance_manager.cc:418] P 0791364971ba49f3b2813bdc36c11eda: Scheduling FlushMRSOp(29c42185024a48cda0403d61962cf176): perf score=0.057129
W20250116 02:56:32.975489 9366 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.7.228.68:41593, user_credentials={real_user=slave}} blocked reactor thread for 70481.1us
I20250116 02:56:33.715566 8116 ts_manager.cc:281] Unset tserver state for 9b5741c935d64e23af0c52c50e78420c from MAINTENANCE_MODE
I20250116 02:56:33.717764 9925 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:33.896633 8699 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:34.099805 9651 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:34.100600 11135 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:34.254685 11279 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:34.445955 10601 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:34.469363 10316 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:34.469369 9787 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:34.567821 10461 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:35.989101 8119 ts_manager.cc:281] Unset tserver state for 497640d49e7242aaa63256055e91414e from MAINTENANCE_MODE
I20250116 02:56:36.113077 11135 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:36.114477 9651 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:36.260880 11279 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:36.453783 10601 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:36.474138 10316 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:36.474208 9787 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:36.573583 10461 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:36.730242 9925 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:36.911860 8699 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:39.111316 8081 maintenance_mode-itest.cc:745] Restarting batch of 1 tservers: 0791364971ba49f3b2813bdc36c11eda
I20250116 02:56:40.304471 8119 ts_manager.cc:292] Set tserver state for 0791364971ba49f3b2813bdc36c11eda to MAINTENANCE_MODE
W20250116 02:56:43.290283 8587 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.7.228.66:41805, user_credentials={real_user=slave}} blocked reactor thread for 58345.5us
I20250116 02:56:44.607622 8634 tablet_service.cc:1422] Tablet server 0791364971ba49f3b2813bdc36c11eda set to quiescing
I20250116 02:56:44.607860 8634 tablet_service.cc:1429] Tablet server has 1 leaders and 2 scanners
I20250116 02:56:44.612282 11745 raft_consensus.cc:988] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: : Instructing follower 429bf4133bf54797b117ef8fad632920 to start an election
I20250116 02:56:44.612412 11745 raft_consensus.cc:1076] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 1 LEADER]: Signalling peer 429bf4133bf54797b117ef8fad632920 to start an election
I20250116 02:56:44.621439 11746 raft_consensus.cc:988] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: : Instructing follower 429bf4133bf54797b117ef8fad632920 to start an election
I20250116 02:56:44.621596 11746 raft_consensus.cc:1076] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 1 LEADER]: Signalling peer 429bf4133bf54797b117ef8fad632920 to start an election
I20250116 02:56:44.621804 11746 raft_consensus.cc:988] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: : Instructing follower 497640d49e7242aaa63256055e91414e to start an election
I20250116 02:56:44.621870 11746 raft_consensus.cc:1076] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 1 LEADER]: Signalling peer 497640d49e7242aaa63256055e91414e to start an election
I20250116 02:56:44.622443 11234 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "29c42185024a48cda0403d61962cf176"
dest_uuid: "497640d49e7242aaa63256055e91414e"
from {username='slave'} at 127.7.228.68:40187
I20250116 02:56:44.622686 11234 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:56:44.622788 11234 raft_consensus.cc:3054] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:56:44.622798 10414 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "29c42185024a48cda0403d61962cf176"
dest_uuid: "429bf4133bf54797b117ef8fad632920"
from {username='slave'} at 127.7.228.68:35049
I20250116 02:56:44.623018 10414 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:56:44.623145 10414 raft_consensus.cc:3054] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:56:44.627408 11234 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:44.627429 10414 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:44.628810 10413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12080 } } ignore_live_leader: true dest_uuid: "429bf4133bf54797b117ef8fad632920"
I20250116 02:56:44.629128 10413 raft_consensus.cc:2388] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 497640d49e7242aaa63256055e91414e in current term 2: Already voted for candidate 429bf4133bf54797b117ef8fad632920 in this term.
I20250116 02:56:44.632769 10413 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "29c42185024a48cda0403d61962cf176"
dest_uuid: "429bf4133bf54797b117ef8fad632920"
from {username='slave'} at 127.7.228.68:35049
I20250116 02:56:44.632938 10413 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250116 02:56:44.633047 10413 raft_consensus.cc:3054] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:56:44.635522 11234 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 election: Requested vote from peers 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805), 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:56:44.636165 10413 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:44.649587 10413 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [CANDIDATE]: Term 3 election: Requested vote from peers 497640d49e7242aaa63256055e91414e (127.7.228.73:42313), 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:56:44.650512 10414 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [CANDIDATE]: Term 2 election: Requested vote from peers 497640d49e7242aaa63256055e91414e (127.7.228.73:42313), 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:56:44.656962 8653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "429bf4133bf54797b117ef8fad632920" candidate_term: 2 candidate_status { last_received { term: 1 index: 12080 } } ignore_live_leader: true dest_uuid: "0791364971ba49f3b2813bdc36c11eda"
I20250116 02:56:44.657202 8653 raft_consensus.cc:3049] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 1 LEADER]: Stepping down as leader of term 1
I20250116 02:56:44.657279 8653 raft_consensus.cc:738] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0791364971ba49f3b2813bdc36c11eda, State: Running, Role: LEADER
I20250116 02:56:44.656985 8654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "429bf4133bf54797b117ef8fad632920" candidate_term: 3 candidate_status { last_received { term: 1 index: 12080 } } ignore_live_leader: true dest_uuid: "0791364971ba49f3b2813bdc36c11eda"
I20250116 02:56:44.658207 8653 consensus_queue.cc:260] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12080, Committed index: 12080, Last appended: 1.12080, Last appended by leader: 12080, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:44.658481 8653 logging.cc:424] LogThrottler TrackedPeer Lag: suppressed but not reported on 3 messages since previous log ~81 seconds ago
I20250116 02:56:44.658556 8653 logging.cc:424] LogThrottler TrackedPeer Lag: suppressed but not reported on 3 messages since previous log ~17 seconds ago
I20250116 02:56:44.658891 8653 raft_consensus.cc:3054] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 1 FOLLOWER]: Advancing to term 2
I20250116 02:56:44.662086 8653 raft_consensus.cc:2463] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 429bf4133bf54797b117ef8fad632920 in term 2.
I20250116 02:56:44.662611 10349 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0791364971ba49f3b2813bdc36c11eda, 429bf4133bf54797b117ef8fad632920; no voters:
W20250116 02:56:44.669744 8614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37300: Illegal state: replica 0791364971ba49f3b2813bdc36c11eda is not leader of this config: current role FOLLOWER
W20250116 02:56:44.670128 9404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37300: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.037s ago.
W20250116 02:56:44.671496 9404 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37300: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.050s ago.
W20250116 02:56:44.677716 11192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53560: Illegal state: replica 497640d49e7242aaa63256055e91414e is not leader of this config: current role FOLLOWER
W20250116 02:56:44.679252 11190 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53560: Illegal state: replica 497640d49e7242aaa63256055e91414e is not leader of this config: current role FOLLOWER
I20250116 02:56:44.681797 11762 raft_consensus.cc:2758] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
W20250116 02:56:44.699158 11192 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53560: Illegal state: replica 497640d49e7242aaa63256055e91414e is not leader of this config: current role FOLLOWER
W20250116 02:56:44.721316 8608 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37300: Illegal state: replica 0791364971ba49f3b2813bdc36c11eda is not leader of this config: current role FOLLOWER
W20250116 02:56:44.722139 8610 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37300: Illegal state: replica 0791364971ba49f3b2813bdc36c11eda is not leader of this config: current role FOLLOWER
I20250116 02:56:44.725898 8653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12080 } } ignore_live_leader: true dest_uuid: "0791364971ba49f3b2813bdc36c11eda"
I20250116 02:56:44.726234 8653 raft_consensus.cc:2388] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 497640d49e7242aaa63256055e91414e in current term 2: Already voted for candidate 429bf4133bf54797b117ef8fad632920 in this term.
I20250116 02:56:44.726707 11166 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 497640d49e7242aaa63256055e91414e; no voters: 0791364971ba49f3b2813bdc36c11eda, 429bf4133bf54797b117ef8fad632920
W20250116 02:56:44.722102 8614 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37300: Illegal state: replica 0791364971ba49f3b2813bdc36c11eda is not leader of this config: current role FOLLOWER
W20250116 02:56:44.736789 11190 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53560: Illegal state: replica 497640d49e7242aaa63256055e91414e is not leader of this config: current role FOLLOWER
I20250116 02:56:44.737936 11765 raft_consensus.cc:2743] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250116 02:56:44.742570 11182 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53560: Illegal state: replica 497640d49e7242aaa63256055e91414e is not leader of this config: current role FOLLOWER
I20250116 02:56:44.744752 11234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "429bf4133bf54797b117ef8fad632920" candidate_term: 2 candidate_status { last_received { term: 1 index: 12080 } } ignore_live_leader: true dest_uuid: "497640d49e7242aaa63256055e91414e"
I20250116 02:56:44.745136 11234 raft_consensus.cc:2388] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 429bf4133bf54797b117ef8fad632920 in current term 2: Already voted for candidate 497640d49e7242aaa63256055e91414e in this term.
I20250116 02:56:44.745545 11233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "429bf4133bf54797b117ef8fad632920" candidate_term: 3 candidate_status { last_received { term: 1 index: 12080 } } ignore_live_leader: true dest_uuid: "497640d49e7242aaa63256055e91414e"
I20250116 02:56:44.745723 11233 raft_consensus.cc:3054] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 2 FOLLOWER]: Advancing to term 3
W20250116 02:56:44.747190 11182 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53560: Illegal state: replica 497640d49e7242aaa63256055e91414e is not leader of this config: current role FOLLOWER
I20250116 02:56:44.748621 11233 raft_consensus.cc:2463] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 429bf4133bf54797b117ef8fad632920 in term 3.
I20250116 02:56:44.750417 10347 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 429bf4133bf54797b117ef8fad632920, 497640d49e7242aaa63256055e91414e; no voters: 0791364971ba49f3b2813bdc36c11eda
I20250116 02:56:44.750878 11762 raft_consensus.cc:2798] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 3 FOLLOWER]: Leader election won for term 3
I20250116 02:56:44.762059 11762 raft_consensus.cc:695] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [term 3 LEADER]: Becoming Leader. State: Replica: 429bf4133bf54797b117ef8fad632920, State: Running, Role: LEADER
I20250116 02:56:44.762626 11762 consensus_queue.cc:237] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12080, Committed index: 12080, Last appended: 1.12080, Last appended by leader: 12080, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:56:44.767340 8116 catalog_manager.cc:5526] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 reported cstate change: term changed from 1 to 3, leader changed from 0791364971ba49f3b2813bdc36c11eda (127.7.228.68) to 429bf4133bf54797b117ef8fad632920 (127.7.228.66). New cstate: current_term: 3 leader_uuid: "429bf4133bf54797b117ef8fad632920" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } health_report { overall_health: UNKNOWN } } }
I20250116 02:56:44.813335 8653 raft_consensus.cc:3054] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 2 FOLLOWER]: Advancing to term 3
I20250116 02:56:44.813383 11233 raft_consensus.cc:1270] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 3 FOLLOWER]: Refusing update from remote peer 429bf4133bf54797b117ef8fad632920: Log matching property violated. Preceding OpId in replica: term: 1 index: 12080. Preceding OpId from leader: term: 3 index: 12082. (index mismatch)
I20250116 02:56:44.814042 11762 consensus_queue.cc:1035] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12081, Last known committed idx: 12080, Time since last communication: 0.000s
I20250116 02:56:44.816704 8653 raft_consensus.cc:1270] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 3 FOLLOWER]: Refusing update from remote peer 429bf4133bf54797b117ef8fad632920: Log matching property violated. Preceding OpId in replica: term: 1 index: 12080. Preceding OpId from leader: term: 3 index: 12082. (index mismatch)
I20250116 02:56:44.818823 11762 consensus_queue.cc:1035] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12081, Last known committed idx: 12080, Time since last communication: 0.000s
I20250116 02:56:44.843326 10717 mvcc.cc:204] Tried to move back new op lower bound from 7114736454909620224 to 7114736454712754176. Current Snapshot: MvccSnapshot[applied={T|T < 7114736454909620224 or (T in {7114736454909620224})}]
I20250116 02:56:48.850247 8634 tablet_service.cc:1422] Tablet server 0791364971ba49f3b2813bdc36c11eda set to quiescing
I20250116 02:56:48.850484 8634 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:56:49.899829 9392 meta_cache.cc:1261] Time spent looking up entry by key: real 0.063s user 0.003s sys 0.000s
I20250116 02:56:50.762807 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 8571
I20250116 02:56:50.807353 8694 generic_service.cc:196] Checking for leaks (request via RPC)
W20250116 02:56:51.642858 8694 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:39170 (request call id 0) took 835 ms (client timeout 1000 ms). Trace:
W20250116 02:56:51.643139 8694 rpcz_store.cc:269] 0116 02:56:50.807156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0116 02:56:50.807209 (+ 53us) service_pool.cc:225] Handling call
0116 02:56:51.642817 (+835608us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250116 02:56:51.674135 11818 raft_consensus.cc:668] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: failed to trigger leader election: Illegal state: leader elections are disabled
I20250116 02:56:51.674257 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 8571
I20250116 02:56:51.780149 11587 mvcc.cc:204] Tried to move back new op lower bound from 7114736454909620224 to 7114736454712754176. Current Snapshot: MvccSnapshot[applied={T|T < 7114736454909620224 or (T in {7114736454909620224})}]
W20250116 02:56:51.868438 10349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.7.228.68:41593 (error 108)
I20250116 02:56:51.870882 8081 external_mini_cluster.cc:1351] Running /tmp/dist-test-task8SxoCU/build/asan/bin/kudu
/tmp/dist-test-task8SxoCU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.7.228.68:41593
--local_ip_for_outbound_sockets=127.7.228.68
--tserver_master_addrs=127.7.228.126:43731
--webserver_port=44091
--webserver_interface=127.7.228.68
--builtin_ntp_servers=127.7.228.84:36635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250116 02:56:51.896381 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.68:41593: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250116 02:56:52.368629 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.68:41593: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250116 02:56:52.466565 11823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:56:52.467036 11823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250116 02:56:52.467221 11823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:56:52.479667 11823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250116 02:56:52.480033 11823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.7.228.68
I20250116 02:56:52.499224 11823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.7.228.84:36635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.7.228.68:41593
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.7.228.68
--webserver_port=44091
--tserver_master_addrs=127.7.228.126:43731
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.7.228.68
--log_dir=/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 16 Jan 2025 02:43:24 UTC on 24a791456cd2
build id 4199
ASAN enabled
I20250116 02:56:52.501507 11823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250116 02:56:52.503860 11823 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250116 02:56:52.536607 11831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250116 02:56:52.537814 11830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:56:52.547869 11823 server_base.cc:1034] running on GCE node
W20250116 02:56:52.550221 11833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250116 02:56:52.551115 11823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250116 02:56:52.590720 11823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250116 02:56:52.601141 11823 hybrid_clock.cc:648] HybridClock initialized: now 1736996212601951 us; error 1926 us; skew 500 ppm
I20250116 02:56:52.601961 11823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250116 02:56:52.634347 11823 webserver.cc:458] Webserver started at http://127.7.228.68:44091/ using document root <none> and password file <none>
I20250116 02:56:52.639725 11823 fs_manager.cc:362] Metadata directory not provided
I20250116 02:56:52.640002 11823 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250116 02:56:52.675928 11823 fs_manager.cc:714] Time spent opening directory manager: real 0.033s user 0.010s sys 0.016s
I20250116 02:56:52.736493 11842 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250116 02:56:52.738366 11823 fs_manager.cc:730] Time spent opening block manager: real 0.060s user 0.003s sys 0.000s
I20250116 02:56:52.738662 11823 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data,/tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/wal
uuid: "0791364971ba49f3b2813bdc36c11eda"
format_stamp: "Formatted at 2025-01-16 02:54:28 on dist-test-slave-ldbs"
I20250116 02:56:52.739982 11823 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250116 02:56:52.809075 11823 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250116 02:56:52.810192 11823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250116 02:56:52.810652 11823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250116 02:56:52.814357 11823 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250116 02:56:52.838976 11850 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250116 02:56:52.847362 11823 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250116 02:56:52.847599 11823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.003s
I20250116 02:56:52.847816 11823 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250116 02:56:52.852562 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap starting.
W20250116 02:56:52.865887 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.68:41593: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250116 02:56:52.874998 11823 ts_tablet_manager.cc:610] Registered 1 tablets
I20250116 02:56:52.875150 11823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.027s user 0.002s sys 0.002s
W20250116 02:56:53.432560 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.68:41593: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250116 02:56:54.149969 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.68:41593: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250116 02:56:54.402185 11823 rpc_server.cc:307] RPC server started. Bound to: 127.7.228.68:41593
I20250116 02:56:54.402261 11961 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.7.228.68:41593 every 8 connection(s)
I20250116 02:56:54.404255 11823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data/info.pb
I20250116 02:56:54.411885 8081 external_mini_cluster.cc:1413] Started /tmp/dist-test-task8SxoCU/build/asan/bin/kudu as pid 11823
I20250116 02:56:54.433728 11962 heartbeater.cc:346] Connected to a master server at 127.7.228.126:43731
I20250116 02:56:54.434119 11962 heartbeater.cc:463] Registering TS with master...
I20250116 02:56:54.435341 11962 heartbeater.cc:509] Master 127.7.228.126:43731 requested a full tablet report, sending...
I20250116 02:56:54.438359 8119 ts_manager.cc:194] Re-registered known tserver with Master: 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:56:54.439884 8119 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.7.228.68:38421
W20250116 02:56:54.666970 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
W20250116 02:56:55.234015 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250116 02:56:55.432430 11850 log.cc:826] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Log is configured to *not* fsync() on all Append() calls
I20250116 02:56:55.443094 11962 heartbeater.cc:501] Master 127.7.228.126:43731 was elected leader, sending a full tablet report...
I20250116 02:56:55.643275 11899 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 12 ops behind the committed index
W20250116 02:56:55.936441 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20250116 02:56:55.982836 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:56.016404 11894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:56.025295 10394 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250116 02:56:56.054418 11214 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:56.126534 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:56.157092 11070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:56.171815 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:56.178704 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:56:56.228492 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:56:56.704700 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20250116 02:56:57.396543 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250116 02:56:57.985020 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
W20250116 02:56:58.162667 9363 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.7.228.66:41805, user_credentials={real_user=slave}} blocked reactor thread for 51826.3us
W20250116 02:56:58.453732 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
I20250116 02:56:58.699837 12016 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 301 ops behind the committed index [suppressed 24 similar messages]
W20250116 02:56:59.009897 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
W20250116 02:56:59.554962 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
W20250116 02:57:00.139323 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
I20250116 02:57:00.443678 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replaying log segment 1/3 (1.56M/8.00M this segment, stats: ops{read=903 overwritten=0 applied=903 ignored=0} inserts{seen=45100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:57:00.608180 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
W20250116 02:57:01.134078 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250116 02:57:01.432591 9363 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.7.228.66:41805, user_credentials={real_user=slave}} blocked reactor thread for 52816.1us
W20250116 02:57:01.675509 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
I20250116 02:57:01.700877 12021 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 618 ops behind the committed index [suppressed 27 similar messages]
W20250116 02:57:02.101293 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
W20250116 02:57:02.717851 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
W20250116 02:57:03.278057 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
I20250116 02:57:03.355026 11894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:03.389488 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:03.406301 10394 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250116 02:57:03.428901 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:03.450802 11214 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:03.556226 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:03.580144 11070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:03.593688 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:03.620029 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:57:04.064428 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20250116 02:57:04.524489 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
I20250116 02:57:04.752501 11968 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 895 ops behind the committed index [suppressed 25 similar messages]
W20250116 02:57:05.103789 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
I20250116 02:57:05.444718 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replaying log segment 1/3 (3.20M/8.00M this segment, stats: ops{read=1849 overwritten=0 applied=1849 ignored=0} inserts{seen=92400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:57:05.685814 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
W20250116 02:57:06.193581 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
W20250116 02:57:06.714701 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
W20250116 02:57:07.201234 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
W20250116 02:57:07.734632 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
I20250116 02:57:07.834452 12025 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 1227 ops behind the committed index [suppressed 28 similar messages]
W20250116 02:57:08.303463 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
W20250116 02:57:08.897116 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
W20250116 02:57:09.501829 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 156: this message will repeat every 5th retry.
W20250116 02:57:10.163277 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 161: this message will repeat every 5th retry.
I20250116 02:57:10.381073 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:10.406888 11894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:10.414570 10394 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250116 02:57:10.415863 11214 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:10.448165 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replaying log segment 1/3 (5.31M/8.00M this segment, stats: ops{read=3069 overwritten=0 applied=3069 ignored=0} inserts{seen=153400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250116 02:57:10.489733 11070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:10.515166 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:10.560627 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:10.579473 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:10.613047 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:57:10.860061 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 166: this message will repeat every 5th retry.
I20250116 02:57:10.914922 12092 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 1469 ops behind the committed index [suppressed 24 similar messages]
W20250116 02:57:11.411613 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 171: this message will repeat every 5th retry.
W20250116 02:57:12.025081 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 176: this message will repeat every 5th retry.
W20250116 02:57:12.515215 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 181: this message will repeat every 5th retry.
W20250116 02:57:13.060231 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 186: this message will repeat every 5th retry.
W20250116 02:57:13.580603 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 191: this message will repeat every 5th retry.
I20250116 02:57:13.965875 12025 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 1810 ops behind the committed index [suppressed 27 similar messages]
W20250116 02:57:14.106911 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 196: this message will repeat every 5th retry.
W20250116 02:57:14.693449 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 201: this message will repeat every 5th retry.
W20250116 02:57:15.204694 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 206: this message will repeat every 5th retry.
I20250116 02:57:15.453708 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replaying log segment 1/3 (7.12M/8.00M this segment, stats: ops{read=4110 overwritten=0 applied=4109 ignored=0} inserts{seen=205400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:57:15.709965 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 211: this message will repeat every 5th retry.
W20250116 02:57:16.270282 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 216: this message will repeat every 5th retry.
W20250116 02:57:16.782277 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 221: this message will repeat every 5th retry.
I20250116 02:57:17.060323 12147 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 2120 ops behind the committed index [suppressed 28 similar messages]
W20250116 02:57:17.367116 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 226: this message will repeat every 5th retry.
I20250116 02:57:17.742259 11894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:17.779479 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:17.780614 11214 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:17.809209 10394 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250116 02:57:17.835809 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:17.860574 11070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:17.890187 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:17.956657 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:57:17.987828 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 231: this message will repeat every 5th retry.
I20250116 02:57:18.079282 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:18.347487 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replayed 1/3 log segments. Stats: ops{read=4622 overwritten=0 applied=4621 ignored=0} inserts{seen=231000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20250116 02:57:18.461795 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 236: this message will repeat every 5th retry.
W20250116 02:57:18.892331 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 241: this message will repeat every 5th retry.
W20250116 02:57:19.432986 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 246: this message will repeat every 5th retry.
W20250116 02:57:19.902873 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 251: this message will repeat every 5th retry.
I20250116 02:57:20.084862 12025 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 2411 ops behind the committed index [suppressed 28 similar messages]
I20250116 02:57:20.454878 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replaying log segment 2/3 (727.3K/8.00M this segment, stats: ops{read=5032 overwritten=0 applied=5032 ignored=0} inserts{seen=251550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:57:20.473845 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 256: this message will repeat every 5th retry.
W20250116 02:57:20.927546 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 261: this message will repeat every 5th retry.
W20250116 02:57:21.541349 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 266: this message will repeat every 5th retry.
W20250116 02:57:22.019554 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 271: this message will repeat every 5th retry.
W20250116 02:57:22.649813 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 276: this message will repeat every 5th retry.
I20250116 02:57:23.045154 10465 maintenance_manager.cc:418] P 429bf4133bf54797b117ef8fad632920: Scheduling FlushMRSOp(29c42185024a48cda0403d61962cf176): perf score=0.079414
I20250116 02:57:23.180986 12207 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 2743 ops behind the committed index [suppressed 27 similar messages]
W20250116 02:57:23.215056 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 281: this message will repeat every 5th retry.
W20250116 02:57:23.797677 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 286: this message will repeat every 5th retry.
W20250116 02:57:24.412117 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 291: this message will repeat every 5th retry.
W20250116 02:57:24.938763 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 296: this message will repeat every 5th retry.
I20250116 02:57:25.250276 11894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:25.260854 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:25.283490 10394 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250116 02:57:25.429869 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:25.476485 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replaying log segment 2/3 (2.24M/8.00M this segment, stats: ops{read=5917 overwritten=0 applied=5917 ignored=0} inserts{seen=295800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250116 02:57:25.499884 11214 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:25.550180 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:25.552024 11070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:25.561077 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:25.561550 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:57:25.571496 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 301: this message will repeat every 5th retry.
I20250116 02:57:26.220248 12266 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 2951 ops behind the committed index [suppressed 24 similar messages]
W20250116 02:57:26.260233 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 306: this message will repeat every 5th retry.
W20250116 02:57:26.917308 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 311: this message will repeat every 5th retry.
W20250116 02:57:27.459160 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 316: this message will repeat every 5th retry.
W20250116 02:57:27.984155 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 321: this message will repeat every 5th retry.
W20250116 02:57:28.528431 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 326: this message will repeat every 5th retry.
W20250116 02:57:28.970942 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 331: this message will repeat every 5th retry.
I20250116 02:57:29.248191 12272 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 3194 ops behind the committed index [suppressed 27 similar messages]
W20250116 02:57:29.517361 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 336: this message will repeat every 5th retry.
W20250116 02:57:30.053424 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 341: this message will repeat every 5th retry.
I20250116 02:57:30.477890 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replaying log segment 2/3 (3.70M/8.00M this segment, stats: ops{read=6758 overwritten=0 applied=6756 ignored=0} inserts{seen=337750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:57:30.641624 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 346: this message will repeat every 5th retry.
W20250116 02:57:31.254263 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 351: this message will repeat every 5th retry.
W20250116 02:57:31.737388 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 356: this message will repeat every 5th retry.
W20250116 02:57:32.241446 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 361: this message will repeat every 5th retry.
I20250116 02:57:32.289484 12279 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 3448 ops behind the committed index [suppressed 27 similar messages]
I20250116 02:57:32.677913 11894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:32.701478 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:32.730250 11214 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:32.806474 10394 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250116 02:57:32.809921 11070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:32.828397 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:32.866464 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:32.899137 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:32.905951 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:57:32.921623 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 366: this message will repeat every 5th retry.
W20250116 02:57:33.588259 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 371: this message will repeat every 5th retry.
W20250116 02:57:34.287585 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 376: this message will repeat every 5th retry.
W20250116 02:57:34.839695 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 381: this message will repeat every 5th retry.
I20250116 02:57:35.369268 12279 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 3667 ops behind the committed index [suppressed 23 similar messages]
W20250116 02:57:35.434775 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 386: this message will repeat every 5th retry.
I20250116 02:57:35.479138 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replaying log segment 2/3 (5.21M/8.00M this segment, stats: ops{read=7633 overwritten=0 applied=7633 ignored=0} inserts{seen=381600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:57:35.956640 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 391: this message will repeat every 5th retry.
W20250116 02:57:36.466271 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 396: this message will repeat every 5th retry.
W20250116 02:57:36.981032 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 401: this message will repeat every 5th retry.
W20250116 02:57:37.629355 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 406: this message will repeat every 5th retry.
W20250116 02:57:38.177140 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 411: this message will repeat every 5th retry.
I20250116 02:57:38.445744 12279 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 3898 ops behind the committed index [suppressed 27 similar messages]
W20250116 02:57:38.670998 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 416: this message will repeat every 5th retry.
W20250116 02:57:39.268828 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 421: this message will repeat every 5th retry.
W20250116 02:57:40.231427 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 426: this message will repeat every 5th retry.
I20250116 02:57:40.481119 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replaying log segment 2/3 (6.98M/8.00M this segment, stats: ops{read=8653 overwritten=0 applied=8653 ignored=0} inserts{seen=432600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250116 02:57:40.786084 11894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:40.788733 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:40.904109 10394 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250116 02:57:40.922230 11214 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:40.941915 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:40.957993 11070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:41.005550 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:41.011768 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:41.027522 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:57:41.064947 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 431: this message will repeat every 5th retry.
I20250116 02:57:41.528084 12275 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 4117 ops behind the committed index [suppressed 20 similar messages]
W20250116 02:57:41.712739 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 436: this message will repeat every 5th retry.
W20250116 02:57:42.313385 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 441: this message will repeat every 5th retry.
W20250116 02:57:42.995797 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 446: this message will repeat every 5th retry.
W20250116 02:57:43.475947 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 451: this message will repeat every 5th retry.
W20250116 02:57:44.015271 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 456: this message will repeat every 5th retry.
W20250116 02:57:44.546895 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 461: this message will repeat every 5th retry.
I20250116 02:57:44.576785 12385 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 4356 ops behind the committed index [suppressed 26 similar messages]
I20250116 02:57:44.702664 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replayed 2/3 log segments. Stats: ops{read=9241 overwritten=0 applied=9239 ignored=0} inserts{seen=461900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
W20250116 02:57:45.111204 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 466: this message will repeat every 5th retry.
I20250116 02:57:45.482054 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replaying log segment 3/3 (267.9K/5.40M this segment, stats: ops{read=9392 overwritten=0 applied=9391 ignored=0} inserts{seen=469500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:57:45.642047 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 471: this message will repeat every 5th retry.
W20250116 02:57:46.280460 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 476: this message will repeat every 5th retry.
W20250116 02:57:46.838100 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 481: this message will repeat every 5th retry.
W20250116 02:57:47.548635 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 486: this message will repeat every 5th retry.
I20250116 02:57:47.626019 12385 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 4600 ops behind the committed index [suppressed 24 similar messages]
W20250116 02:57:48.163802 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 491: this message will repeat every 5th retry.
W20250116 02:57:48.640040 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 496: this message will repeat every 5th retry.
W20250116 02:57:49.151960 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 501: this message will repeat every 5th retry.
I20250116 02:57:49.245106 10249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:49.246984 11894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:49.334717 11214 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:49.353967 10394 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250116 02:57:49.415104 11070 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:57:49.437992 9392 meta_cache.cc:1261] Time spent looking up entry by key: real 0.052s user 0.000s sys 0.000s
I20250116 02:57:49.441119 9721 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:49.468145 10535 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:49.470240 9858 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250116 02:57:49.563681 9584 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250116 02:57:50.068717 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 506: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/maintenance_mode-itest.cc:753: Failure
Failed
Bad status: Timed out: cluster didn't become healthy: Running 'cluster ksck 127.7.228.126:43731' did not succeed in 60 seconds: Runtime error: error running 'cluster ksck 127.7.228.126:43731': stdout: Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
56ddececa4d0424dae4d75d6a003686b | 127.7.228.126:43731 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.228.84:36635 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server States
Server | State
----------------------------------+------------------
0791364971ba49f3b2813bdc36c11eda | MAINTENANCE_MODE
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0791364971ba49f3b2813bdc36c11eda | 127.7.228.68:41593 | HEALTHY | /L1 | 0 | 0
1361eef60904481394d5230d1bc84711 | 127.7.228.65:40373 | HEALTHY | /L0 | 0 | 0
429bf4133bf54797b117ef8fad632920 | 127.7.228.66:41805 | HEALTHY | /L0 | 1 | 3
497640d49e7242aaa63256055e91414e | 127.7.228.73:42313 | HEALTHY | /L2 | 0 | 0
660967712270485c81d6a5f1239c89b1 | 127.7.228.67:33499 | HEALTHY | /L0 | 0 | 0
9b5741c935d64e23af0c52c50e78420c | 127.7.228.72:45211 | HEALTHY | /L2 | 0 | 0
d5ba0801ecba49269379dd93eedde2f8 | 127.7.228.70:42173 | HEALTHY | /L1 | 0 | 0
dfb2cd26d0bb42c99dbd98c0c1a2f731 | 127.7.228.71:44289 | HEALTHY | /L1 | 0 | 0
ff972e037b0e40bba36f7896effdb71f | 127.7.228.69:46881 | HEALTHY | /L1 | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/L2 | 2
/L1 | 4
/L0 | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 9 server(s) checked
local_ip_for_outbound_sockets | 127.7.228.65 | experimental | 127.7.228.65:40373
local_ip_for_outbound_sockets | 127.7.228.66 | experimental | 127.7.228.66:41805
local_ip_for_outbound_sockets | 127.7.228.67 | experimental | 127.7.228.67:33499
local_ip_for_outbound_sockets | 127.7.228.68 | experimental | 127.7.228.68:41593
local_ip_for_outbound_sockets | 127.7.228.69 | experimental | 127.7.228.69:46881
local_ip_for_outbound_sockets | 127.7.228.70 | experimental | 127.7.228.70:42173
local_ip_for_outbound_sockets | 127.7.228.71 | experimental | 127.7.228.71:44289
local_ip_for_outbound_sockets | 127.7.228.72 | experimental | 127.7.228.72:45211
local_ip_for_outbound_sockets | 127.7.228.73 | experimental | 127.7.228.73:42313
never_fsync | true | unsafe,advanced | all 9 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 9 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked
server_dump_info_format | pb | hidden | all 9 server(s) checked
server_dump_info_path | /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-0/data/info.pb | hidden | 127.7.228.65:40373
server_dump_info_path | /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/data/info.pb | hidden | 127.7.228.66:41805
server_dump_info_path | /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-2/data/info.pb | hidden | 127.7.228.67:33499
server_dump_info_path | /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-3/data/info.pb | hidden | 127.7.228.68:41593
server_dump_info_path | /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-4/data/info.pb | hidden | 127.7.228.69:46881
server_dump_info_path | /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-5/data/info.pb | hidden | 127.7.228.70:42173
server_dump_info_path | /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-6/data/info.pb | hidden | 127.7.228.71:44289
server_dump_info_path | /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-7/data/info.pb | hidden | 127.7.228.72:45211
server_dump_info_path | /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-8/data/info.pb | hidden | 127.7.228.73:42313
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.7.228.84:36635 | all 9 server(s) checked
time_source | builtin | all 9 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------
1.18.0-SNAPSHOT | all 10 server(s) checked
Tablet Summary
Tablet 29c42185024a48cda0403d61962cf176 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
497640d49e7242aaa63256055e91414e (127.7.228.73:42313): RUNNING
429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): RUNNING [LEADER]
0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): not running
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap replaying log segment 3/3 (267.9K/5.40M this segment, stats: ops{read=9392 overwritten=0 applied=9391 ignored=0} inserts{seen=469500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
All reported replicas are:
A = 497640d49e7242aaa63256055e91414e
B = 429bf4133bf54797b117ef8fad632920
C = 0791364971ba49f3b2813bdc36c11eda
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 3 | -1 | Yes
B | A B* C | 3 | -1 | Yes
C | A B C | 3 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 9
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
, stderr: W20250116 02:57:48.569818 12391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250116 02:57:48.570236 12391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250116 02:57:48.583732 12391 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Runtime error: ksck discovered errors
: /tmp/dist-test-task8SxoCU/build/asan/bin/kudu: process exited with non-zero status 1
I20250116 02:57:50.484889 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replaying log segment 3/3 (1.36M/5.40M this segment, stats: ops{read=10025 overwritten=0 applied=10025 ignored=0} inserts{seen=501200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250116 02:57:50.659464 12392 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 4783 ops behind the committed index [suppressed 23 similar messages]
W20250116 02:57:50.680511 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 511: this message will repeat every 5th retry.
W20250116 02:57:51.250862 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 516: this message will repeat every 5th retry.
W20250116 02:57:51.838160 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 521: this message will repeat every 5th retry.
W20250116 02:57:52.393083 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 526: this message will repeat every 5th retry.
W20250116 02:57:52.921151 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 531: this message will repeat every 5th retry.
W20250116 02:57:53.489663 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 536: this message will repeat every 5th retry.
I20250116 02:57:53.682515 12392 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 4783 ops behind the committed index [suppressed 26 similar messages]
W20250116 02:57:54.021126 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 541: this message will repeat every 5th retry.
W20250116 02:57:54.521950 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 546: this message will repeat every 5th retry.
W20250116 02:57:55.024049 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 551: this message will repeat every 5th retry.
I20250116 02:57:55.497570 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replaying log segment 3/3 (3.44M/5.40M this segment, stats: ops{read=11226 overwritten=0 applied=11226 ignored=0} inserts{seen=561250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250116 02:57:55.574892 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 556: this message will repeat every 5th retry.
W20250116 02:57:56.052258 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 561: this message will repeat every 5th retry.
W20250116 02:57:56.596120 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 566: this message will repeat every 5th retry.
I20250116 02:57:56.697484 12444 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 4783 ops behind the committed index [suppressed 28 similar messages]
W20250116 02:57:57.117458 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 571: this message will repeat every 5th retry.
W20250116 02:57:57.647528 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 576: this message will repeat every 5th retry.
W20250116 02:57:58.093763 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 581: this message will repeat every 5th retry.
I20250116 02:57:58.378621 10350 maintenance_manager.cc:642] P 429bf4133bf54797b117ef8fad632920: FlushMRSOp(29c42185024a48cda0403d61962cf176) complete. Timing: real 35.333s user 23.939s sys 0.468s Metrics: {"bytes_written":21525135,"cfile_init":1,"dirs.queue_time_us":6341,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":14296,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":143710,"lbm_writes_1-10_ms":8,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":927,"peak_mem_usage":0,"rows_written":776450,"thread_start_us":21828,"threads_started":2,"wal-append.queue_time_us":15416}
I20250116 02:57:58.380211 10465 maintenance_manager.cc:418] P 429bf4133bf54797b117ef8fad632920: Scheduling UndoDeltaBlockGCOp(29c42185024a48cda0403d61962cf176): 9150401 bytes on disk
I20250116 02:57:58.381161 10350 maintenance_manager.cc:642] P 429bf4133bf54797b117ef8fad632920: UndoDeltaBlockGCOp(29c42185024a48cda0403d61962cf176) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
W20250116 02:57:58.651674 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 586: this message will repeat every 5th retry.
W20250116 02:57:59.195825 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 591: this message will repeat every 5th retry.
W20250116 02:57:59.655331 10349 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 596: this message will repeat every 5th retry.
I20250116 02:57:59.737376 12446 consensus_queue.cc:786] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920 [LEADER]: Peer 0791364971ba49f3b2813bdc36c11eda is lagging by at least 4783 ops behind the committed index [suppressed 29 similar messages]
I20250116 02:58:00.023926 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap replayed 3/3 log segments. Stats: ops{read=12374 overwritten=0 applied=12368 ignored=0} inserts{seen=618300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 6 replicates
I20250116 02:58:00.025002 11850 tablet_bootstrap.cc:492] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Bootstrap complete.
I20250116 02:58:00.031044 11850 ts_tablet_manager.cc:1397] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Time spent bootstrapping tablet: real 67.179s user 48.432s sys 1.605s
I20250116 02:58:00.035980 11850 raft_consensus.cc:357] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 3 FOLLOWER]: Replica starting. Triggering 6 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:58:00.040774 11850 raft_consensus.cc:738] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0791364971ba49f3b2813bdc36c11eda, State: Initialized, Role: FOLLOWER
I20250116 02:58:00.041740 11850 consensus_queue.cc:260] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12368, Last appended: 3.12374, Last appended by leader: 12374, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:58:00.051594 11850 ts_tablet_manager.cc:1428] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda: Time spent starting tablet: real 0.020s user 0.017s sys 0.004s
I20250116 02:58:00.814015 12451 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 429bf4133bf54797b117ef8fad632920)
I20250116 02:58:00.814282 12451 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:58:00.816289 12451 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 497640d49e7242aaa63256055e91414e (127.7.228.73:42313), 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805)
I20250116 02:58:00.841533 10414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "0791364971ba49f3b2813bdc36c11eda" candidate_term: 4 candidate_status { last_received { term: 3 index: 13649 } } ignore_live_leader: false dest_uuid: "429bf4133bf54797b117ef8fad632920" is_pre_election: true
I20250116 02:58:00.848601 11233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "0791364971ba49f3b2813bdc36c11eda" candidate_term: 4 candidate_status { last_received { term: 3 index: 13649 } } ignore_live_leader: false dest_uuid: "497640d49e7242aaa63256055e91414e" is_pre_election: true
I20250116 02:58:00.850423 11844 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0791364971ba49f3b2813bdc36c11eda; no voters: 429bf4133bf54797b117ef8fad632920, 497640d49e7242aaa63256055e91414e
I20250116 02:58:00.850929 12451 raft_consensus.cc:2743] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250116 02:58:01.711553 8081 client.cc:2040] Couldn't close scanner e9f1bdbb909849d9ac8c025f82e2d45b: Aborted: reactor is shutting down (error 108)
I20250116 02:58:01.712000 8081 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20250116 02:58:01.712158 8081 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 1361eef60904481394d5230d1bc84711 and pid 10185
I20250116 02:58:01.894812 10465 maintenance_manager.cc:418] P 429bf4133bf54797b117ef8fad632920: Scheduling LogGCOp(29c42185024a48cda0403d61962cf176): free 25169683 bytes of WAL
I20250116 02:58:01.895202 10350 log_reader.cc:385] T 29c42185024a48cda0403d61962cf176: removed 3 log segments from log reader
I20250116 02:58:01.895320 10350 log.cc:1078] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Deleting log segment in path: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal/wals/29c42185024a48cda0403d61962cf176/wal-000000001 (ops 1-4980)
I20250116 02:58:01.895437 10350 log.cc:1078] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Deleting log segment in path: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal/wals/29c42185024a48cda0403d61962cf176/wal-000000002 (ops 4981-9646)
I20250116 02:58:01.895498 10350 log.cc:1078] T 29c42185024a48cda0403d61962cf176 P 429bf4133bf54797b117ef8fad632920: Deleting log segment in path: /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0/minicluster-data/ts-1/wal/wals/29c42185024a48cda0403d61962cf176/wal-000000003 (ops 9647-14305)
I20250116 02:58:01.901952 10350 maintenance_manager.cc:642] P 429bf4133bf54797b117ef8fad632920: LogGCOp(29c42185024a48cda0403d61962cf176) complete. Timing: real 0.007s user 0.000s sys 0.007s Metrics: {}
************************ BEGIN STACKS **************************
[New LWP 10187]
[New LWP 10188]
[New LWP 10189]
[New LWP 10190]
[New LWP 10196]
[New LWP 10197]
[New LWP 10198]
[New LWP 10201]
[New LWP 10202]
[New LWP 10203]
[New LWP 10204]
[New LWP 10205]
[New LWP 10206]
[New LWP 10207]
[New LWP 10209]
[New LWP 10210]
[New LWP 10211]
[New LWP 10212]
[New LWP 10213]
[New LWP 10214]
[New LWP 10215]
[New LWP 10216]
[New LWP 10217]
[New LWP 10218]
[New LWP 10219]
[New LWP 10220]
[New LWP 10221]
[New LWP 10222]
[New LWP 10223]
[New LWP 10224]
[New LWP 10225]
[New LWP 10226]
[New LWP 10227]
[New LWP 10228]
[New LWP 10229]
[New LWP 10230]
[New LWP 10231]
[New LWP 10232]
[New LWP 10233]
[New LWP 10234]
[New LWP 10235]
[New LWP 10236]
[New LWP 10237]
[New LWP 10238]
[New LWP 10239]
[New LWP 10240]
[New LWP 10241]
[New LWP 10242]
[New LWP 10243]
[New LWP 10244]
[New LWP 10245]
[New LWP 10246]
[New LWP 10247]
[New LWP 10248]
[New LWP 10249]
[New LWP 10250]
[New LWP 10251]
[New LWP 10252]
[New LWP 10253]
[New LWP 10254]
[New LWP 10255]
[New LWP 10256]
[New LWP 10257]
[New LWP 10258]
[New LWP 10260]
[New LWP 10261]
[New LWP 10262]
[New LWP 10263]
[New LWP 10264]
[New LWP 10265]
[New LWP 10266]
[New LWP 10267]
[New LWP 10268]
[New LWP 10269]
[New LWP 10270]
[New LWP 10271]
[New LWP 10272]
[New LWP 10273]
[New LWP 10274]
[New LWP 10275]
[New LWP 10276]
[New LWP 10277]
[New LWP 10278]
[New LWP 10279]
[New LWP 10280]
[New LWP 10281]
[New LWP 10282]
[New LWP 10283]
[New LWP 10284]
[New LWP 10285]
[New LWP 10286]
[New LWP 10287]
[New LWP 10288]
[New LWP 10289]
[New LWP 10290]
[New LWP 10291]
[New LWP 10292]
[New LWP 10293]
[New LWP 10294]
[New LWP 10295]
[New LWP 10296]
[New LWP 10297]
[New LWP 10298]
[New LWP 10299]
[New LWP 10300]
[New LWP 10301]
[New LWP 10302]
[New LWP 10303]
[New LWP 10304]
[New LWP 10305]
[New LWP 10306]
[New LWP 10307]
[New LWP 10308]
[New LWP 10309]
[New LWP 10310]
[New LWP 10311]
[New LWP 10312]
[New LWP 10313]
[New LWP 10314]
[New LWP 10316]
[New LWP 10320]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007f78d162dd50 in ?? ()
Id Target Id Frame
* 1 LWP 10185 "kudu" 0x00007f78d162dd50 in ?? ()
2 LWP 10187 "kudu" 0x00007f78d1629fb9 in ?? ()
3 LWP 10188 "kudu" 0x00007f78d1629fb9 in ?? ()
4 LWP 10189 "kudu" 0x00007f78d1629fb9 in ?? ()
5 LWP 10190 "kernel-watcher-" 0x00007f78d1629fb9 in ?? ()
6 LWP 10196 "ntp client-1019" 0x00007f78d162d9e2 in ?? ()
7 LWP 10197 "file cache-evic" 0x00007f78d1629fb9 in ?? ()
8 LWP 10198 "sq_acceptor" 0x00007f78ca38dcb9 in ?? ()
9 LWP 10201 "rpc reactor-102" 0x00007f78ca39aa47 in ?? ()
10 LWP 10202 "rpc reactor-102" 0x00007f78ca39aa47 in ?? ()
11 LWP 10203 "rpc reactor-102" 0x00007f78ca39aa47 in ?? ()
12 LWP 10204 "rpc reactor-102" 0x00007f78ca39aa47 in ?? ()
13 LWP 10205 "MaintenanceMgr " 0x00007f78d1629ad3 in ?? ()
14 LWP 10206 "txn-status-mana" 0x00007f78d1629fb9 in ?? ()
15 LWP 10207 "collect_and_rem" 0x00007f78d1629fb9 in ?? ()
16 LWP 10209 "tc-session-exp-" 0x00007f78d1629fb9 in ?? ()
17 LWP 10210 "rpc worker-1021" 0x00007f78d1629ad3 in ?? ()
18 LWP 10211 "rpc worker-1021" 0x00007f78d1629ad3 in ?? ()
19 LWP 10212 "rpc worker-1021" 0x00007f78d1629ad3 in ?? ()
20 LWP 10213 "rpc worker-1021" 0x00007f78d1629ad3 in ?? ()
21 LWP 10214 "rpc worker-1021" 0x00007f78d1629ad3 in ?? ()
22 LWP 10215 "rpc worker-1021" 0x00007f78d1629ad3 in ?? ()
23 LWP 10216 "rpc worker-1021" 0x00007f78d1629ad3 in ?? ()
24 LWP 10217 "rpc worker-1021" 0x00007f78d1629ad3 in ?? ()
25 LWP 10218 "rpc worker-1021" 0x00007f78d1629ad3 in ?? ()
26 LWP 10219 "rpc worker-1021" 0x00007f78d1629ad3 in ?? ()
27 LWP 10220 "rpc worker-1022" 0x00007f78d1629ad3 in ?? ()
28 LWP 10221 "rpc worker-1022" 0x00007f78d1629ad3 in ?? ()
29 LWP 10222 "rpc worker-1022" 0x00007f78d1629ad3 in ?? ()
30 LWP 10223 "rpc worker-1022" 0x00007f78d1629ad3 in ?? ()
31 LWP 10224 "rpc worker-1022" 0x00007f78d1629ad3 in ?? ()
32 LWP 10225 "rpc worker-1022" 0x00007f78d1629ad3 in ?? ()
33 LWP 10226 "rpc worker-1022" 0x00007f78d1629ad3 in ?? ()
34 LWP 10227 "rpc worker-1022" 0x00007f78d1629ad3 in ?? ()
35 LWP 10228 "rpc worker-1022" 0x00007f78d1629ad3 in ?? ()
36 LWP 10229 "rpc worker-1022" 0x00007f78d1629ad3 in ?? ()
37 LWP 10230 "rpc worker-1023" 0x00007f78d1629ad3 in ?? ()
38 LWP 10231 "rpc worker-1023" 0x00007f78d1629ad3 in ?? ()
39 LWP 10232 "rpc worker-1023" 0x00007f78d1629ad3 in ?? ()
40 LWP 10233 "rpc worker-1023" 0x00007f78d1629ad3 in ?? ()
41 LWP 10234 "rpc worker-1023" 0x00007f78d1629ad3 in ?? ()
42 LWP 10235 "rpc worker-1023" 0x00007f78d1629ad3 in ?? ()
43 LWP 10236 "rpc worker-1023" 0x00007f78d1629ad3 in ?? ()
44 LWP 10237 "rpc worker-1023" 0x00007f78d1629ad3 in ?? ()
45 LWP 10238 "rpc worker-1023" 0x00007f78d1629ad3 in ?? ()
46 LWP 10239 "rpc worker-1023" 0x00007f78d1629ad3 in ?? ()
47 LWP 10240 "rpc worker-1024" 0x00007f78d1629ad3 in ?? ()
48 LWP 10241 "rpc worker-1024" 0x00007f78d1629ad3 in ?? ()
49 LWP 10242 "rpc worker-1024" 0x00007f78d1629ad3 in ?? ()
50 LWP 10243 "rpc worker-1024" 0x00007f78d1629ad3 in ?? ()
51 LWP 10244 "rpc worker-1024" 0x00007f78d1629ad3 in ?? ()
52 LWP 10245 "rpc worker-1024" 0x00007f78d1629ad3 in ?? ()
53 LWP 10246 "rpc worker-1024" 0x00007f78d1629ad3 in ?? ()
54 LWP 10247 "rpc worker-1024" 0x00007f78d1629ad3 in ?? ()
55 LWP 10248 "rpc worker-1024" 0x00007f78d1629ad3 in ?? ()
56 LWP 10249 "rpc worker-1024" 0x00007f78d1629ad3 in ?? ()
57 LWP 10250 "rpc worker-1025" 0x00007f78d1629ad3 in ?? ()
58 LWP 10251 "rpc worker-1025" 0x00007f78d1629ad3 in ?? ()
59 LWP 10252 "rpc worker-1025" 0x00007f78d1629ad3 in ?? ()
60 LWP 10253 "rpc worker-1025" 0x00007f78d1629ad3 in ?? ()
61 LWP 10254 "rpc worker-1025" 0x00007f78d1629ad3 in ?? ()
62 LWP 10255 "rpc worker-1025" 0x00007f78d1629ad3 in ?? ()
63 LWP 10256 "rpc worker-1025" 0x00007f78d1629ad3 in ?? ()
64 LWP 10257 "rpc worker-1025" 0x00007f78d1629ad3 in ?? ()
65 LWP 10258 "rpc worker-1025" 0x00007f78d1629ad3 in ?? ()
66 LWP 10260 "rpc worker-1026" 0x00007f78d1629ad3 in ?? ()
67 LWP 10261 "rpc worker-1026" 0x00007f78d1629ad3 in ?? ()
68 LWP 10262 "rpc worker-1026" 0x00007f78d1629ad3 in ?? ()
69 LWP 10263 "rpc worker-1026" 0x00007f78d1629ad3 in ?? ()
70 LWP 10264 "rpc worker-1026" 0x00007f78d1629ad3 in ?? ()
71 LWP 10265 "rpc worker-1026" 0x00007f78d1629ad3 in ?? ()
72 LWP 10266 "rpc worker-1026" 0x00007f78d1629ad3 in ?? ()
73 LWP 10267 "rpc worker-1026" 0x00007f78d1629ad3 in ?? ()
74 LWP 10268 "rpc worker-1026" 0x00007f78d1629ad3 in ?? ()
75 LWP 10269 "rpc worker-1026" 0x00007f78d1629ad3 in ?? ()
76 LWP 10270 "rpc worker-1027" 0x00007f78d1629ad3 in ?? ()
77 LWP 10271 "rpc worker-1027" 0x00007f78d1629ad3 in ?? ()
78 LWP 10272 "rpc worker-1027" 0x00007f78d1629ad3 in ?? ()
79 LWP 10273 "rpc worker-1027" 0x00007f78d1629ad3 in ?? ()
80 LWP 10274 "rpc worker-1027" 0x00007f78d1629ad3 in ?? ()
81 LWP 10275 "rpc worker-1027" 0x00007f78d1629ad3 in ?? ()
82 LWP 10276 "rpc worker-1027" 0x00007f78d1629ad3 in ?? ()
83 LWP 10277 "rpc worker-1027" 0x00007f78d1629ad3 in ?? ()
84 LWP 10278 "rpc worker-1027" 0x00007f78d1629ad3 in ?? ()
85 LWP 10279 "rpc worker-1027" 0x00007f78d1629ad3 in ?? ()
86 LWP 10280 "rpc worker-1028" 0x00007f78d1629ad3 in ?? ()
87 LWP 10281 "rpc worker-1028" 0x00007f78d1629ad3 in ?? ()
88 LWP 10282 "rpc worker-1028" 0x00007f78d1629ad3 in ?? ()
89 LWP 10283 "rpc worker-1028" 0x00007f78d1629ad3 in ?? ()
90 LWP 10284 "rpc worker-1028" 0x00007f78d1629ad3 in ?? ()
91 LWP 10285 "rpc worker-1028" 0x00007f78d1629ad3 in ?? ()
92 LWP 10286 "rpc worker-1028" 0x00007f78d1629ad3 in ?? ()
93 LWP 10287 "rpc worker-1028" 0x00007f78d1629ad3 in ?? ()
94 LWP 10288 "rpc worker-1028" 0x00007f78d1629ad3 in ?? ()
95 LWP 10289 "rpc worker-1028" 0x00007f78d1629ad3 in ?? ()
96 LWP 10290 "rpc worker-1029" 0x00007f78d1629ad3 in ?? ()
97 LWP 10291 "rpc worker-1029" 0x00007f78d1629ad3 in ?? ()
98 LWP 10292 "rpc worker-1029" 0x00007f78d1629ad3 in ?? ()
99 LWP 10293 "rpc worker-1029" 0x00007f78d1629ad3 in ?? ()
100 LWP 10294 "rpc worker-1029" 0x00007f78d1629ad3 in ?? ()
101 LWP 10295 "rpc worker-1029" 0x00007f78d1629ad3 in ?? ()
102 LWP 10296 "rpc worker-1029" 0x00007f78d1629ad3 in ?? ()
103 LWP 10297 "rpc worker-1029" 0x00007f78d1629ad3 in ?? ()
104 LWP 10298 "rpc worker-1029" 0x00007f78d1629ad3 in ?? ()
105 LWP 10299 "rpc worker-1029" 0x00007f78d1629ad3 in ?? ()
106 LWP 10300 "rpc worker-1030" 0x00007f78d1629ad3 in ?? ()
107 LWP 10301 "rpc worker-1030" 0x00007f78d1629ad3 in ?? ()
108 LWP 10302 "rpc worker-1030" 0x00007f78d1629ad3 in ?? ()
109 LWP 10303 "rpc worker-1030" 0x00007f78d1629ad3 in ?? ()
110 LWP 10304 "rpc worker-1030" 0x00007f78d1629ad3 in ?? ()
111 LWP 10305 "rpc worker-1030" 0x00007f78d1629ad3 in ?? ()
112 LWP 10306 "rpc worker-1030" 0x00007f78d1629ad3 in ?? ()
113 LWP 10307 "rpc worker-1030" 0x00007f78d1629ad3 in ?? ()
114 LWP 10308 "rpc worker-1030" 0x00007f78d1629ad3 in ?? ()
115 LWP 10309 "rpc worker-1030" 0x00007f78d1629ad3 in ?? ()
116 LWP 10310 "rpc worker-1031" 0x00007f78d1629ad3 in ?? ()
117 LWP 10311 "diag-logger-103" 0x00007f78d1629fb9 in ?? ()
118 LWP 10312 "result-tracker-" 0x00007f78d1629fb9 in ?? ()
119 LWP 10313 "excess-log-dele" 0x00007f78d1629fb9 in ?? ()
120 LWP 10314 "acceptor-10314" 0x00007f78ca39c0c7 in ?? ()
121 LWP 10316 "heartbeat-10316" 0x00007f78d1629fb9 in ?? ()
122 LWP 10320 "maintenance_sch" 0x00007f78d1629fb9 in ?? ()
Thread 122 (LWP 10320):
#0 0x00007f78d1629fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000002c1 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007f7886f14550 in ?? ()
#6 0x0000000000000582 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 10316):
#0 0x00007f78d1629fb9 in ?? ()
#1 0x81afaaa989cdd16b in ?? ()
#2 0x00000000000000b0 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007f788772c5b0 in ?? ()
#6 0x0000000000000160 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 10314):
#0 0x00007f78ca39c0c7 in ?? ()
#1 0x0000614000021aa8 in ?? ()
#2 0x00007f7887f54d50 in ?? ()
#3 0x00007f7887f54d80 in ?? ()
#4 0x00007f7887f54e80 in ?? ()
#5 0x00007f7887f54d70 in ?? ()
#6 0x00007f7887f54de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007f78ce220ef0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 10313):
#0 0x00007f78d1629fb9 in ?? ()
#1 0x00007f788876df00 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fffdb67a170 in ?? ()
#5 0x00007f788876def0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 10312):
#0 0x00007f78d1629fb9 in ?? ()
#1 0x00007f7888f860c0 in ?? ()
#2 0x00000000000000b0 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007f7888f860b0 in ?? ()
#6 0x0000000000000160 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 10311):
#0 0x00007f78d1629fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 10310):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x000000000000002b in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007f7889fb5e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7889fb5e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007f78d1629770 in ?? ()
#10 0x00007f7889fb5e70 in ?? ()
#11 0x00000fef113f6bd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 10309):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000019 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007f788a7cde50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f788a7cde70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007f78d1629770 in ?? ()
#10 0x00007f788a7cde70 in ?? ()
#11 0x00000fef114f9bd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 10308):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 10307):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 10306):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 10305):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 10304):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 10303):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 10302):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 10301):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 10300):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 10299):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 10298):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 10297):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 10296):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 10295):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 10294):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 10293):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 10292):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 10291):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 10290):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 10289):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 10288):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 10287):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 10286):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 10285):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 10284):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 10283):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 10282):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 10281):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 10280):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 10279):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 10278):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 10277):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 10276):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 10275):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 10274):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 10273):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 10272):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 10271):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 10270):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007f789e386e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f789e386e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007f78d1629770 in ?? ()
#10 0x00007f789e386e70 in ?? ()
#11 0x00000fef13c70dd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 10269):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 10268):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 10267):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 10266):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 10265):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 10264):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 10263):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 10262):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 10261):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 10260):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 10258):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 10257):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 10256):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 10255):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 10254):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 10253):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 10252):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 10251):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 10250):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 10249):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007f78a8567e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f78a8567e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007f78d1629770 in ?? ()
#10 0x00007f78a8567e70 in ?? ()
#11 0x00000fef150acfd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 10248):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 10247):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 10246):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 10245):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 10244):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 10243):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 10242):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 10241):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 10240):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 10239):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 10238):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 10237):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 10236):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 10235):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 10234):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 10233):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 10232):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 10231):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 10230):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 10229):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007f78b2748e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f78b2748e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007f78d1629770 in ?? ()
#10 0x00007f78b2748e70 in ?? ()
#11 0x00000fef164e91d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 10228):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 10227):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 10226):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 10225):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 10224):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 10223):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 10222):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 10221):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 10220):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 10219):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 10218):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 10217):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 10216):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 10215):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 10214):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 10213):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 10212):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 10211):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 10210):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 10209):
#0 0x00007f78d1629fb9 in ?? ()
#1 0x00007f78bc133c80 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001b360 in ?? ()
#5 0x00007f78bc133c70 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 10207):
#0 0x00007f78d1629fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 10206):
#0 0x00007f78d1629fb9 in ?? ()
#1 0x00007f78bd1711c0 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007f78bd1711b0 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 10205):
#0 0x00007f78d1629ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 10204):
#0 0x00007f78ca39aa47 in ?? ()
#1 0x00007f78be1b22e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007f78be1b22d0 in ?? ()
#4 0x00007f78be1b24e0 in ?? ()
#5 0x00007f78be1b2320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007f78be1b23a0 in ?? ()
#8 0x00007f78cc9fa25d in ?? ()
#9 0x3fb9496f8f1f6000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000037800000 in ?? ()
#14 0x00007f7800000000 in ?? ()
#15 0x41d9e21ced72049a in ?? ()
#16 0x00000fef97c2e474 in ?? ()
#17 0x00007f78be1b2380 in ?? ()
#18 0x00007f78cc9feba3 in ?? ()
#19 0x00007f78be1b2350 in ?? ()
#20 0x3fb9496f8f1f6000 in ?? ()
#21 0x00000000be1b23a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb9496f8f1f6000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 10203):
#0 0x00007f78ca39aa47 in ?? ()
#1 0x00007f78be9b32e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007f78be9b32d0 in ?? ()
#4 0x00007f78be9b34e0 in ?? ()
#5 0x00007f78be9b3320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007f78be9b33a0 in ?? ()
#8 0x00007f78cc9fa25d in ?? ()
#9 0x3fb98a86bd86a000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x00000000dd000000 in ?? ()
#14 0x00007f7800000000 in ?? ()
#15 0x41d9e21ced720498 in ?? ()
#16 0x00000fef97d2e674 in ?? ()
#17 0x00007f78be9b3380 in ?? ()
#18 0x00007f78cc9feba3 in ?? ()
#19 0x00007f78be9b3350 in ?? ()
#20 0x3fb98a86bd86a000 in ?? ()
#21 0x00000000be9b33a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb98a86bd86a000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 10202):
#0 0x00007f78ca39aa47 in ?? ()
#1 0x00007f78bf1b42e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007f78bf1b42d0 in ?? ()
#4 0x00007f78bf1b44e0 in ?? ()
#5 0x00007f78bf1b4320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007f78bf1b43a0 in ?? ()
#8 0x00007f78cc9fa25d in ?? ()
#9 0x3fb97538735ce000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000031000000 in ?? ()
#14 0x00007f7800000000 in ?? ()
#15 0x41d9e21ced720495 in ?? ()
#16 0x00000fef97e2e874 in ?? ()
#17 0x00007f78bf1b4380 in ?? ()
#18 0x00007f78cc9feba3 in ?? ()
#19 0x00007f78bf1b4350 in ?? ()
#20 0x3fb97538735ce000 in ?? ()
#21 0x00000000bf1b43a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb97538735ce000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 10201):
#0 0x00007f78ca39aa47 in ?? ()
#1 0x00007f78c11ad2e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007f78c11ad2d0 in ?? ()
#4 0x00007f78c11ad4e0 in ?? ()
#5 0x00007f78c11ad320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f78c11ad3a0 in ?? ()
#8 0x00007f78cc9fa25d in ?? ()
#9 0x3fb69439d1920000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x00000000a7a00000 in ?? ()
#14 0x00007f7800000000 in ?? ()
#15 0x41d9e21ced720498 in ?? ()
#16 0x00000fef9822da74 in ?? ()
#17 0x00007f78c11ad380 in ?? ()
#18 0x00007f78cc9feba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 10198):
#0 0x00007f78ca38dcb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f78c2bb7788 in ?? ()
#3 0x00006020000159d0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 10197):
#0 0x00007f78d1629fb9 in ?? ()
#1 0x00007f78c23b6080 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007f78c23b6070 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 10196):
#0 0x00007f78d162d9e2 in ?? ()
#1 0x00007f78c1bb3b60 in ?? ()
#2 0x00007f78c1bb3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f78c1bb3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007f78c1bb3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007f78c1bb3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007f78d6f51f0c in ?? ()
#12 0x00007f78d6f417b9 in ?? ()
#13 0x00007f78d6f45da6 in ?? ()
#14 0x00007f78d6f4a7e0 in ?? ()
#15 0x00007f78d6f4a609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007f78ce24198e in ?? ()
#18 0x00007f78d16236db in ?? ()
#19 0x00000fef9836e67c in ?? ()
#20 0x00007f78c1bb3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 10190):
#0 0x00007f78d1629fb9 in ?? ()
#1 0x00007f78c3bb8c40 in ?? ()
#2 0x0000000000000377 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007f78c3bb8c30 in ?? ()
#6 0x00000000000006ee in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 10189):
#0 0x00007f78d1629fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 10188):
#0 0x00007f78d1629fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 10187):
#0 0x00007f78d1629fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f78c53bc410 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 10185):
#0 0x00007f78d162dd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250116 02:58:02.923687 8081 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID 429bf4133bf54797b117ef8fad632920 and pid 10327
************************ BEGIN STACKS **************************
[New LWP 10331]
[New LWP 10332]
[New LWP 10333]
[New LWP 10335]
[New LWP 10341]
[New LWP 10342]
[New LWP 10343]
[New LWP 10346]
[New LWP 10347]
[New LWP 10348]
[New LWP 10349]
[New LWP 10350]
[New LWP 10351]
[New LWP 10353]
[New LWP 10354]
[New LWP 10355]
[New LWP 10356]
[New LWP 10357]
[New LWP 10358]
[New LWP 10359]
[New LWP 10360]
[New LWP 10361]
[New LWP 10362]
[New LWP 10363]
[New LWP 10364]
[New LWP 10365]
[New LWP 10366]
[New LWP 10367]
[New LWP 10368]
[New LWP 10369]
[New LWP 10370]
[New LWP 10371]
[New LWP 10372]
[New LWP 10373]
[New LWP 10374]
[New LWP 10375]
[New LWP 10376]
[New LWP 10377]
[New LWP 10378]
[New LWP 10379]
[New LWP 10380]
[New LWP 10381]
[New LWP 10382]
[New LWP 10383]
[New LWP 10384]
[New LWP 10385]
[New LWP 10386]
[New LWP 10387]
[New LWP 10388]
[New LWP 10389]
[New LWP 10390]
[New LWP 10391]
[New LWP 10392]
[New LWP 10393]
[New LWP 10394]
[New LWP 10395]
[New LWP 10396]
[New LWP 10397]
[New LWP 10398]
[New LWP 10399]
[New LWP 10400]
[New LWP 10401]
[New LWP 10402]
[New LWP 10403]
[New LWP 10404]
[New LWP 10405]
[New LWP 10406]
[New LWP 10407]
[New LWP 10408]
[New LWP 10409]
[New LWP 10410]
[New LWP 10411]
[New LWP 10412]
[New LWP 10413]
[New LWP 10414]
[New LWP 10415]
[New LWP 10416]
[New LWP 10417]
[New LWP 10418]
[New LWP 10419]
[New LWP 10420]
[New LWP 10421]
[New LWP 10422]
[New LWP 10423]
[New LWP 10424]
[New LWP 10425]
[New LWP 10426]
[New LWP 10427]
[New LWP 10428]
[New LWP 10429]
[New LWP 10430]
[New LWP 10431]
[New LWP 10432]
[New LWP 10433]
[New LWP 10434]
[New LWP 10435]
[New LWP 10436]
[New LWP 10437]
[New LWP 10438]
[New LWP 10439]
[New LWP 10440]
[New LWP 10441]
[New LWP 10442]
[New LWP 10443]
[New LWP 10444]
[New LWP 10445]
[New LWP 10446]
[New LWP 10447]
[New LWP 10448]
[New LWP 10449]
[New LWP 10450]
[New LWP 10451]
[New LWP 10452]
[New LWP 10454]
[New LWP 10455]
[New LWP 10457]
[New LWP 10458]
[New LWP 10459]
[New LWP 10460]
[New LWP 10461]
[New LWP 10465]
[New LWP 12470]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007f7e6cec7d50 in ?? ()
Id Target Id Frame
* 1 LWP 10327 "kudu" 0x00007f7e6cec7d50 in ?? ()
2 LWP 10331 "kudu" 0x00007f7e6cec3fb9 in ?? ()
3 LWP 10332 "kudu" 0x00007f7e6cec3fb9 in ?? ()
4 LWP 10333 "kudu" 0x00007f7e6cec3fb9 in ?? ()
5 LWP 10335 "kernel-watcher-" 0x00007f7e6cec3fb9 in ?? ()
6 LWP 10341 "ntp client-1034" 0x00007f7e6cec79e2 in ?? ()
7 LWP 10342 "file cache-evic" 0x00007f7e6cec3fb9 in ?? ()
8 LWP 10343 "sq_acceptor" 0x00007f7e65c27cb9 in ?? ()
9 LWP 10346 "rpc reactor-103" 0x00007f7e65c34a47 in ?? ()
10 LWP 10347 "rpc reactor-103" 0x00007f7e65c34a47 in ?? ()
11 LWP 10348 "rpc reactor-103" 0x00007f7e65c34a47 in ?? ()
12 LWP 10349 "rpc reactor-103" 0x00007f7e65c34a47 in ?? ()
13 LWP 10350 "MaintenanceMgr " 0x00007f7e6cec3ad3 in ?? ()
14 LWP 10351 "txn-status-mana" 0x00007f7e6cec3fb9 in ?? ()
15 LWP 10353 "collect_and_rem" 0x00007f7e6cec3fb9 in ?? ()
16 LWP 10354 "tc-session-exp-" 0x00007f7e6cec3fb9 in ?? ()
17 LWP 10355 "rpc worker-1035" 0x00007f7e6cec3ad3 in ?? ()
18 LWP 10356 "rpc worker-1035" 0x00007f7e6cec3ad3 in ?? ()
19 LWP 10357 "rpc worker-1035" 0x00007f7e6cec3ad3 in ?? ()
20 LWP 10358 "rpc worker-1035" 0x00007f7e6cec3ad3 in ?? ()
21 LWP 10359 "rpc worker-1035" 0x00007f7e6cec3ad3 in ?? ()
22 LWP 10360 "rpc worker-1036" 0x00007f7e6cec3ad3 in ?? ()
23 LWP 10361 "rpc worker-1036" 0x00007f7e6cec3ad3 in ?? ()
24 LWP 10362 "rpc worker-1036" 0x00007f7e6cec3ad3 in ?? ()
25 LWP 10363 "rpc worker-1036" 0x00007f7e6cec3ad3 in ?? ()
26 LWP 10364 "rpc worker-1036" 0x00007f7e6cec3ad3 in ?? ()
27 LWP 10365 "rpc worker-1036" 0x00007f7e6cec3ad3 in ?? ()
28 LWP 10366 "rpc worker-1036" 0x00007f7e6cec3ad3 in ?? ()
29 LWP 10367 "rpc worker-1036" 0x00007f7e6cec3ad3 in ?? ()
30 LWP 10368 "rpc worker-1036" 0x00007f7e6cec3ad3 in ?? ()
31 LWP 10369 "rpc worker-1036" 0x00007f7e6cec3ad3 in ?? ()
32 LWP 10370 "rpc worker-1037" 0x00007f7e6cec3ad3 in ?? ()
33 LWP 10371 "rpc worker-1037" 0x00007f7e6cec3ad3 in ?? ()
34 LWP 10372 "rpc worker-1037" 0x00007f7e6cec3ad3 in ?? ()
35 LWP 10373 "rpc worker-1037" 0x00007f7e6cec3ad3 in ?? ()
36 LWP 10374 "rpc worker-1037" 0x00007f7e6cec3ad3 in ?? ()
37 LWP 10375 "rpc worker-1037" 0x00007f7e6cec3ad3 in ?? ()
38 LWP 10376 "rpc worker-1037" 0x00007f7e6cec3ad3 in ?? ()
39 LWP 10377 "rpc worker-1037" 0x00007f7e6cec3ad3 in ?? ()
40 LWP 10378 "rpc worker-1037" 0x00007f7e6cec3ad3 in ?? ()
41 LWP 10379 "rpc worker-1037" 0x00007f7e6cec3ad3 in ?? ()
42 LWP 10380 "rpc worker-1038" 0x00007f7e6cec3ad3 in ?? ()
43 LWP 10381 "rpc worker-1038" 0x00007f7e6cec3ad3 in ?? ()
44 LWP 10382 "rpc worker-1038" 0x00007f7e6cec3ad3 in ?? ()
45 LWP 10383 "rpc worker-1038" 0x00007f7e6cec3ad3 in ?? ()
46 LWP 10384 "rpc worker-1038" 0x00007f7e6cec3ad3 in ?? ()
47 LWP 10385 "rpc worker-1038" 0x00007f7e6cec3ad3 in ?? ()
48 LWP 10386 "rpc worker-1038" 0x00007f7e6cec3ad3 in ?? ()
49 LWP 10387 "rpc worker-1038" 0x00007f7e6cec3ad3 in ?? ()
50 LWP 10388 "rpc worker-1038" 0x00007f7e6cec3ad3 in ?? ()
51 LWP 10389 "rpc worker-1038" 0x00007f7e6cec3ad3 in ?? ()
52 LWP 10390 "rpc worker-1039" 0x00007f7e6cec3ad3 in ?? ()
53 LWP 10391 "rpc worker-1039" 0x00007f7e6cec3ad3 in ?? ()
54 LWP 10392 "rpc worker-1039" 0x00007f7e6cec3ad3 in ?? ()
55 LWP 10393 "rpc worker-1039" 0x00007f7e6cec3ad3 in ?? ()
56 LWP 10394 "rpc worker-1039" 0x00007f7e6cec3ad3 in ?? ()
57 LWP 10395 "rpc worker-1039" 0x00007f7e6cec3ad3 in ?? ()
58 LWP 10396 "rpc worker-1039" 0x00007f7e6cec3ad3 in ?? ()
59 LWP 10397 "rpc worker-1039" 0x00007f7e6cec3ad3 in ?? ()
60 LWP 10398 "rpc worker-1039" 0x00007f7e6cec3ad3 in ?? ()
61 LWP 10399 "rpc worker-1039" 0x00007f7e6cec3ad3 in ?? ()
62 LWP 10400 "rpc worker-1040" 0x00007f7e6cec3ad3 in ?? ()
63 LWP 10401 "rpc worker-1040" 0x00007f7e6cec3ad3 in ?? ()
64 LWP 10402 "rpc worker-1040" 0x00007f7e6cec3ad3 in ?? ()
65 LWP 10403 "rpc worker-1040" 0x00007f7e6cec3ad3 in ?? ()
66 LWP 10404 "rpc worker-1040" 0x00007f7e6cec3ad3 in ?? ()
67 LWP 10405 "rpc worker-1040" 0x00007f7e6cec3ad3 in ?? ()
68 LWP 10406 "rpc worker-1040" 0x00007f7e6cec3ad3 in ?? ()
69 LWP 10407 "rpc worker-1040" 0x00007f7e6cec3ad3 in ?? ()
70 LWP 10408 "rpc worker-1040" 0x00007f7e6cec3ad3 in ?? ()
71 LWP 10409 "rpc worker-1040" 0x00007f7e6cec3ad3 in ?? ()
72 LWP 10410 "rpc worker-1041" 0x00007f7e6cec3ad3 in ?? ()
73 LWP 10411 "rpc worker-1041" 0x00007f7e6cec3ad3 in ?? ()
74 LWP 10412 "rpc worker-1041" 0x00007f7e6cec3ad3 in ?? ()
75 LWP 10413 "rpc worker-1041" 0x00007f7e6cec3ad3 in ?? ()
76 LWP 10414 "rpc worker-1041" 0x00007f7e6cec3ad3 in ?? ()
77 LWP 10415 "rpc worker-1041" 0x00007f7e6cec3ad3 in ?? ()
78 LWP 10416 "rpc worker-1041" 0x00007f7e6cec3ad3 in ?? ()
79 LWP 10417 "rpc worker-1041" 0x00007f7e6cec3ad3 in ?? ()
80 LWP 10418 "rpc worker-1041" 0x00007f7e6cec3ad3 in ?? ()
81 LWP 10419 "rpc worker-1041" 0x00007f7e6cec3ad3 in ?? ()
82 LWP 10420 "rpc worker-1042" 0x00007f7e6cec3ad3 in ?? ()
83 LWP 10421 "rpc worker-1042" 0x00007f7e6cec3ad3 in ?? ()
84 LWP 10422 "rpc worker-1042" 0x00007f7e6cec3ad3 in ?? ()
85 LWP 10423 "rpc worker-1042" 0x00007f7e6cec3ad3 in ?? ()
86 LWP 10424 "rpc worker-1042" 0x00007f7e6cec3ad3 in ?? ()
87 LWP 10425 "rpc worker-1042" 0x00007f7e6cec3ad3 in ?? ()
88 LWP 10426 "rpc worker-1042" 0x00007f7e6cec3ad3 in ?? ()
89 LWP 10427 "rpc worker-1042" 0x00007f7e6cec3ad3 in ?? ()
90 LWP 10428 "rpc worker-1042" 0x00007f7e6cec3ad3 in ?? ()
91 LWP 10429 "rpc worker-1042" 0x00007f7e6cec3ad3 in ?? ()
92 LWP 10430 "rpc worker-1043" 0x00007f7e6cec3ad3 in ?? ()
93 LWP 10431 "rpc worker-1043" 0x00007f7e6cec3ad3 in ?? ()
94 LWP 10432 "rpc worker-1043" 0x00007f7e6cec3ad3 in ?? ()
95 LWP 10433 "rpc worker-1043" 0x00007f7e6cec3ad3 in ?? ()
96 LWP 10434 "rpc worker-1043" 0x00007f7e6cec3ad3 in ?? ()
97 LWP 10435 "rpc worker-1043" 0x00007f7e6cec3ad3 in ?? ()
98 LWP 10436 "rpc worker-1043" 0x00007f7e6cec3ad3 in ?? ()
99 LWP 10437 "rpc worker-1043" 0x00007f7e6cec3ad3 in ?? ()
100 LWP 10438 "rpc worker-1043" 0x00007f7e6cec3ad3 in ?? ()
101 LWP 10439 "rpc worker-1043" 0x00007f7e6cec3ad3 in ?? ()
102 LWP 10440 "rpc worker-1044" 0x00007f7e6cec3ad3 in ?? ()
103 LWP 10441 "rpc worker-1044" 0x00007f7e6cec3ad3 in ?? ()
104 LWP 10442 "rpc worker-1044" 0x00007f7e6cec3ad3 in ?? ()
105 LWP 10443 "rpc worker-1044" 0x00007f7e6cec3ad3 in ?? ()
106 LWP 10444 "rpc worker-1044" 0x00007f7e6cec3ad3 in ?? ()
107 LWP 10445 "rpc worker-1044" 0x00007f7e6cec3ad3 in ?? ()
108 LWP 10446 "rpc worker-1044" 0x00007f7e6cec3ad3 in ?? ()
109 LWP 10447 "rpc worker-1044" 0x00007f7e6cec3ad3 in ?? ()
110 LWP 10448 "rpc worker-1044" 0x00007f7e6cec3ad3 in ?? ()
111 LWP 10449 "rpc worker-1044" 0x00007f7e6cec3ad3 in ?? ()
112 LWP 10450 "rpc worker-1045" 0x00007f7e6cec3ad3 in ?? ()
113 LWP 10451 "rpc worker-1045" 0x00007f7e6cec3ad3 in ?? ()
114 LWP 10452 "rpc worker-1045" 0x00007f7e6cec3ad3 in ?? ()
115 LWP 10454 "rpc worker-1045" 0x00007f7e6cec3ad3 in ?? ()
116 LWP 10455 "rpc worker-1045" 0x00007f7e6cec3ad3 in ?? ()
117 LWP 10457 "diag-logger-104" 0x00007f7e6cec3fb9 in ?? ()
118 LWP 10458 "result-tracker-" 0x00007f7e6cec3fb9 in ?? ()
119 LWP 10459 "excess-log-dele" 0x00007f7e6cec3fb9 in ?? ()
120 LWP 10460 "acceptor-10460" 0x00007f7e65c360c7 in ?? ()
121 LWP 10461 "heartbeat-10461" 0x00007f7e6cec3fb9 in ?? ()
122 LWP 10465 "maintenance_sch" 0x00007f7e6cec3fb9 in ?? ()
123 LWP 12470 "raft [worker]-1" 0x00007f7e6cec3fb9 in ?? ()
Thread 123 (LWP 12470):
#0 0x00007f7e6cec3fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000038 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007f7e1b0e7b20 in ?? ()
#5 0x00007f7e1b0e77b0 in ?? ()
#6 0x0000000000000070 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 10465):
#0 0x00007f7e6cec3fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000002be in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x0000616000017584 in ?? ()
#5 0x00007f7e2173e550 in ?? ()
#6 0x000000000000057d in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f7e2173e570 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0xffffffff0000006e in ?? ()
#12 0x00007f7e2173f030 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 121 (LWP 10461):
#0 0x00007f7e6cec3fb9 in ?? ()
#1 0x81afaaa989cdd16b in ?? ()
#2 0x00000000000000b1 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x000061300001aac4 in ?? ()
#5 0x00007f7e2276e5b0 in ?? ()
#6 0x0000000000000163 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f7e2276e5d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007f7e2276e650 in ?? ()
#12 0x00000fefc44edccc in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 120 (LWP 10460):
#0 0x00007f7e65c360c7 in ?? ()
#1 0x00006140000218a8 in ?? ()
#2 0x00007f7e22f95d50 in ?? ()
#3 0x00007f7e22f95d80 in ?? ()
#4 0x00007f7e22f95e80 in ?? ()
#5 0x00007f7e22f95d70 in ?? ()
#6 0x00007f7e22f95de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007f7e69abaef0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 10459):
#0 0x00007f7e6cec3fb9 in ?? ()
#1 0x00007f7e237aef00 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffda438c670 in ?? ()
#5 0x00007f7e237aeef0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 10458):
#0 0x00007f7e6cec3fb9 in ?? ()
#1 0x00007f7e23fc80c0 in ?? ()
#2 0x00000000000000af in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007f7e23fc80b0 in ?? ()
#6 0x000000000000015e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 10457):
#0 0x00007f7e6cec3fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 10455):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000020 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002fd008 in ?? ()
#4 0x00007f7e24ff7e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7e24ff7e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 10454):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000024 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002f6808 in ?? ()
#4 0x00007f7e2580fe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7e2580fe70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 10452):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 10451):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 10450):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 10449):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 10448):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 10447):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 10446):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 10445):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 10444):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 10443):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 10442):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 10441):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 10440):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 10439):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 10438):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 10437):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 10436):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 10435):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 10434):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 10433):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 10432):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 10431):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 10430):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 10429):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 10428):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 10427):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 10426):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 10425):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 10424):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 10423):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 10422):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 10421):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 10420):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 10419):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 10418):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 10417):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 10416):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 10415):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 10414):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000001604 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001d6858 in ?? ()
#4 0x00007f7e393c8e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7e393c8e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 10413):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000001774 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001d0058 in ?? ()
#4 0x00007f7e39be0e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7e39be0e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 10412):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001cd068 in ?? ()
#4 0x00007f7e3a3f8e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7e3a3f8e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 73 (LWP 10411):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 10410):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 10409):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 10408):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 10407):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 10406):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 10405):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 10404):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 10403):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 10402):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 10401):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 10400):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 10399):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 10398):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 10397):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 10396):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 10395):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 10394):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00014681c in ?? ()
#4 0x00007f7e435a9e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7e435a9e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001467d0 in ?? ()
#9 0x00007f7e6cec3770 in ?? ()
#10 0x00007f7e435a9e70 in ?? ()
#11 0x00000fefc86b53d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 10393):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 10392):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 10391):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 10390):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 10389):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 10388):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 10387):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 10386):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 10385):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 10384):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 10383):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 10382):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 10381):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 10380):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 10379):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 10378):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 10377):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 10376):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 10375):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 10374):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000001c33 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000bffcc in ?? ()
#4 0x00007f7e4cf72e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7e4cf72e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000bff80 in ?? ()
#9 0x00007f7e6cec3770 in ?? ()
#10 0x00007f7e4cf72e70 in ?? ()
#11 0x00000fefc99ee5d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 10373):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000001783 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000b67dc in ?? ()
#4 0x00007f7e4d78ae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7e4d78ae70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000b6790 in ?? ()
#9 0x00007f7e6cec3770 in ?? ()
#10 0x00007f7e4d78ae70 in ?? ()
#11 0x00000fefc9af15d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 34 (LWP 10372):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000929 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000acfec in ?? ()
#4 0x00007f7e4dfa2e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7e4dfa2e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000acfa0 in ?? ()
#9 0x00007f7e6cec3770 in ?? ()
#10 0x00007f7e4dfa2e70 in ?? ()
#11 0x00000fefc9bf45d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 33 (LWP 10371):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000001e72 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00009ffe8 in ?? ()
#4 0x00007f7e4e7bae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7e4e7bae70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 32 (LWP 10370):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000001e40 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000967f8 in ?? ()
#4 0x00007f7e4efd2e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7e4efd2e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 31 (LWP 10369):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x00000000000019d6 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00008fff8 in ?? ()
#4 0x00007f7e4f7eae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7e4f7eae70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 30 (LWP 10368):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000baf in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00008d00c in ?? ()
#4 0x00007f7e50002e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7e50002e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00008cfc0 in ?? ()
#9 0x00007f7e6cec3770 in ?? ()
#10 0x00007f7e50002e70 in ?? ()
#11 0x00000fefca0005d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 29 (LWP 10367):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 10366):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 10365):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 10364):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 10363):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 10362):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 10361):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 10360):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 10359):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 10358):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 10357):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 10356):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 10355):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 10354):
#0 0x00007f7e6cec3fb9 in ?? ()
#1 0x00007f7e5716dc80 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ba60 in ?? ()
#5 0x00007f7e5716dc70 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 10353):
#0 0x00007f7e6cec3fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 10351):
#0 0x00007f7e6cec3fb9 in ?? ()
#1 0x00007f7e58a021c0 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007f7e58a021b0 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 10350):
#0 0x00007f7e6cec3ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007f7e59229b24 in ?? ()
#4 0x00007f7e59229830 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f7e59229850 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00006160000177c0 in ?? ()
#9 0x00007f7e6cec3770 in ?? ()
#10 0x00007f7e59229850 in ?? ()
#11 0x00007f7e59229850 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 12 (LWP 10349):
#0 0x00007f7e65c34a47 in ?? ()
#1 0x00007f7e59a512e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007f7e59a512d0 in ?? ()
#4 0x00007f7e59a514e0 in ?? ()
#5 0x00007f7e59a51320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007f7e59a513a0 in ?? ()
#8 0x00007f7e6829425d in ?? ()
#9 0x3f9da7fe9e160000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x00000000f8500000 in ?? ()
#14 0x00007f7e00000000 in ?? ()
#15 0x41d9e21ced720496 in ?? ()
#16 0x00000ff04b342274 in ?? ()
#17 0x00007f7e59a51380 in ?? ()
#18 0x00007f7e68298ba3 in ?? ()
#19 0x00007f7e59a51350 in ?? ()
#20 0x3f9da7fe9e160000 in ?? ()
#21 0x0000000059a513a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3f9da7fe9e160000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 10348):
#0 0x00007f7e65c34a47 in ?? ()
#1 0x00007f7e5a2682e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007f7e5a2682d0 in ?? ()
#4 0x00007f7e5a2684e0 in ?? ()
#5 0x00007f7e5a268320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007f7e5a2683a0 in ?? ()
#8 0x00007f7e6829425d in ?? ()
#9 0x3fb950846bd00000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x00000000fb000000 in ?? ()
#14 0x00007f7e00000000 in ?? ()
#15 0x41d9e21ced720498 in ?? ()
#16 0x00000ff04b445074 in ?? ()
#17 0x00007f7e5a268380 in ?? ()
#18 0x00007f7e68298ba3 in ?? ()
#19 0x00007f7e5a268350 in ?? ()
#20 0x3fb950846bd00000 in ?? ()
#21 0x000000005a2683a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb950846bd00000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 10347):
#0 0x00007f7e65c34a47 in ?? ()
#1 0x00007f7e5aa7f2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007f7e5aa7f2d0 in ?? ()
#4 0x00007f7e5aa7f4e0 in ?? ()
#5 0x00007f7e5aa7f320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007f7e5aa7f3a0 in ?? ()
#8 0x00007f7e6829425d in ?? ()
#9 0x3f9e334bbb220000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000098000000 in ?? ()
#14 0x00007f7e00000000 in ?? ()
#15 0x41d9e21ced720497 in ?? ()
#16 0x00000ff04b547e74 in ?? ()
#17 0x00007f7e5aa7f380 in ?? ()
#18 0x00007f7e68298ba3 in ?? ()
#19 0x00007f7e5aa7f350 in ?? ()
#20 0x3f9e334bbb220000 in ?? ()
#21 0x000000005aa7f3a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3f9e334bbb220000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 10346):
#0 0x00007f7e65c34a47 in ?? ()
#1 0x00007f7e5caa52e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007f7e5caa52d0 in ?? ()
#4 0x00007f7e5caa54e0 in ?? ()
#5 0x00007f7e5caa5320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f7e5caa53a0 in ?? ()
#8 0x00007f7e6829425d in ?? ()
#9 0x3fb98ed4e2cfa000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x0000000035000000 in ?? ()
#14 0x00007f7e00000000 in ?? ()
#15 0x41d9e21ced720498 in ?? ()
#16 0x00000ff04b94ca74 in ?? ()
#17 0x00007f7e5caa5380 in ?? ()
#18 0x00007f7e68298ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 10343):
#0 0x00007f7e65c27cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f7e5e4b7788 in ?? ()
#3 0x0000602000017cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 10342):
#0 0x00007f7e6cec3fb9 in ?? ()
#1 0x00007f7e5dcb6080 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007f7e5dcb6070 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 10341):
#0 0x00007f7e6cec79e2 in ?? ()
#1 0x00007f7e5d4b3b60 in ?? ()
#2 0x00007f7e5d4b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f7e5d4b3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007f7e5d4b3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007f7e5d4b3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007f7e727ebf0c in ?? ()
#12 0x00007f7e727db7b9 in ?? ()
#13 0x00007f7e727dfda6 in ?? ()
#14 0x00007f7e727e47e0 in ?? ()
#15 0x00007f7e727e4609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007f7e69adb98e in ?? ()
#18 0x00007f7e6cebd6db in ?? ()
#19 0x00000ff04ba8e67c in ?? ()
#20 0x00007f7e5d4b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 10335):
#0 0x00007f7e6cec3fb9 in ?? ()
#1 0x00007f7e5f4b8c40 in ?? ()
#2 0x0000000000000370 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007f7e5f4b8c30 in ?? ()
#6 0x00000000000006e0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 10333):
#0 0x00007f7e6cec3fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 10332):
#0 0x00007f7e6cec3fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 10331):
#0 0x00007f7e6cec3fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f7e60cbc410 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 10327):
#0 0x00007f7e6cec7d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250116 02:58:04.132781 8081 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID 660967712270485c81d6a5f1239c89b1 and pid 10469
************************ BEGIN STACKS **************************
[New LWP 10472]
[New LWP 10473]
[New LWP 10474]
[New LWP 10475]
[New LWP 10481]
[New LWP 10482]
[New LWP 10483]
[New LWP 10487]
[New LWP 10488]
[New LWP 10489]
[New LWP 10490]
[New LWP 10491]
[New LWP 10492]
[New LWP 10493]
[New LWP 10494]
[New LWP 10495]
[New LWP 10496]
[New LWP 10497]
[New LWP 10498]
[New LWP 10499]
[New LWP 10500]
[New LWP 10501]
[New LWP 10502]
[New LWP 10503]
[New LWP 10504]
[New LWP 10505]
[New LWP 10506]
[New LWP 10507]
[New LWP 10508]
[New LWP 10509]
[New LWP 10510]
[New LWP 10511]
[New LWP 10512]
[New LWP 10513]
[New LWP 10514]
[New LWP 10516]
[New LWP 10517]
[New LWP 10518]
[New LWP 10519]
[New LWP 10520]
[New LWP 10521]
[New LWP 10522]
[New LWP 10523]
[New LWP 10524]
[New LWP 10525]
[New LWP 10526]
[New LWP 10527]
[New LWP 10528]
[New LWP 10529]
[New LWP 10530]
[New LWP 10531]
[New LWP 10532]
[New LWP 10533]
[New LWP 10534]
[New LWP 10535]
[New LWP 10536]
[New LWP 10538]
[New LWP 10539]
[New LWP 10540]
[New LWP 10541]
[New LWP 10542]
[New LWP 10543]
[New LWP 10544]
[New LWP 10545]
[New LWP 10546]
[New LWP 10547]
[New LWP 10548]
[New LWP 10549]
[New LWP 10550]
[New LWP 10551]
[New LWP 10552]
[New LWP 10553]
[New LWP 10554]
[New LWP 10555]
[New LWP 10556]
[New LWP 10557]
[New LWP 10558]
[New LWP 10559]
[New LWP 10560]
[New LWP 10561]
[New LWP 10562]
[New LWP 10563]
[New LWP 10564]
[New LWP 10565]
[New LWP 10566]
[New LWP 10567]
[New LWP 10568]
[New LWP 10569]
[New LWP 10570]
[New LWP 10571]
[New LWP 10572]
[New LWP 10573]
[New LWP 10574]
[New LWP 10575]
[New LWP 10576]
[New LWP 10577]
[New LWP 10578]
[New LWP 10579]
[New LWP 10580]
[New LWP 10581]
[New LWP 10582]
[New LWP 10583]
[New LWP 10584]
[New LWP 10585]
[New LWP 10586]
[New LWP 10587]
[New LWP 10588]
[New LWP 10589]
[New LWP 10590]
[New LWP 10591]
[New LWP 10592]
[New LWP 10593]
[New LWP 10594]
[New LWP 10595]
[New LWP 10596]
[New LWP 10597]
[New LWP 10598]
[New LWP 10599]
[New LWP 10600]
[New LWP 10601]
[New LWP 10602]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007fafe4d80d50 in ?? ()
Id Target Id Frame
* 1 LWP 10469 "kudu" 0x00007fafe4d80d50 in ?? ()
2 LWP 10472 "kudu" 0x00007fafe4d7cfb9 in ?? ()
3 LWP 10473 "kudu" 0x00007fafe4d7cfb9 in ?? ()
4 LWP 10474 "kudu" 0x00007fafe4d7cfb9 in ?? ()
5 LWP 10475 "kernel-watcher-" 0x00007fafe4d7cfb9 in ?? ()
6 LWP 10481 "ntp client-1048" 0x00007fafe4d809e2 in ?? ()
7 LWP 10482 "file cache-evic" 0x00007fafe4d7cfb9 in ?? ()
8 LWP 10483 "sq_acceptor" 0x00007fafddae0cb9 in ?? ()
9 LWP 10487 "rpc reactor-104" 0x00007fafddaeda47 in ?? ()
10 LWP 10488 "rpc reactor-104" 0x00007fafddaeda47 in ?? ()
11 LWP 10489 "rpc reactor-104" 0x00007fafddaeda47 in ?? ()
12 LWP 10490 "rpc reactor-104" 0x00007fafddaeda47 in ?? ()
13 LWP 10491 "MaintenanceMgr " 0x00007fafe4d7cad3 in ?? ()
14 LWP 10492 "txn-status-mana" 0x00007fafe4d7cfb9 in ?? ()
15 LWP 10493 "collect_and_rem" 0x00007fafe4d7cfb9 in ?? ()
16 LWP 10494 "tc-session-exp-" 0x00007fafe4d7cfb9 in ?? ()
17 LWP 10495 "rpc worker-1049" 0x00007fafe4d7cad3 in ?? ()
18 LWP 10496 "rpc worker-1049" 0x00007fafe4d7cad3 in ?? ()
19 LWP 10497 "rpc worker-1049" 0x00007fafe4d7cad3 in ?? ()
20 LWP 10498 "rpc worker-1049" 0x00007fafe4d7cad3 in ?? ()
21 LWP 10499 "rpc worker-1049" 0x00007fafe4d7cad3 in ?? ()
22 LWP 10500 "rpc worker-1050" 0x00007fafe4d7cad3 in ?? ()
23 LWP 10501 "rpc worker-1050" 0x00007fafe4d7cad3 in ?? ()
24 LWP 10502 "rpc worker-1050" 0x00007fafe4d7cad3 in ?? ()
25 LWP 10503 "rpc worker-1050" 0x00007fafe4d7cad3 in ?? ()
26 LWP 10504 "rpc worker-1050" 0x00007fafe4d7cad3 in ?? ()
27 LWP 10505 "rpc worker-1050" 0x00007fafe4d7cad3 in ?? ()
28 LWP 10506 "rpc worker-1050" 0x00007fafe4d7cad3 in ?? ()
29 LWP 10507 "rpc worker-1050" 0x00007fafe4d7cad3 in ?? ()
30 LWP 10508 "rpc worker-1050" 0x00007fafe4d7cad3 in ?? ()
31 LWP 10509 "rpc worker-1050" 0x00007fafe4d7cad3 in ?? ()
32 LWP 10510 "rpc worker-1051" 0x00007fafe4d7cad3 in ?? ()
33 LWP 10511 "rpc worker-1051" 0x00007fafe4d7cad3 in ?? ()
34 LWP 10512 "rpc worker-1051" 0x00007fafe4d7cad3 in ?? ()
35 LWP 10513 "rpc worker-1051" 0x00007fafe4d7cad3 in ?? ()
36 LWP 10514 "rpc worker-1051" 0x00007fafe4d7cad3 in ?? ()
37 LWP 10516 "rpc worker-1051" 0x00007fafe4d7cad3 in ?? ()
38 LWP 10517 "rpc worker-1051" 0x00007fafe4d7cad3 in ?? ()
39 LWP 10518 "rpc worker-1051" 0x00007fafe4d7cad3 in ?? ()
40 LWP 10519 "rpc worker-1051" 0x00007fafe4d7cad3 in ?? ()
41 LWP 10520 "rpc worker-1052" 0x00007fafe4d7cad3 in ?? ()
42 LWP 10521 "rpc worker-1052" 0x00007fafe4d7cad3 in ?? ()
43 LWP 10522 "rpc worker-1052" 0x00007fafe4d7cad3 in ?? ()
44 LWP 10523 "rpc worker-1052" 0x00007fafe4d7cad3 in ?? ()
45 LWP 10524 "rpc worker-1052" 0x00007fafe4d7cad3 in ?? ()
46 LWP 10525 "rpc worker-1052" 0x00007fafe4d7cad3 in ?? ()
47 LWP 10526 "rpc worker-1052" 0x00007fafe4d7cad3 in ?? ()
48 LWP 10527 "rpc worker-1052" 0x00007fafe4d7cad3 in ?? ()
49 LWP 10528 "rpc worker-1052" 0x00007fafe4d7cad3 in ?? ()
50 LWP 10529 "rpc worker-1052" 0x00007fafe4d7cad3 in ?? ()
51 LWP 10530 "rpc worker-1053" 0x00007fafe4d7cad3 in ?? ()
52 LWP 10531 "rpc worker-1053" 0x00007fafe4d7cad3 in ?? ()
53 LWP 10532 "rpc worker-1053" 0x00007fafe4d7cad3 in ?? ()
54 LWP 10533 "rpc worker-1053" 0x00007fafe4d7cad3 in ?? ()
55 LWP 10534 "rpc worker-1053" 0x00007fafe4d7cad3 in ?? ()
56 LWP 10535 "rpc worker-1053" 0x00007fafe4d7cad3 in ?? ()
57 LWP 10536 "rpc worker-1053" 0x00007fafe4d7cad3 in ?? ()
58 LWP 10538 "rpc worker-1053" 0x00007fafe4d7cad3 in ?? ()
59 LWP 10539 "rpc worker-1053" 0x00007fafe4d7cad3 in ?? ()
60 LWP 10540 "rpc worker-1054" 0x00007fafe4d7cad3 in ?? ()
61 LWP 10541 "rpc worker-1054" 0x00007fafe4d7cad3 in ?? ()
62 LWP 10542 "rpc worker-1054" 0x00007fafe4d7cad3 in ?? ()
63 LWP 10543 "rpc worker-1054" 0x00007fafe4d7cad3 in ?? ()
64 LWP 10544 "rpc worker-1054" 0x00007fafe4d7cad3 in ?? ()
65 LWP 10545 "rpc worker-1054" 0x00007fafe4d7cad3 in ?? ()
66 LWP 10546 "rpc worker-1054" 0x00007fafe4d7cad3 in ?? ()
67 LWP 10547 "rpc worker-1054" 0x00007fafe4d7cad3 in ?? ()
68 LWP 10548 "rpc worker-1054" 0x00007fafe4d7cad3 in ?? ()
69 LWP 10549 "rpc worker-1054" 0x00007fafe4d7cad3 in ?? ()
70 LWP 10550 "rpc worker-1055" 0x00007fafe4d7cad3 in ?? ()
71 LWP 10551 "rpc worker-1055" 0x00007fafe4d7cad3 in ?? ()
72 LWP 10552 "rpc worker-1055" 0x00007fafe4d7cad3 in ?? ()
73 LWP 10553 "rpc worker-1055" 0x00007fafe4d7cad3 in ?? ()
74 LWP 10554 "rpc worker-1055" 0x00007fafe4d7cad3 in ?? ()
75 LWP 10555 "rpc worker-1055" 0x00007fafe4d7cad3 in ?? ()
76 LWP 10556 "rpc worker-1055" 0x00007fafe4d7cad3 in ?? ()
77 LWP 10557 "rpc worker-1055" 0x00007fafe4d7cad3 in ?? ()
78 LWP 10558 "rpc worker-1055" 0x00007fafe4d7cad3 in ?? ()
79 LWP 10559 "rpc worker-1055" 0x00007fafe4d7cad3 in ?? ()
80 LWP 10560 "rpc worker-1056" 0x00007fafe4d7cad3 in ?? ()
81 LWP 10561 "rpc worker-1056" 0x00007fafe4d7cad3 in ?? ()
82 LWP 10562 "rpc worker-1056" 0x00007fafe4d7cad3 in ?? ()
83 LWP 10563 "rpc worker-1056" 0x00007fafe4d7cad3 in ?? ()
84 LWP 10564 "rpc worker-1056" 0x00007fafe4d7cad3 in ?? ()
85 LWP 10565 "rpc worker-1056" 0x00007fafe4d7cad3 in ?? ()
86 LWP 10566 "rpc worker-1056" 0x00007fafe4d7cad3 in ?? ()
87 LWP 10567 "rpc worker-1056" 0x00007fafe4d7cad3 in ?? ()
88 LWP 10568 "rpc worker-1056" 0x00007fafe4d7cad3 in ?? ()
89 LWP 10569 "rpc worker-1056" 0x00007fafe4d7cad3 in ?? ()
90 LWP 10570 "rpc worker-1057" 0x00007fafe4d7cad3 in ?? ()
91 LWP 10571 "rpc worker-1057" 0x00007fafe4d7cad3 in ?? ()
92 LWP 10572 "rpc worker-1057" 0x00007fafe4d7cad3 in ?? ()
93 LWP 10573 "rpc worker-1057" 0x00007fafe4d7cad3 in ?? ()
94 LWP 10574 "rpc worker-1057" 0x00007fafe4d7cad3 in ?? ()
95 LWP 10575 "rpc worker-1057" 0x00007fafe4d7cad3 in ?? ()
96 LWP 10576 "rpc worker-1057" 0x00007fafe4d7cad3 in ?? ()
97 LWP 10577 "rpc worker-1057" 0x00007fafe4d7cad3 in ?? ()
98 LWP 10578 "rpc worker-1057" 0x00007fafe4d7cad3 in ?? ()
99 LWP 10579 "rpc worker-1057" 0x00007fafe4d7cad3 in ?? ()
100 LWP 10580 "rpc worker-1058" 0x00007fafe4d7cad3 in ?? ()
101 LWP 10581 "rpc worker-1058" 0x00007fafe4d7cad3 in ?? ()
102 LWP 10582 "rpc worker-1058" 0x00007fafe4d7cad3 in ?? ()
103 LWP 10583 "rpc worker-1058" 0x00007fafe4d7cad3 in ?? ()
104 LWP 10584 "rpc worker-1058" 0x00007fafe4d7cad3 in ?? ()
105 LWP 10585 "rpc worker-1058" 0x00007fafe4d7cad3 in ?? ()
106 LWP 10586 "rpc worker-1058" 0x00007fafe4d7cad3 in ?? ()
107 LWP 10587 "rpc worker-1058" 0x00007fafe4d7cad3 in ?? ()
108 LWP 10588 "rpc worker-1058" 0x00007fafe4d7cad3 in ?? ()
109 LWP 10589 "rpc worker-1058" 0x00007fafe4d7cad3 in ?? ()
110 LWP 10590 "rpc worker-1059" 0x00007fafe4d7cad3 in ?? ()
111 LWP 10591 "rpc worker-1059" 0x00007fafe4d7cad3 in ?? ()
112 LWP 10592 "rpc worker-1059" 0x00007fafe4d7cad3 in ?? ()
113 LWP 10593 "rpc worker-1059" 0x00007fafe4d7cad3 in ?? ()
114 LWP 10594 "rpc worker-1059" 0x00007fafe4d7cad3 in ?? ()
115 LWP 10595 "rpc worker-1059" 0x00007fafe4d7cad3 in ?? ()
116 LWP 10596 "rpc worker-1059" 0x00007fafe4d7cad3 in ?? ()
117 LWP 10597 "diag-logger-105" 0x00007fafe4d7cfb9 in ?? ()
118 LWP 10598 "result-tracker-" 0x00007fafe4d7cfb9 in ?? ()
119 LWP 10599 "excess-log-dele" 0x00007fafe4d7cfb9 in ?? ()
120 LWP 10600 "acceptor-10600" 0x00007fafddaef0c7 in ?? ()
121 LWP 10601 "heartbeat-10601" 0x00007fafe4d7cfb9 in ?? ()
122 LWP 10602 "maintenance_sch" 0x00007fafe4d7cfb9 in ?? ()
Thread 122 (LWP 10602):
#0 0x00007fafe4d7cfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000002b7 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007faf9a667550 in ?? ()
#6 0x000000000000056e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 10601):
#0 0x00007fafe4d7cfb9 in ?? ()
#1 0x81afaaa989cdd16b in ?? ()
#2 0x00000000000000ad in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007faf9ae7f5b0 in ?? ()
#6 0x000000000000015a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 10600):
#0 0x00007fafddaef0c7 in ?? ()
#1 0x00006140000216a8 in ?? ()
#2 0x00007faf9b6a6d50 in ?? ()
#3 0x00007faf9b6a6d80 in ?? ()
#4 0x00007faf9b6a6e80 in ?? ()
#5 0x00007faf9b6a6d70 in ?? ()
#6 0x00007faf9b6a6de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007fafe1973ef0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 10599):
#0 0x00007fafe4d7cfb9 in ?? ()
#1 0x00007faf9bec0f00 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffe35555930 in ?? ()
#5 0x00007faf9bec0ef0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 10598):
#0 0x00007fafe4d7cfb9 in ?? ()
#1 0x00007faf9c6d90c0 in ?? ()
#2 0x00000000000000ae in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007faf9c6d90b0 in ?? ()
#6 0x000000000000015c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 10597):
#0 0x00007fafe4d7cfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 10596):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x000000000000001c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007faf9d708e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007faf9d708e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 10595):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000028 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002a6858 in ?? ()
#4 0x00007faf9df20e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007faf9df20e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 10594):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 10593):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 10592):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 10591):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 10590):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 10589):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 10588):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 10587):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 10586):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 10585):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 10584):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 10583):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 10582):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 10581):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 10580):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 10579):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 10578):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 10577):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 10576):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 10575):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 10574):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 10573):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 10572):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 10571):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 10570):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 10569):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 10568):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 10567):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 10566):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 10565):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 10564):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 10563):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 10562):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 10561):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 10560):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 10559):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 10558):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 10557):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 10556):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007fafb1ad9e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fafb1ad9e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007fafe4d7c770 in ?? ()
#10 0x00007fafb1ad9e70 in ?? ()
#11 0x00000ff5f635b3d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 10555):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 10554):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 10553):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 10552):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 10551):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 10550):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 10549):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 10548):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 10547):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 10546):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 10545):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 10544):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 10543):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 10542):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 10541):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 10540):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 10539):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 10538):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 10536):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 10535):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007fafbbcbae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fafbbcbae70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007fafe4d7c770 in ?? ()
#10 0x00007fafbbcbae70 in ?? ()
#11 0x00000ff5f77975d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 10534):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 10533):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 10532):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 10531):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 10530):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 10529):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 10528):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 10527):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 10526):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 10525):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 10524):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 10523):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 10522):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 10521):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 10520):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 10519):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 10518):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 10517):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 10516):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 10514):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007fafc5683e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fafc5683e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007fafe4d7c770 in ?? ()
#10 0x00007fafc5683e70 in ?? ()
#11 0x00000ff5f8ad07d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 10513):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 10512):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 10511):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 10510):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 10509):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 10508):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 10507):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 10506):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 10505):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 10504):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 10503):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 10502):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 10501):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 10500):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 10499):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 10498):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 10497):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 10496):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 10495):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 10494):
#0 0x00007fafe4d7cfb9 in ?? ()
#1 0x00007fafcf6c3c80 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001b360 in ?? ()
#5 0x00007fafcf6c3c70 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 10493):
#0 0x00007fafe4d7cfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 10492):
#0 0x00007fafe4d7cfb9 in ?? ()
#1 0x00007fafd06c51c0 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007fafd06c51b0 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 10491):
#0 0x00007fafe4d7cad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 10490):
#0 0x00007fafddaeda47 in ?? ()
#1 0x00007fafd16c82e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007fafd16c82d0 in ?? ()
#4 0x00007fafd16c84e0 in ?? ()
#5 0x00007fafd16c8320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007fafd16c83a0 in ?? ()
#8 0x00007fafe014d25d in ?? ()
#9 0x3fb96b9440f3c000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x00000000df000000 in ?? ()
#14 0x00007faf00000000 in ?? ()
#15 0x41d9e21ced720497 in ?? ()
#16 0x00000ff67a2d1074 in ?? ()
#17 0x00007fafd16c8380 in ?? ()
#18 0x00007fafe0151ba3 in ?? ()
#19 0x00007fafd16c8350 in ?? ()
#20 0x3fb96b9440f3c000 in ?? ()
#21 0x00000000d16c83a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb96b9440f3c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 10489):
#0 0x00007fafddaeda47 in ?? ()
#1 0x00007fafd1ec92e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007fafd1ec92d0 in ?? ()
#4 0x00007fafd1ec94e0 in ?? ()
#5 0x00007fafd1ec9320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007fafd1ec93a0 in ?? ()
#8 0x00007fafe014d25d in ?? ()
#9 0x3fb905f8b94bc000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x000000006a000000 in ?? ()
#14 0x00007faf00000000 in ?? ()
#15 0x41d9e21ced720499 in ?? ()
#16 0x00000ff67a3d1274 in ?? ()
#17 0x00007fafd1ec9380 in ?? ()
#18 0x00007fafe0151ba3 in ?? ()
#19 0x00007fafd1ec9350 in ?? ()
#20 0x3fb905f8b94bc000 in ?? ()
#21 0x00000000d1ec93a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb905f8b94bc000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 10488):
#0 0x00007fafddaeda47 in ?? ()
#1 0x00007fafd26ca2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007fafd26ca2d0 in ?? ()
#4 0x00007fafd26ca4e0 in ?? ()
#5 0x00007fafd26ca320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007fafd26ca3a0 in ?? ()
#8 0x00007fafe014d25d in ?? ()
#9 0x3fb900edaaada000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x00000000cc000000 in ?? ()
#14 0x00007faf00000000 in ?? ()
#15 0x41d9e21ced720497 in ?? ()
#16 0x00000ff67a4d1474 in ?? ()
#17 0x00007fafd26ca380 in ?? ()
#18 0x00007fafe0151ba3 in ?? ()
#19 0x00007fafd26ca350 in ?? ()
#20 0x3fb900edaaada000 in ?? ()
#21 0x00000000d26ca3a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb900edaaada000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 10487):
#0 0x00007fafddaeda47 in ?? ()
#1 0x00007fafd42b32e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007fafd42b32d0 in ?? ()
#4 0x00007fafd42b34e0 in ?? ()
#5 0x00007fafd42b3320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fafd42b33a0 in ?? ()
#8 0x00007fafe014d25d in ?? ()
#9 0x3fb963d7dab7a000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x00000000b6800000 in ?? ()
#14 0x00007faf00000000 in ?? ()
#15 0x41d9e21ced720499 in ?? ()
#16 0x00000ff67a84e674 in ?? ()
#17 0x00007fafd42b3380 in ?? ()
#18 0x00007fafe0151ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 10483):
#0 0x00007fafddae0cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fafd62b7788 in ?? ()
#3 0x0000602000017c90 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 10482):
#0 0x00007fafe4d7cfb9 in ?? ()
#1 0x00007fafd5ab6080 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007fafd5ab6070 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 10481):
#0 0x00007fafe4d809e2 in ?? ()
#1 0x00007fafd4ab2b60 in ?? ()
#2 0x00007fafd4ab2c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fafd4ab2bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007fafd4ab2c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007fafd4ab2c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007fafea6a4f0c in ?? ()
#12 0x00007fafea6947b9 in ?? ()
#13 0x00007fafea698da6 in ?? ()
#14 0x00007fafea69d7e0 in ?? ()
#15 0x00007fafea69d609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007fafe199498e in ?? ()
#18 0x00007fafe4d766db in ?? ()
#19 0x00000ff67a94e47c in ?? ()
#20 0x00007fafd4ab2400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 10475):
#0 0x00007fafe4d7cfb9 in ?? ()
#1 0x00007fafd72b8c40 in ?? ()
#2 0x000000000000036b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007fafd72b8c30 in ?? ()
#6 0x00000000000006d6 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 10474):
#0 0x00007fafe4d7cfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 10473):
#0 0x00007fafe4d7cfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 10472):
#0 0x00007fafe4d7cfb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fafd8abc410 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 10469):
#0 0x00007fafe4d80d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250116 02:58:05.356774 8081 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 3 with UUID 0791364971ba49f3b2813bdc36c11eda and pid 11823
************************ BEGIN STACKS **************************
[New LWP 11825]
[New LWP 11826]
[New LWP 11827]
[New LWP 11829]
[New LWP 11835]
[New LWP 11836]
[New LWP 11837]
[New LWP 11843]
[New LWP 11844]
[New LWP 11845]
[New LWP 11846]
[New LWP 11848]
[New LWP 11849]
[New LWP 11851]
[New LWP 11852]
[New LWP 11853]
[New LWP 11854]
[New LWP 11855]
[New LWP 11856]
[New LWP 11857]
[New LWP 11858]
[New LWP 11859]
[New LWP 11860]
[New LWP 11861]
[New LWP 11862]
[New LWP 11863]
[New LWP 11864]
[New LWP 11865]
[New LWP 11866]
[New LWP 11867]
[New LWP 11868]
[New LWP 11869]
[New LWP 11870]
[New LWP 11871]
[New LWP 11872]
[New LWP 11873]
[New LWP 11874]
[New LWP 11876]
[New LWP 11877]
[New LWP 11878]
[New LWP 11879]
[New LWP 11880]
[New LWP 11881]
[New LWP 11882]
[New LWP 11883]
[New LWP 11884]
[New LWP 11885]
[New LWP 11886]
[New LWP 11887]
[New LWP 11888]
[New LWP 11889]
[New LWP 11890]
[New LWP 11891]
[New LWP 11892]
[New LWP 11894]
[New LWP 11896]
[New LWP 11897]
[New LWP 11900]
[New LWP 11901]
[New LWP 11902]
[New LWP 11903]
[New LWP 11904]
[New LWP 11905]
[New LWP 11906]
[New LWP 11907]
[New LWP 11908]
[New LWP 11909]
[New LWP 11910]
[New LWP 11911]
[New LWP 11912]
[New LWP 11913]
[New LWP 11914]
[New LWP 11915]
[New LWP 11916]
[New LWP 11917]
[New LWP 11918]
[New LWP 11919]
[New LWP 11920]
[New LWP 11921]
[New LWP 11922]
[New LWP 11923]
[New LWP 11924]
[New LWP 11925]
[New LWP 11926]
[New LWP 11927]
[New LWP 11928]
[New LWP 11929]
[New LWP 11930]
[New LWP 11931]
[New LWP 11932]
[New LWP 11933]
[New LWP 11934]
[New LWP 11935]
[New LWP 11936]
[New LWP 11937]
[New LWP 11938]
[New LWP 11939]
[New LWP 11940]
[New LWP 11941]
[New LWP 11942]
[New LWP 11943]
[New LWP 11944]
[New LWP 11945]
[New LWP 11946]
[New LWP 11947]
[New LWP 11948]
[New LWP 11949]
[New LWP 11950]
[New LWP 11951]
[New LWP 11952]
[New LWP 11953]
[New LWP 11954]
[New LWP 11955]
[New LWP 11956]
[New LWP 11957]
[New LWP 11958]
[New LWP 11959]
[New LWP 11960]
[New LWP 11961]
[New LWP 11962]
[New LWP 11963]
[New LWP 12450]
[New LWP 12452]
[New LWP 12453]
[New LWP 12454]
[New LWP 12455]
[New LWP 12456]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007fc1c8f98d50 in ?? ()
Id Target Id Frame
* 1 LWP 11823 "kudu" 0x00007fc1c8f98d50 in ?? ()
2 LWP 11825 "kudu" 0x00007fc1c8f94fb9 in ?? ()
3 LWP 11826 "kudu" 0x00007fc1c8f94fb9 in ?? ()
4 LWP 11827 "kudu" 0x00007fc1c8f94fb9 in ?? ()
5 LWP 11829 "kernel-watcher-" 0x00007fc1c8f94fb9 in ?? ()
6 LWP 11835 "ntp client-1183" 0x00007fc1c8f989e2 in ?? ()
7 LWP 11836 "file cache-evic" 0x00007fc1c8f94fb9 in ?? ()
8 LWP 11837 "sq_acceptor" 0x00007fc1c1cf8cb9 in ?? ()
9 LWP 11843 "rpc reactor-118" 0x00007fc1c1d05a47 in ?? ()
10 LWP 11844 "rpc reactor-118" 0x00007fc1c1d05a47 in ?? ()
11 LWP 11845 "rpc reactor-118" 0x00007fc1c1d05a47 in ?? ()
12 LWP 11846 "rpc reactor-118" 0x00007fc1c1d05a47 in ?? ()
13 LWP 11848 "MaintenanceMgr " 0x00007fc1c8f94ad3 in ?? ()
14 LWP 11849 "txn-status-mana" 0x00007fc1c8f94fb9 in ?? ()
15 LWP 11851 "collect_and_rem" 0x00007fc1c8f94fb9 in ?? ()
16 LWP 11852 "tc-session-exp-" 0x00007fc1c8f94fb9 in ?? ()
17 LWP 11853 "rpc worker-1185" 0x00007fc1c8f94ad3 in ?? ()
18 LWP 11854 "rpc worker-1185" 0x00007fc1c8f94ad3 in ?? ()
19 LWP 11855 "rpc worker-1185" 0x00007fc1c8f94ad3 in ?? ()
20 LWP 11856 "rpc worker-1185" 0x00007fc1c8f94ad3 in ?? ()
21 LWP 11857 "rpc worker-1185" 0x00007fc1c8f94ad3 in ?? ()
22 LWP 11858 "rpc worker-1185" 0x00007fc1c8f94ad3 in ?? ()
23 LWP 11859 "rpc worker-1185" 0x00007fc1c8f94ad3 in ?? ()
24 LWP 11860 "rpc worker-1186" 0x00007fc1c8f94ad3 in ?? ()
25 LWP 11861 "rpc worker-1186" 0x00007fc1c8f94ad3 in ?? ()
26 LWP 11862 "rpc worker-1186" 0x00007fc1c8f94ad3 in ?? ()
27 LWP 11863 "rpc worker-1186" 0x00007fc1c8f94ad3 in ?? ()
28 LWP 11864 "rpc worker-1186" 0x00007fc1c8f94ad3 in ?? ()
29 LWP 11865 "rpc worker-1186" 0x00007fc1c8f94ad3 in ?? ()
30 LWP 11866 "rpc worker-1186" 0x00007fc1c8f94ad3 in ?? ()
31 LWP 11867 "rpc worker-1186" 0x00007fc1c8f94ad3 in ?? ()
32 LWP 11868 "rpc worker-1186" 0x00007fc1c8f94ad3 in ?? ()
33 LWP 11869 "rpc worker-1186" 0x00007fc1c8f94ad3 in ?? ()
34 LWP 11870 "rpc worker-1187" 0x00007fc1c8f94ad3 in ?? ()
35 LWP 11871 "rpc worker-1187" 0x00007fc1c8f94ad3 in ?? ()
36 LWP 11872 "rpc worker-1187" 0x00007fc1c8f94ad3 in ?? ()
37 LWP 11873 "rpc worker-1187" 0x00007fc1c8f94ad3 in ?? ()
38 LWP 11874 "rpc worker-1187" 0x00007fc1c8f94ad3 in ?? ()
39 LWP 11876 "rpc worker-1187" 0x00007fc1c8f94ad3 in ?? ()
40 LWP 11877 "rpc worker-1187" 0x00007fc1c8f94ad3 in ?? ()
41 LWP 11878 "rpc worker-1187" 0x00007fc1c8f94ad3 in ?? ()
42 LWP 11879 "rpc worker-1187" 0x00007fc1c8f94ad3 in ?? ()
43 LWP 11880 "rpc worker-1188" 0x00007fc1c8f94ad3 in ?? ()
44 LWP 11881 "rpc worker-1188" 0x00007fc1c8f94ad3 in ?? ()
45 LWP 11882 "rpc worker-1188" 0x00007fc1c8f94ad3 in ?? ()
46 LWP 11883 "rpc worker-1188" 0x00007fc1c8f94ad3 in ?? ()
47 LWP 11884 "rpc worker-1188" 0x00007fc1c8f94ad3 in ?? ()
48 LWP 11885 "rpc worker-1188" 0x00007fc1c8f94ad3 in ?? ()
49 LWP 11886 "rpc worker-1188" 0x00007fc1c8f94ad3 in ?? ()
50 LWP 11887 "rpc worker-1188" 0x00007fc1c8f94ad3 in ?? ()
51 LWP 11888 "rpc worker-1188" 0x00007fc1c8f94ad3 in ?? ()
52 LWP 11889 "rpc worker-1188" 0x00007fc1c8f94ad3 in ?? ()
53 LWP 11890 "rpc worker-1189" 0x00007fc1c8f94ad3 in ?? ()
54 LWP 11891 "rpc worker-1189" 0x00007fc1c8f94ad3 in ?? ()
55 LWP 11892 "rpc worker-1189" 0x00007fc1c8f94ad3 in ?? ()
56 LWP 11894 "rpc worker-1189" 0x00007fc1c8f94ad3 in ?? ()
57 LWP 11896 "rpc worker-1189" 0x00007fc1c8f94ad3 in ?? ()
58 LWP 11897 "rpc worker-1189" 0x00007fc1c8f94ad3 in ?? ()
59 LWP 11900 "rpc worker-1190" 0x00007fc1c8f94ad3 in ?? ()
60 LWP 11901 "rpc worker-1190" 0x00007fc1c8f94ad3 in ?? ()
61 LWP 11902 "rpc worker-1190" 0x00007fc1c8f94ad3 in ?? ()
62 LWP 11903 "rpc worker-1190" 0x00007fc1c8f94ad3 in ?? ()
63 LWP 11904 "rpc worker-1190" 0x00007fc1c8f94ad3 in ?? ()
64 LWP 11905 "rpc worker-1190" 0x00007fc1c8f94ad3 in ?? ()
65 LWP 11906 "rpc worker-1190" 0x00007fc1c8f94ad3 in ?? ()
66 LWP 11907 "rpc worker-1190" 0x00007fc1c8f94ad3 in ?? ()
67 LWP 11908 "rpc worker-1190" 0x00007fc1c8f94ad3 in ?? ()
68 LWP 11909 "rpc worker-1190" 0x00007fc1c8f94ad3 in ?? ()
69 LWP 11910 "rpc worker-1191" 0x00007fc1c8f94ad3 in ?? ()
70 LWP 11911 "rpc worker-1191" 0x00007fc1c8f94ad3 in ?? ()
71 LWP 11912 "rpc worker-1191" 0x00007fc1c8f94ad3 in ?? ()
72 LWP 11913 "rpc worker-1191" 0x00007fc1c8f94ad3 in ?? ()
73 LWP 11914 "rpc worker-1191" 0x00007fc1c8f94ad3 in ?? ()
74 LWP 11915 "rpc worker-1191" 0x00007fc1c8f94ad3 in ?? ()
75 LWP 11916 "rpc worker-1191" 0x00007fc1c8f94ad3 in ?? ()
76 LWP 11917 "rpc worker-1191" 0x00007fc1c8f94ad3 in ?? ()
77 LWP 11918 "rpc worker-1191" 0x00007fc1c8f94ad3 in ?? ()
78 LWP 11919 "rpc worker-1191" 0x00007fc1c8f94ad3 in ?? ()
79 LWP 11920 "rpc worker-1192" 0x00007fc1c8f94ad3 in ?? ()
80 LWP 11921 "rpc worker-1192" 0x00007fc1c8f94ad3 in ?? ()
81 LWP 11922 "rpc worker-1192" 0x00007fc1c8f94ad3 in ?? ()
82 LWP 11923 "rpc worker-1192" 0x00007fc1c8f94ad3 in ?? ()
83 LWP 11924 "rpc worker-1192" 0x00007fc1c8f94ad3 in ?? ()
84 LWP 11925 "rpc worker-1192" 0x00007fc1c8f94ad3 in ?? ()
85 LWP 11926 "rpc worker-1192" 0x00007fc1c8f94ad3 in ?? ()
86 LWP 11927 "rpc worker-1192" 0x00007fc1c8f94ad3 in ?? ()
87 LWP 11928 "rpc worker-1192" 0x00007fc1c8f94ad3 in ?? ()
88 LWP 11929 "rpc worker-1192" 0x00007fc1c8f94ad3 in ?? ()
89 LWP 11930 "rpc worker-1193" 0x00007fc1c8f94ad3 in ?? ()
90 LWP 11931 "rpc worker-1193" 0x00007fc1c8f94ad3 in ?? ()
91 LWP 11932 "rpc worker-1193" 0x00007fc1c8f94ad3 in ?? ()
92 LWP 11933 "rpc worker-1193" 0x00007fc1c8f94ad3 in ?? ()
93 LWP 11934 "rpc worker-1193" 0x00007fc1c8f94ad3 in ?? ()
94 LWP 11935 "rpc worker-1193" 0x00007fc1c8f94ad3 in ?? ()
95 LWP 11936 "rpc worker-1193" 0x00007fc1c8f94ad3 in ?? ()
96 LWP 11937 "rpc worker-1193" 0x00007fc1c8f94ad3 in ?? ()
97 LWP 11938 "rpc worker-1193" 0x00007fc1c8f94ad3 in ?? ()
98 LWP 11939 "rpc worker-1193" 0x00007fc1c8f94ad3 in ?? ()
99 LWP 11940 "rpc worker-1194" 0x00007fc1c8f94ad3 in ?? ()
100 LWP 11941 "rpc worker-1194" 0x00007fc1c8f94ad3 in ?? ()
101 LWP 11942 "rpc worker-1194" 0x00007fc1c8f94ad3 in ?? ()
102 LWP 11943 "rpc worker-1194" 0x00007fc1c8f94ad3 in ?? ()
103 LWP 11944 "rpc worker-1194" 0x00007fc1c8f94ad3 in ?? ()
104 LWP 11945 "rpc worker-1194" 0x00007fc1c8f94ad3 in ?? ()
105 LWP 11946 "rpc worker-1194" 0x00007fc1c8f94ad3 in ?? ()
106 LWP 11947 "rpc worker-1194" 0x00007fc1c8f94ad3 in ?? ()
107 LWP 11948 "rpc worker-1194" 0x00007fc1c8f94ad3 in ?? ()
108 LWP 11949 "rpc worker-1194" 0x00007fc1c8f94ad3 in ?? ()
109 LWP 11950 "rpc worker-1195" 0x00007fc1c8f94ad3 in ?? ()
110 LWP 11951 "rpc worker-1195" 0x00007fc1c8f94ad3 in ?? ()
111 LWP 11952 "rpc worker-1195" 0x00007fc1c8f94ad3 in ?? ()
112 LWP 11953 "rpc worker-1195" 0x00007fc1c8f94ad3 in ?? ()
113 LWP 11954 "rpc worker-1195" 0x00007fc1c8f94ad3 in ?? ()
114 LWP 11955 "rpc worker-1195" 0x00007fc1c8f94ad3 in ?? ()
115 LWP 11956 "rpc worker-1195" 0x00007fc1c8f94ad3 in ?? ()
116 LWP 11957 "rpc worker-1195" 0x00007fc1c8f94ad3 in ?? ()
117 LWP 11958 "diag-logger-119" 0x00007fc1c5a2a706 in ?? ()
118 LWP 11959 "result-tracker-" 0x00007fc1c8f94fb9 in ?? ()
119 LWP 11960 "excess-log-dele" 0x00007fc1c8f94fb9 in ?? ()
120 LWP 11961 "acceptor-11961" 0x00007fc1c1d070c7 in ?? ()
121 LWP 11962 "heartbeat-11962" 0x00007fc1c8f94fb9 in ?? ()
122 LWP 11963 "maintenance_sch" 0x00007fc1c8f94fb9 in ?? ()
123 LWP 12450 "prepare [worker" 0x00007fc1d2ac7030 in ?? ()
124 LWP 12452 "apply [worker]-" 0x00007fc1c8f94fb9 in ?? ()
125 LWP 12453 "apply [worker]-" 0x00007fc1c8f94fb9 in ?? ()
126 LWP 12454 "wal-append [wor" 0x00007fc1c8f94fb9 in ?? ()
127 LWP 12455 "apply [worker]-" 0x00007fc1c8f94fb9 in ?? ()
128 LWP 12456 "apply [worker]-" 0x00007fc1c8f94fb9 in ?? ()
Thread 128 (LWP 12456):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000001c5 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00007fc1727a8b24 in ?? ()
#5 0x00007fc1727a87b0 in ?? ()
#6 0x000000000000038b in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fc1727a87d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 127 (LWP 12455):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000001f2 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fc172fa9b20 in ?? ()
#5 0x00007fc172fa97b0 in ?? ()
#6 0x00000000000003e4 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 126 (LWP 12454):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x000060200035b178 in ?? ()
#2 0x00000000000010a3 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x000061a0009b95b4 in ?? ()
#5 0x00007fc1737aa090 in ?? ()
#6 0x0000000000002147 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fc1737aa0b0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 125 (LWP 12453):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000001ef in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00007fc173fabb24 in ?? ()
#5 0x00007fc173fab7b0 in ?? ()
#6 0x00000000000003df in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fc173fab7d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 124 (LWP 12452):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000001f1 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00007fc1747acb24 in ?? ()
#5 0x00007fc1747ac7b0 in ?? ()
#6 0x00000000000003e3 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fc1747ac7d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 123 (LWP 12450):
#0 0x00007fc1d2ac7030 in ?? ()
#1 0x00007fc1d27c42b7 in ?? ()
#2 0x00007fc17d8177c0 in ?? ()
#3 0x00007fc17d817710 in ?? ()
#4 0x00007fc17d817890 in ?? ()
#5 0x00007fc1d27be1f7 in ?? ()
#6 0x0000000041b58ab3 in ?? ()
#7 0x00007fc1d21566ca in ?? ()
#8 0x00007fc1d27bddc0 in ?? ()
#9 0x00007fc1c798c8ee in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 122 (LWP 11963):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000120 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007fc17e027550 in ?? ()
#6 0x0000000000000240 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 11962):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x81afaaa989cdd16b in ?? ()
#2 0x0000000000000048 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x000061300001aac4 in ?? ()
#5 0x00007fc17e83f5b0 in ?? ()
#6 0x0000000000000091 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fc17e83f5d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007fc17e83f650 in ?? ()
#12 0x00000ff82fd07ecc in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 120 (LWP 11961):
#0 0x00007fc1c1d070c7 in ?? ()
#1 0x00006140000218a8 in ?? ()
#2 0x00007fc17f056d50 in ?? ()
#3 0x00007fc17f056d80 in ?? ()
#4 0x00007fc17f056e80 in ?? ()
#5 0x00007fc17f056d70 in ?? ()
#6 0x00007fc17f056de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 11960):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x00007fc17f87ff00 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffc53f14150 in ?? ()
#5 0x00007fc17f87fef0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 11959):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x00007fc1800980c0 in ?? ()
#2 0x0000000000000048 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007fc1800980b0 in ?? ()
#6 0x0000000000000090 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 11958):
#0 0x00007fc1c5a2a706 in ?? ()
#1 0x00007fc1808aecc8 in ?? ()
#2 0x00007fc1808aea50 in ?? ()
#3 0x00007fc1c5a2ac74 in ?? ()
#4 0x0000000000000000 in ?? ()
Thread 116 (LWP 11957):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002fd008 in ?? ()
#4 0x00007fc1810c7e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc1810c7e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 11956):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x000000000000000e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002f6808 in ?? ()
#4 0x00007fc1818e0e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc1818e0e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 11955):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 11954):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 11953):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 11952):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 11951):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 11950):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 11949):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 11948):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 11947):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 11946):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 11945):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 11944):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 11943):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 11942):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 11941):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 11940):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 11939):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 11938):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 11937):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 11936):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 11935):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 11934):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 11933):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 11932):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 11931):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 11930):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 11929):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 11928):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 11927):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 11926):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 11925):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 11924):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 11923):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 11922):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 11921):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 11920):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 11919):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 11918):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 11917):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x000000000000027d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001e684c in ?? ()
#4 0x00007fc195489e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc195489e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001e6800 in ?? ()
#9 0x00007fc1c8f94770 in ?? ()
#10 0x00007fc195489e70 in ?? ()
#11 0x00000ff832a913d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 11916):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001d685c in ?? ()
#4 0x00007fc195ca1e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc195ca1e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001d6810 in ?? ()
#9 0x00007fc1c8f94770 in ?? ()
#10 0x00007fc195ca1e70 in ?? ()
#11 0x00000ff832b943d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 74 (LWP 11915):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 11914):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 11913):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 11912):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 11911):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 11910):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 11909):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 11908):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 11907):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 11906):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 11905):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 11904):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 11903):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 11902):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 11901):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 11900):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 11897):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 11896):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 11894):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000156808 in ?? ()
#4 0x00007fc19f66ae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc19f66ae70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 11892):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 11891):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 11890):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 11889):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 11888):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 11887):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 11886):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 11885):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 11884):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 11883):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 11882):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 11881):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 11880):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 11879):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 11878):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 11877):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 11876):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 11874):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 11873):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 11872):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000bffc8 in ?? ()
#4 0x00007fc1a984be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc1a984be70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 11871):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 11870):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 11869):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 11868):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 11867):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 11866):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 11865):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 11864):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 11863):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 11862):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 11861):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 11860):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 11859):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 11858):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 11857):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 11856):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 11855):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 11854):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 11853):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 11852):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x00007fc1b19efc80 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ba60 in ?? ()
#5 0x00007fc1b19efc70 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 11851):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 11849):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x00007fc1b32861c0 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007fc1b32861b0 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 11848):
#0 0x00007fc1c8f94ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 11846):
#0 0x00007fc1c1d05a47 in ?? ()
#1 0x00007fc1b42af2e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007fc1b42af2d0 in ?? ()
#4 0x00007fc1b42af4e0 in ?? ()
#5 0x00007fc1b42af320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007fc1b42af3a0 in ?? ()
#8 0x00007fc1c436525d in ?? ()
#9 0x3fb9771f019b8000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000006000000 in ?? ()
#14 0x00007fc100000000 in ?? ()
#15 0x41d9e21ced720495 in ?? ()
#16 0x00000ff8b684de74 in ?? ()
#17 0x00007fc1b42af380 in ?? ()
#18 0x00007fc1c4369ba3 in ?? ()
#19 0x00007fc1b42af350 in ?? ()
#20 0x3fb9771f019b8000 in ?? ()
#21 0x00000000b42af3a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb9771f019b8000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 11845):
#0 0x00007fc1c1d05a47 in ?? ()
#1 0x00007fc1b4ab02e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007fc1b4ab02d0 in ?? ()
#4 0x00007fc1b4ab04e0 in ?? ()
#5 0x00007fc1b4ab0320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007fc1b4ab03a0 in ?? ()
#8 0x00007fc1c436525d in ?? ()
#9 0x3fb9613d4745a000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x00000000e6000000 in ?? ()
#14 0x00007fc100000000 in ?? ()
#15 0x41d9e21ced72049a in ?? ()
#16 0x00000ff8b694e074 in ?? ()
#17 0x00007fc1b4ab0380 in ?? ()
#18 0x00007fc1c4369ba3 in ?? ()
#19 0x00007fc1b4ab0350 in ?? ()
#20 0x3fb9613d4745a000 in ?? ()
#21 0x00000000b4ab03a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb9613d4745a000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 11844):
#0 0x00007fc1c1d05a47 in ?? ()
#1 0x00007fc1b52b12e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007fc1b52b12d0 in ?? ()
#4 0x00007fc1b52b14e0 in ?? ()
#5 0x00007fc1b52b1320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007fc1b52b13a0 in ?? ()
#8 0x00007fc1c436525d in ?? ()
#9 0x3fa04412a108c000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000043e00000 in ?? ()
#14 0x00007fc100000000 in ?? ()
#15 0x41d9e21ced720496 in ?? ()
#16 0x00000ff8b6a4e274 in ?? ()
#17 0x00007fc1b52b1380 in ?? ()
#18 0x00007fc1c4369ba3 in ?? ()
#19 0x00007fc1b52b1350 in ?? ()
#20 0x3fa04412a108c000 in ?? ()
#21 0x00000000b52b13a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fa04412a108c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 11843):
#0 0x00007fc1c1d05a47 in ?? ()
#1 0x00007fc1b72aa2e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007fc1b72aa2d0 in ?? ()
#4 0x00007fc1b72aa4e0 in ?? ()
#5 0x00007fc1b72aa320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fc1b72aa3a0 in ?? ()
#8 0x00007fc1c436525d in ?? ()
#9 0x3fb956cdbddc2000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x00000000f6000000 in ?? ()
#14 0x00007fc100000000 in ?? ()
#15 0x41d9e21ced720498 in ?? ()
#16 0x00000ff8b6e4d474 in ?? ()
#17 0x00007fc1b72aa380 in ?? ()
#18 0x00007fc1c4369ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 11837):
#0 0x00007fc1c1cf8cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fc1ba4b7788 in ?? ()
#3 0x0000602000017cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 11836):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x00007fc1b9cb6080 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007fc1b9cb6070 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 11835):
#0 0x00007fc1c8f989e2 in ?? ()
#1 0x00007fc1b94b3b60 in ?? ()
#2 0x00007fc1b94b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fc1b94b3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007fc1b94b3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007fc1b94b3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007fc1ce8bcf0c in ?? ()
#12 0x00007fc1ce8ac7b9 in ?? ()
#13 0x00007fc1ce8b0da6 in ?? ()
#14 0x00007fc1ce8b57e0 in ?? ()
#15 0x00007fc1ce8b5609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007fc1c5bac98e in ?? ()
#18 0x00007fc1c8f8e6db in ?? ()
#19 0x00000ff8b728e67c in ?? ()
#20 0x00007fc1b94b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 11829):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x00007fc1bb4b8c40 in ?? ()
#2 0x0000000000000171 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007fc1bb4b8c30 in ?? ()
#6 0x00000000000002e2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 11827):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 11826):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 11825):
#0 0x00007fc1c8f94fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000004 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fc1bccbc410 in ?? ()
#6 0x0000000000000008 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 11823):
#0 0x00007fc1c8f98d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250116 02:58:06.719134 8081 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 4 with UUID ff972e037b0e40bba36f7896effdb71f and pid 9520
************************ BEGIN STACKS **************************
[New LWP 9522]
[New LWP 9523]
[New LWP 9524]
[New LWP 9525]
[New LWP 9531]
[New LWP 9532]
[New LWP 9533]
[New LWP 9536]
[New LWP 9537]
[New LWP 9538]
[New LWP 9539]
[New LWP 9540]
[New LWP 9541]
[New LWP 9542]
[New LWP 9543]
[New LWP 9544]
[New LWP 9545]
[New LWP 9546]
[New LWP 9547]
[New LWP 9548]
[New LWP 9549]
[New LWP 9550]
[New LWP 9551]
[New LWP 9552]
[New LWP 9553]
[New LWP 9554]
[New LWP 9555]
[New LWP 9556]
[New LWP 9557]
[New LWP 9558]
[New LWP 9559]
[New LWP 9560]
[New LWP 9561]
[New LWP 9562]
[New LWP 9563]
[New LWP 9564]
[New LWP 9565]
[New LWP 9567]
[New LWP 9568]
[New LWP 9569]
[New LWP 9570]
[New LWP 9571]
[New LWP 9572]
[New LWP 9573]
[New LWP 9574]
[New LWP 9575]
[New LWP 9576]
[New LWP 9577]
[New LWP 9578]
[New LWP 9579]
[New LWP 9580]
[New LWP 9581]
[New LWP 9582]
[New LWP 9583]
[New LWP 9584]
[New LWP 9585]
[New LWP 9586]
[New LWP 9587]
[New LWP 9588]
[New LWP 9589]
[New LWP 9590]
[New LWP 9591]
[New LWP 9592]
[New LWP 9593]
[New LWP 9594]
[New LWP 9595]
[New LWP 9596]
[New LWP 9597]
[New LWP 9598]
[New LWP 9599]
[New LWP 9600]
[New LWP 9601]
[New LWP 9602]
[New LWP 9603]
[New LWP 9604]
[New LWP 9605]
[New LWP 9606]
[New LWP 9607]
[New LWP 9608]
[New LWP 9609]
[New LWP 9610]
[New LWP 9611]
[New LWP 9612]
[New LWP 9613]
[New LWP 9614]
[New LWP 9616]
[New LWP 9617]
[New LWP 9618]
[New LWP 9619]
[New LWP 9620]
[New LWP 9621]
[New LWP 9622]
[New LWP 9623]
[New LWP 9624]
[New LWP 9625]
[New LWP 9626]
[New LWP 9627]
[New LWP 9628]
[New LWP 9629]
[New LWP 9630]
[New LWP 9631]
[New LWP 9632]
[New LWP 9633]
[New LWP 9634]
[New LWP 9635]
[New LWP 9636]
[New LWP 9637]
[New LWP 9638]
[New LWP 9639]
[New LWP 9640]
[New LWP 9641]
[New LWP 9642]
[New LWP 9644]
[New LWP 9645]
[New LWP 9646]
[New LWP 9647]
[New LWP 9648]
[New LWP 9649]
[New LWP 9650]
[New LWP 9651]
[New LWP 9655]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007fc153dd5d50 in ?? ()
Id Target Id Frame
* 1 LWP 9520 "kudu" 0x00007fc153dd5d50 in ?? ()
2 LWP 9522 "kudu" 0x00007fc153dd1fb9 in ?? ()
3 LWP 9523 "kudu" 0x00007fc153dd1fb9 in ?? ()
4 LWP 9524 "kudu" 0x00007fc153dd1fb9 in ?? ()
5 LWP 9525 "kernel-watcher-" 0x00007fc153dd1fb9 in ?? ()
6 LWP 9531 "ntp client-9531" 0x00007fc153dd59e2 in ?? ()
7 LWP 9532 "file cache-evic" 0x00007fc153dd1fb9 in ?? ()
8 LWP 9533 "sq_acceptor" 0x00007fc14cb35cb9 in ?? ()
9 LWP 9536 "rpc reactor-953" 0x00007fc14cb42a47 in ?? ()
10 LWP 9537 "rpc reactor-953" 0x00007fc14cb42a47 in ?? ()
11 LWP 9538 "rpc reactor-953" 0x00007fc14cb42a47 in ?? ()
12 LWP 9539 "rpc reactor-953" 0x00007fc14cb42a47 in ?? ()
13 LWP 9540 "MaintenanceMgr " 0x00007fc153dd1ad3 in ?? ()
14 LWP 9541 "txn-status-mana" 0x00007fc153dd1fb9 in ?? ()
15 LWP 9542 "collect_and_rem" 0x00007fc153dd1fb9 in ?? ()
16 LWP 9543 "tc-session-exp-" 0x00007fc153dd1fb9 in ?? ()
17 LWP 9544 "rpc worker-9544" 0x00007fc153dd1ad3 in ?? ()
18 LWP 9545 "rpc worker-9545" 0x00007fc153dd1ad3 in ?? ()
19 LWP 9546 "rpc worker-9546" 0x00007fc153dd1ad3 in ?? ()
20 LWP 9547 "rpc worker-9547" 0x00007fc153dd1ad3 in ?? ()
21 LWP 9548 "rpc worker-9548" 0x00007fc153dd1ad3 in ?? ()
22 LWP 9549 "rpc worker-9549" 0x00007fc153dd1ad3 in ?? ()
23 LWP 9550 "rpc worker-9550" 0x00007fc153dd1ad3 in ?? ()
24 LWP 9551 "rpc worker-9551" 0x00007fc153dd1ad3 in ?? ()
25 LWP 9552 "rpc worker-9552" 0x00007fc153dd1ad3 in ?? ()
26 LWP 9553 "rpc worker-9553" 0x00007fc153dd1ad3 in ?? ()
27 LWP 9554 "rpc worker-9554" 0x00007fc153dd1ad3 in ?? ()
28 LWP 9555 "rpc worker-9555" 0x00007fc153dd1ad3 in ?? ()
29 LWP 9556 "rpc worker-9556" 0x00007fc153dd1ad3 in ?? ()
30 LWP 9557 "rpc worker-9557" 0x00007fc153dd1ad3 in ?? ()
31 LWP 9558 "rpc worker-9558" 0x00007fc153dd1ad3 in ?? ()
32 LWP 9559 "rpc worker-9559" 0x00007fc153dd1ad3 in ?? ()
33 LWP 9560 "rpc worker-9560" 0x00007fc153dd1ad3 in ?? ()
34 LWP 9561 "rpc worker-9561" 0x00007fc153dd1ad3 in ?? ()
35 LWP 9562 "rpc worker-9562" 0x00007fc153dd1ad3 in ?? ()
36 LWP 9563 "rpc worker-9563" 0x00007fc153dd1ad3 in ?? ()
37 LWP 9564 "rpc worker-9564" 0x00007fc153dd1ad3 in ?? ()
38 LWP 9565 "rpc worker-9565" 0x00007fc153dd1ad3 in ?? ()
39 LWP 9567 "rpc worker-9567" 0x00007fc153dd1ad3 in ?? ()
40 LWP 9568 "rpc worker-9568" 0x00007fc153dd1ad3 in ?? ()
41 LWP 9569 "rpc worker-9569" 0x00007fc153dd1ad3 in ?? ()
42 LWP 9570 "rpc worker-9570" 0x00007fc153dd1ad3 in ?? ()
43 LWP 9571 "rpc worker-9571" 0x00007fc153dd1ad3 in ?? ()
44 LWP 9572 "rpc worker-9572" 0x00007fc153dd1ad3 in ?? ()
45 LWP 9573 "rpc worker-9573" 0x00007fc153dd1ad3 in ?? ()
46 LWP 9574 "rpc worker-9574" 0x00007fc153dd1ad3 in ?? ()
47 LWP 9575 "rpc worker-9575" 0x00007fc153dd1ad3 in ?? ()
48 LWP 9576 "rpc worker-9576" 0x00007fc153dd1ad3 in ?? ()
49 LWP 9577 "rpc worker-9577" 0x00007fc153dd1ad3 in ?? ()
50 LWP 9578 "rpc worker-9578" 0x00007fc153dd1ad3 in ?? ()
51 LWP 9579 "rpc worker-9579" 0x00007fc153dd1ad3 in ?? ()
52 LWP 9580 "rpc worker-9580" 0x00007fc153dd1ad3 in ?? ()
53 LWP 9581 "rpc worker-9581" 0x00007fc153dd1ad3 in ?? ()
54 LWP 9582 "rpc worker-9582" 0x00007fc153dd1ad3 in ?? ()
55 LWP 9583 "rpc worker-9583" 0x00007fc153dd1ad3 in ?? ()
56 LWP 9584 "rpc worker-9584" 0x00007fc153dd1ad3 in ?? ()
57 LWP 9585 "rpc worker-9585" 0x00007fc153dd1ad3 in ?? ()
58 LWP 9586 "rpc worker-9586" 0x00007fc153dd1ad3 in ?? ()
59 LWP 9587 "rpc worker-9587" 0x00007fc153dd1ad3 in ?? ()
60 LWP 9588 "rpc worker-9588" 0x00007fc153dd1ad3 in ?? ()
61 LWP 9589 "rpc worker-9589" 0x00007fc153dd1ad3 in ?? ()
62 LWP 9590 "rpc worker-9590" 0x00007fc153dd1ad3 in ?? ()
63 LWP 9591 "rpc worker-9591" 0x00007fc153dd1ad3 in ?? ()
64 LWP 9592 "rpc worker-9592" 0x00007fc153dd1ad3 in ?? ()
65 LWP 9593 "rpc worker-9593" 0x00007fc153dd1ad3 in ?? ()
66 LWP 9594 "rpc worker-9594" 0x00007fc153dd1ad3 in ?? ()
67 LWP 9595 "rpc worker-9595" 0x00007fc153dd1ad3 in ?? ()
68 LWP 9596 "rpc worker-9596" 0x00007fc153dd1ad3 in ?? ()
69 LWP 9597 "rpc worker-9597" 0x00007fc153dd1ad3 in ?? ()
70 LWP 9598 "rpc worker-9598" 0x00007fc153dd1ad3 in ?? ()
71 LWP 9599 "rpc worker-9599" 0x00007fc153dd1ad3 in ?? ()
72 LWP 9600 "rpc worker-9600" 0x00007fc153dd1ad3 in ?? ()
73 LWP 9601 "rpc worker-9601" 0x00007fc153dd1ad3 in ?? ()
74 LWP 9602 "rpc worker-9602" 0x00007fc153dd1ad3 in ?? ()
75 LWP 9603 "rpc worker-9603" 0x00007fc153dd1ad3 in ?? ()
76 LWP 9604 "rpc worker-9604" 0x00007fc153dd1ad3 in ?? ()
77 LWP 9605 "rpc worker-9605" 0x00007fc153dd1ad3 in ?? ()
78 LWP 9606 "rpc worker-9606" 0x00007fc153dd1ad3 in ?? ()
79 LWP 9607 "rpc worker-9607" 0x00007fc153dd1ad3 in ?? ()
80 LWP 9608 "rpc worker-9608" 0x00007fc153dd1ad3 in ?? ()
81 LWP 9609 "rpc worker-9609" 0x00007fc153dd1ad3 in ?? ()
82 LWP 9610 "rpc worker-9610" 0x00007fc153dd1ad3 in ?? ()
83 LWP 9611 "rpc worker-9611" 0x00007fc153dd1ad3 in ?? ()
84 LWP 9612 "rpc worker-9612" 0x00007fc153dd1ad3 in ?? ()
85 LWP 9613 "rpc worker-9613" 0x00007fc153dd1ad3 in ?? ()
86 LWP 9614 "rpc worker-9614" 0x00007fc153dd1ad3 in ?? ()
87 LWP 9616 "rpc worker-9616" 0x00007fc153dd1ad3 in ?? ()
88 LWP 9617 "rpc worker-9617" 0x00007fc153dd1ad3 in ?? ()
89 LWP 9618 "rpc worker-9618" 0x00007fc153dd1ad3 in ?? ()
90 LWP 9619 "rpc worker-9619" 0x00007fc153dd1ad3 in ?? ()
91 LWP 9620 "rpc worker-9620" 0x00007fc153dd1ad3 in ?? ()
92 LWP 9621 "rpc worker-9621" 0x00007fc153dd1ad3 in ?? ()
93 LWP 9622 "rpc worker-9622" 0x00007fc153dd1ad3 in ?? ()
94 LWP 9623 "rpc worker-9623" 0x00007fc153dd1ad3 in ?? ()
95 LWP 9624 "rpc worker-9624" 0x00007fc153dd1ad3 in ?? ()
96 LWP 9625 "rpc worker-9625" 0x00007fc153dd1ad3 in ?? ()
97 LWP 9626 "rpc worker-9626" 0x00007fc153dd1ad3 in ?? ()
98 LWP 9627 "rpc worker-9627" 0x00007fc153dd1ad3 in ?? ()
99 LWP 9628 "rpc worker-9628" 0x00007fc153dd1ad3 in ?? ()
100 LWP 9629 "rpc worker-9629" 0x00007fc153dd1ad3 in ?? ()
101 LWP 9630 "rpc worker-9630" 0x00007fc153dd1ad3 in ?? ()
102 LWP 9631 "rpc worker-9631" 0x00007fc153dd1ad3 in ?? ()
103 LWP 9632 "rpc worker-9632" 0x00007fc153dd1ad3 in ?? ()
104 LWP 9633 "rpc worker-9633" 0x00007fc153dd1ad3 in ?? ()
105 LWP 9634 "rpc worker-9634" 0x00007fc153dd1ad3 in ?? ()
106 LWP 9635 "rpc worker-9635" 0x00007fc153dd1ad3 in ?? ()
107 LWP 9636 "rpc worker-9636" 0x00007fc153dd1ad3 in ?? ()
108 LWP 9637 "rpc worker-9637" 0x00007fc153dd1ad3 in ?? ()
109 LWP 9638 "rpc worker-9638" 0x00007fc153dd1ad3 in ?? ()
110 LWP 9639 "rpc worker-9639" 0x00007fc153dd1ad3 in ?? ()
111 LWP 9640 "rpc worker-9640" 0x00007fc153dd1ad3 in ?? ()
112 LWP 9641 "rpc worker-9641" 0x00007fc153dd1ad3 in ?? ()
113 LWP 9642 "rpc worker-9642" 0x00007fc153dd1ad3 in ?? ()
114 LWP 9644 "rpc worker-9644" 0x00007fc153dd1ad3 in ?? ()
115 LWP 9645 "rpc worker-9645" 0x00007fc153dd1ad3 in ?? ()
116 LWP 9646 "rpc worker-9646" 0x00007fc153dd1ad3 in ?? ()
117 LWP 9647 "diag-logger-964" 0x00007fc153dd1fb9 in ?? ()
118 LWP 9648 "result-tracker-" 0x00007fc153dd1fb9 in ?? ()
119 LWP 9649 "excess-log-dele" 0x00007fc153dd1fb9 in ?? ()
120 LWP 9650 "acceptor-9650" 0x00007fc14cb440c7 in ?? ()
121 LWP 9651 "heartbeat-9651" 0x00007fc153dd1fb9 in ?? ()
122 LWP 9655 "maintenance_sch" 0x00007fc153dd1fb9 in ?? ()
Thread 122 (LWP 9655):
#0 0x00007fc153dd1fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000339 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007fc1096bc550 in ?? ()
#6 0x0000000000000672 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 9651):
#0 0x00007fc153dd1fb9 in ?? ()
#1 0x81afaaa989cdd16b in ?? ()
#2 0x00000000000000ce in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007fc109ed45b0 in ?? ()
#6 0x000000000000019c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 9650):
#0 0x00007fc14cb440c7 in ?? ()
#1 0x00006140000216a8 in ?? ()
#2 0x00007fc10a6fbd50 in ?? ()
#3 0x00007fc10a6fbd80 in ?? ()
#4 0x00007fc10a6fbe80 in ?? ()
#5 0x00007fc10a6fbd70 in ?? ()
#6 0x00007fc10a6fbde0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007fc1509c8ef0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 9649):
#0 0x00007fc153dd1fb9 in ?? ()
#1 0x00007fc10af15f00 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fffc7366b70 in ?? ()
#5 0x00007fc10af15ef0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 9648):
#0 0x00007fc153dd1fb9 in ?? ()
#1 0x00007fc10b72e0c0 in ?? ()
#2 0x00000000000000ce in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007fc10b72e0b0 in ?? ()
#6 0x000000000000019c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 9647):
#0 0x00007fc153dd1fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 9646):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000013 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007fc10c75de50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc10c75de70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007fc153dd1770 in ?? ()
#10 0x00007fc10c75de70 in ?? ()
#11 0x00000ff8218ebbd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 9645):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000035 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007fc10cf75e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc10cf75e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007fc153dd1770 in ?? ()
#10 0x00007fc10cf75e70 in ?? ()
#11 0x00000ff8219eebd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 9644):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 9642):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 9641):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 9640):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 9639):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 9638):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 9637):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 9636):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 9635):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 9634):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 9633):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 9632):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 9631):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 9630):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 9629):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 9628):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 9627):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 9626):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 9625):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 9624):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 9623):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 9622):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 9621):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 9620):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 9619):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 9618):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 9617):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 9616):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 9614):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 9613):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 9612):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 9611):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 9610):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 9609):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 9608):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 9607):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 9606):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 9605):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 9604):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007fc120316e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc120316e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 9603):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 9602):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 9601):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 9600):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 9599):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 9598):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 9597):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 9596):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 9595):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 9594):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 9593):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 9592):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 9591):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 9590):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 9589):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 9588):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 9587):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 9586):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 9585):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 9584):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007fc12a4f7e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc12a4f7e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 9583):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 9582):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 9581):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 9580):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 9579):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 9578):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 9577):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 9576):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 9575):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 9574):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 9573):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 9572):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 9571):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 9570):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 9569):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 9568):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 9567):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 9565):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 9564):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 9563):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007fc1346d8e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc1346d8e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 9562):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 9561):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 9560):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 9559):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 9558):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 9557):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 9556):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 9555):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 9554):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 9553):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 9552):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 9551):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 9550):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 9549):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 9548):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 9547):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 9546):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 9545):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 9544):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 9543):
#0 0x00007fc153dd1fb9 in ?? ()
#1 0x00007fc13e8d3c80 in ?? ()
#2 0x0000000000000014 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001b360 in ?? ()
#5 0x00007fc13e8d3c70 in ?? ()
#6 0x0000000000000028 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 9542):
#0 0x00007fc153dd1fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 9541):
#0 0x00007fc153dd1fb9 in ?? ()
#1 0x00007fc13f9111c0 in ?? ()
#2 0x0000000000000014 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007fc13f9111b0 in ?? ()
#6 0x0000000000000028 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 9540):
#0 0x00007fc153dd1ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 9539):
#0 0x00007fc14cb42a47 in ?? ()
#1 0x00007fc1409602e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007fc1409602d0 in ?? ()
#4 0x00007fc1409604e0 in ?? ()
#5 0x00007fc140960320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007fc1409603a0 in ?? ()
#8 0x00007fc14f1a225d in ?? ()
#9 0x3fb97b9b2387c000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000088000000 in ?? ()
#14 0x00007fc100000000 in ?? ()
#15 0x41d9e21ced720497 in ?? ()
#16 0x00000ff8a8124074 in ?? ()
#17 0x00007fc140960380 in ?? ()
#18 0x00007fc14f1a6ba3 in ?? ()
#19 0x00007fc140960350 in ?? ()
#20 0x3fb97b9b2387c000 in ?? ()
#21 0x00000000409603a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb97b9b2387c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 9538):
#0 0x00007fc14cb42a47 in ?? ()
#1 0x00007fc1411772e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007fc1411772d0 in ?? ()
#4 0x00007fc1411774e0 in ?? ()
#5 0x00007fc141177320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007fc1411773a0 in ?? ()
#8 0x00007fc14f1a225d in ?? ()
#9 0x3fb979df50772000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x00000000c4000000 in ?? ()
#14 0x00007fc100000000 in ?? ()
#15 0x41d9e21ced720495 in ?? ()
#16 0x00000ff8a8226e74 in ?? ()
#17 0x00007fc141177380 in ?? ()
#18 0x00007fc14f1a6ba3 in ?? ()
#19 0x00007fc141177350 in ?? ()
#20 0x3fb979df50772000 in ?? ()
#21 0x00000000411773a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb979df50772000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 9537):
#0 0x00007fc14cb42a47 in ?? ()
#1 0x00007fc14198e2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007fc14198e2d0 in ?? ()
#4 0x00007fc14198e4e0 in ?? ()
#5 0x00007fc14198e320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007fc14198e3a0 in ?? ()
#8 0x00007fc14f1a225d in ?? ()
#9 0x3fb961d0e2866000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x000000007c000000 in ?? ()
#14 0x00007fc100000000 in ?? ()
#15 0x41d9e21ced720499 in ?? ()
#16 0x00000ff8a8329c74 in ?? ()
#17 0x00007fc14198e380 in ?? ()
#18 0x00007fc14f1a6ba3 in ?? ()
#19 0x00007fc14198e350 in ?? ()
#20 0x3fb961d0e2866000 in ?? ()
#21 0x000000004198e3a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb961d0e2866000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 9536):
#0 0x00007fc14cb42a47 in ?? ()
#1 0x00007fc1439ad2e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007fc1439ad2d0 in ?? ()
#4 0x00007fc1439ad4e0 in ?? ()
#5 0x00007fc1439ad320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fc1439ad3a0 in ?? ()
#8 0x00007fc14f1a225d in ?? ()
#9 0x3fb96d1211a9e000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x0000000074800000 in ?? ()
#14 0x00007fc100000000 in ?? ()
#15 0x41d9e21ced720496 in ?? ()
#16 0x00000ff8a872da74 in ?? ()
#17 0x00007fc1439ad380 in ?? ()
#18 0x00007fc14f1a6ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 9533):
#0 0x00007fc14cb35cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fc1453b7788 in ?? ()
#3 0x0000602000017cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 9532):
#0 0x00007fc153dd1fb9 in ?? ()
#1 0x00007fc144bb6080 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007fc144bb6070 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 9531):
#0 0x00007fc153dd59e2 in ?? ()
#1 0x00007fc1443b3b60 in ?? ()
#2 0x00007fc1443b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fc1443b3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007fc1443b3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007fc1443b3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007fc1596f9f0c in ?? ()
#12 0x00007fc1596e97b9 in ?? ()
#13 0x00007fc1596edda6 in ?? ()
#14 0x00007fc1596f27e0 in ?? ()
#15 0x00007fc1596f2609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007fc1509e998e in ?? ()
#18 0x00007fc153dcb6db in ?? ()
#19 0x00000ff8a886e67c in ?? ()
#20 0x00007fc1443b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 9525):
#0 0x00007fc153dd1fb9 in ?? ()
#1 0x00007fc1463b8c40 in ?? ()
#2 0x000000000000040b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007fc1463b8c30 in ?? ()
#6 0x0000000000000816 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 9524):
#0 0x00007fc153dd1fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 9523):
#0 0x00007fc153dd1fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 9522):
#0 0x00007fc153dd1fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fc147bbc410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 9520):
#0 0x00007fc153dd5d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250116 02:58:07.596604 8081 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 5 with UUID d5ba0801ecba49269379dd93eedde2f8 and pid 9659
************************ BEGIN STACKS **************************
[New LWP 9660]
[New LWP 9661]
[New LWP 9662]
[New LWP 9663]
[New LWP 9669]
[New LWP 9670]
[New LWP 9671]
[New LWP 9674]
[New LWP 9675]
[New LWP 9676]
[New LWP 9677]
[New LWP 9678]
[New LWP 9679]
[New LWP 9680]
[New LWP 9681]
[New LWP 9682]
[New LWP 9683]
[New LWP 9684]
[New LWP 9685]
[New LWP 9686]
[New LWP 9687]
[New LWP 9688]
[New LWP 9689]
[New LWP 9690]
[New LWP 9691]
[New LWP 9692]
[New LWP 9693]
[New LWP 9694]
[New LWP 9695]
[New LWP 9696]
[New LWP 9697]
[New LWP 9698]
[New LWP 9699]
[New LWP 9700]
[New LWP 9701]
[New LWP 9702]
[New LWP 9703]
[New LWP 9704]
[New LWP 9705]
[New LWP 9706]
[New LWP 9707]
[New LWP 9708]
[New LWP 9709]
[New LWP 9710]
[New LWP 9711]
[New LWP 9712]
[New LWP 9713]
[New LWP 9714]
[New LWP 9715]
[New LWP 9716]
[New LWP 9717]
[New LWP 9718]
[New LWP 9719]
[New LWP 9720]
[New LWP 9721]
[New LWP 9722]
[New LWP 9723]
[New LWP 9724]
[New LWP 9725]
[New LWP 9726]
[New LWP 9727]
[New LWP 9728]
[New LWP 9729]
[New LWP 9730]
[New LWP 9731]
[New LWP 9732]
[New LWP 9733]
[New LWP 9734]
[New LWP 9736]
[New LWP 9737]
[New LWP 9738]
[New LWP 9739]
[New LWP 9740]
[New LWP 9741]
[New LWP 9742]
[New LWP 9743]
[New LWP 9744]
[New LWP 9745]
[New LWP 9746]
[New LWP 9747]
[New LWP 9748]
[New LWP 9749]
[New LWP 9750]
[New LWP 9751]
[New LWP 9752]
[New LWP 9753]
[New LWP 9754]
[New LWP 9755]
[New LWP 9756]
[New LWP 9757]
[New LWP 9758]
[New LWP 9759]
[New LWP 9760]
[New LWP 9761]
[New LWP 9762]
[New LWP 9763]
[New LWP 9764]
[New LWP 9765]
[New LWP 9766]
[New LWP 9767]
[New LWP 9768]
[New LWP 9769]
[New LWP 9770]
[New LWP 9771]
[New LWP 9772]
[New LWP 9773]
[New LWP 9774]
[New LWP 9775]
[New LWP 9776]
[New LWP 9777]
[New LWP 9778]
[New LWP 9779]
[New LWP 9780]
[New LWP 9781]
[New LWP 9782]
[New LWP 9783]
[New LWP 9784]
[New LWP 9785]
[New LWP 9786]
[New LWP 9787]
[New LWP 9789]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007fb842758d50 in ?? ()
Id Target Id Frame
* 1 LWP 9659 "kudu" 0x00007fb842758d50 in ?? ()
2 LWP 9660 "kudu" 0x00007fb842754fb9 in ?? ()
3 LWP 9661 "kudu" 0x00007fb842754fb9 in ?? ()
4 LWP 9662 "kudu" 0x00007fb842754fb9 in ?? ()
5 LWP 9663 "kernel-watcher-" 0x00007fb842754fb9 in ?? ()
6 LWP 9669 "ntp client-9669" 0x00007fb8427589e2 in ?? ()
7 LWP 9670 "file cache-evic" 0x00007fb842754fb9 in ?? ()
8 LWP 9671 "sq_acceptor" 0x00007fb83b4b8cb9 in ?? ()
9 LWP 9674 "rpc reactor-967" 0x00007fb83b4c5a47 in ?? ()
10 LWP 9675 "rpc reactor-967" 0x00007fb83b4c5a47 in ?? ()
11 LWP 9676 "rpc reactor-967" 0x00007fb83b4c5a47 in ?? ()
12 LWP 9677 "rpc reactor-967" 0x00007fb83b4c5a47 in ?? ()
13 LWP 9678 "MaintenanceMgr " 0x00007fb842754ad3 in ?? ()
14 LWP 9679 "txn-status-mana" 0x00007fb842754fb9 in ?? ()
15 LWP 9680 "collect_and_rem" 0x00007fb842754fb9 in ?? ()
16 LWP 9681 "tc-session-exp-" 0x00007fb842754fb9 in ?? ()
17 LWP 9682 "rpc worker-9682" 0x00007fb842754ad3 in ?? ()
18 LWP 9683 "rpc worker-9683" 0x00007fb842754ad3 in ?? ()
19 LWP 9684 "rpc worker-9684" 0x00007fb842754ad3 in ?? ()
20 LWP 9685 "rpc worker-9685" 0x00007fb842754ad3 in ?? ()
21 LWP 9686 "rpc worker-9686" 0x00007fb842754ad3 in ?? ()
22 LWP 9687 "rpc worker-9687" 0x00007fb842754ad3 in ?? ()
23 LWP 9688 "rpc worker-9688" 0x00007fb842754ad3 in ?? ()
24 LWP 9689 "rpc worker-9689" 0x00007fb842754ad3 in ?? ()
25 LWP 9690 "rpc worker-9690" 0x00007fb842754ad3 in ?? ()
26 LWP 9691 "rpc worker-9691" 0x00007fb842754ad3 in ?? ()
27 LWP 9692 "rpc worker-9692" 0x00007fb842754ad3 in ?? ()
28 LWP 9693 "rpc worker-9693" 0x00007fb842754ad3 in ?? ()
29 LWP 9694 "rpc worker-9694" 0x00007fb842754ad3 in ?? ()
30 LWP 9695 "rpc worker-9695" 0x00007fb842754ad3 in ?? ()
31 LWP 9696 "rpc worker-9696" 0x00007fb842754ad3 in ?? ()
32 LWP 9697 "rpc worker-9697" 0x00007fb842754ad3 in ?? ()
33 LWP 9698 "rpc worker-9698" 0x00007fb842754ad3 in ?? ()
34 LWP 9699 "rpc worker-9699" 0x00007fb842754ad3 in ?? ()
35 LWP 9700 "rpc worker-9700" 0x00007fb842754ad3 in ?? ()
36 LWP 9701 "rpc worker-9701" 0x00007fb842754ad3 in ?? ()
37 LWP 9702 "rpc worker-9702" 0x00007fb842754ad3 in ?? ()
38 LWP 9703 "rpc worker-9703" 0x00007fb842754ad3 in ?? ()
39 LWP 9704 "rpc worker-9704" 0x00007fb842754ad3 in ?? ()
40 LWP 9705 "rpc worker-9705" 0x00007fb842754ad3 in ?? ()
41 LWP 9706 "rpc worker-9706" 0x00007fb842754ad3 in ?? ()
42 LWP 9707 "rpc worker-9707" 0x00007fb842754ad3 in ?? ()
43 LWP 9708 "rpc worker-9708" 0x00007fb842754ad3 in ?? ()
44 LWP 9709 "rpc worker-9709" 0x00007fb842754ad3 in ?? ()
45 LWP 9710 "rpc worker-9710" 0x00007fb842754ad3 in ?? ()
46 LWP 9711 "rpc worker-9711" 0x00007fb842754ad3 in ?? ()
47 LWP 9712 "rpc worker-9712" 0x00007fb842754ad3 in ?? ()
48 LWP 9713 "rpc worker-9713" 0x00007fb842754ad3 in ?? ()
49 LWP 9714 "rpc worker-9714" 0x00007fb842754ad3 in ?? ()
50 LWP 9715 "rpc worker-9715" 0x00007fb842754ad3 in ?? ()
51 LWP 9716 "rpc worker-9716" 0x00007fb842754ad3 in ?? ()
52 LWP 9717 "rpc worker-9717" 0x00007fb842754ad3 in ?? ()
53 LWP 9718 "rpc worker-9718" 0x00007fb842754ad3 in ?? ()
54 LWP 9719 "rpc worker-9719" 0x00007fb842754ad3 in ?? ()
55 LWP 9720 "rpc worker-9720" 0x00007fb842754ad3 in ?? ()
56 LWP 9721 "rpc worker-9721" 0x00007fb842754ad3 in ?? ()
57 LWP 9722 "rpc worker-9722" 0x00007fb842754ad3 in ?? ()
58 LWP 9723 "rpc worker-9723" 0x00007fb842754ad3 in ?? ()
59 LWP 9724 "rpc worker-9724" 0x00007fb842754ad3 in ?? ()
60 LWP 9725 "rpc worker-9725" 0x00007fb842754ad3 in ?? ()
61 LWP 9726 "rpc worker-9726" 0x00007fb842754ad3 in ?? ()
62 LWP 9727 "rpc worker-9727" 0x00007fb842754ad3 in ?? ()
63 LWP 9728 "rpc worker-9728" 0x00007fb842754ad3 in ?? ()
64 LWP 9729 "rpc worker-9729" 0x00007fb842754ad3 in ?? ()
65 LWP 9730 "rpc worker-9730" 0x00007fb842754ad3 in ?? ()
66 LWP 9731 "rpc worker-9731" 0x00007fb842754ad3 in ?? ()
67 LWP 9732 "rpc worker-9732" 0x00007fb842754ad3 in ?? ()
68 LWP 9733 "rpc worker-9733" 0x00007fb842754ad3 in ?? ()
69 LWP 9734 "rpc worker-9734" 0x00007fb842754ad3 in ?? ()
70 LWP 9736 "rpc worker-9736" 0x00007fb842754ad3 in ?? ()
71 LWP 9737 "rpc worker-9737" 0x00007fb842754ad3 in ?? ()
72 LWP 9738 "rpc worker-9738" 0x00007fb842754ad3 in ?? ()
73 LWP 9739 "rpc worker-9739" 0x00007fb842754ad3 in ?? ()
74 LWP 9740 "rpc worker-9740" 0x00007fb842754ad3 in ?? ()
75 LWP 9741 "rpc worker-9741" 0x00007fb842754ad3 in ?? ()
76 LWP 9742 "rpc worker-9742" 0x00007fb842754ad3 in ?? ()
77 LWP 9743 "rpc worker-9743" 0x00007fb842754ad3 in ?? ()
78 LWP 9744 "rpc worker-9744" 0x00007fb842754ad3 in ?? ()
79 LWP 9745 "rpc worker-9745" 0x00007fb842754ad3 in ?? ()
80 LWP 9746 "rpc worker-9746" 0x00007fb842754ad3 in ?? ()
81 LWP 9747 "rpc worker-9747" 0x00007fb842754ad3 in ?? ()
82 LWP 9748 "rpc worker-9748" 0x00007fb842754ad3 in ?? ()
83 LWP 9749 "rpc worker-9749" 0x00007fb842754ad3 in ?? ()
84 LWP 9750 "rpc worker-9750" 0x00007fb842754ad3 in ?? ()
85 LWP 9751 "rpc worker-9751" 0x00007fb842754ad3 in ?? ()
86 LWP 9752 "rpc worker-9752" 0x00007fb842754ad3 in ?? ()
87 LWP 9753 "rpc worker-9753" 0x00007fb842754ad3 in ?? ()
88 LWP 9754 "rpc worker-9754" 0x00007fb842754ad3 in ?? ()
89 LWP 9755 "rpc worker-9755" 0x00007fb842754ad3 in ?? ()
90 LWP 9756 "rpc worker-9756" 0x00007fb842754ad3 in ?? ()
91 LWP 9757 "rpc worker-9757" 0x00007fb842754ad3 in ?? ()
92 LWP 9758 "rpc worker-9758" 0x00007fb842754ad3 in ?? ()
93 LWP 9759 "rpc worker-9759" 0x00007fb842754ad3 in ?? ()
94 LWP 9760 "rpc worker-9760" 0x00007fb842754ad3 in ?? ()
95 LWP 9761 "rpc worker-9761" 0x00007fb842754ad3 in ?? ()
96 LWP 9762 "rpc worker-9762" 0x00007fb842754ad3 in ?? ()
97 LWP 9763 "rpc worker-9763" 0x00007fb842754ad3 in ?? ()
98 LWP 9764 "rpc worker-9764" 0x00007fb842754ad3 in ?? ()
99 LWP 9765 "rpc worker-9765" 0x00007fb842754ad3 in ?? ()
100 LWP 9766 "rpc worker-9766" 0x00007fb842754ad3 in ?? ()
101 LWP 9767 "rpc worker-9767" 0x00007fb842754ad3 in ?? ()
102 LWP 9768 "rpc worker-9768" 0x00007fb842754ad3 in ?? ()
103 LWP 9769 "rpc worker-9769" 0x00007fb842754ad3 in ?? ()
104 LWP 9770 "rpc worker-9770" 0x00007fb842754ad3 in ?? ()
105 LWP 9771 "rpc worker-9771" 0x00007fb842754ad3 in ?? ()
106 LWP 9772 "rpc worker-9772" 0x00007fb842754ad3 in ?? ()
107 LWP 9773 "rpc worker-9773" 0x00007fb842754ad3 in ?? ()
108 LWP 9774 "rpc worker-9774" 0x00007fb842754ad3 in ?? ()
109 LWP 9775 "rpc worker-9775" 0x00007fb842754ad3 in ?? ()
110 LWP 9776 "rpc worker-9776" 0x00007fb842754ad3 in ?? ()
111 LWP 9777 "rpc worker-9777" 0x00007fb842754ad3 in ?? ()
112 LWP 9778 "rpc worker-9778" 0x00007fb842754ad3 in ?? ()
113 LWP 9779 "rpc worker-9779" 0x00007fb842754ad3 in ?? ()
114 LWP 9780 "rpc worker-9780" 0x00007fb842754ad3 in ?? ()
115 LWP 9781 "rpc worker-9781" 0x00007fb842754ad3 in ?? ()
116 LWP 9782 "rpc worker-9782" 0x00007fb842754ad3 in ?? ()
117 LWP 9783 "diag-logger-978" 0x00007fb842754fb9 in ?? ()
118 LWP 9784 "result-tracker-" 0x00007fb842754fb9 in ?? ()
119 LWP 9785 "excess-log-dele" 0x00007fb842754fb9 in ?? ()
120 LWP 9786 "acceptor-9786" 0x00007fb83b4c70c7 in ?? ()
121 LWP 9787 "heartbeat-9787" 0x00007fb842754fb9 in ?? ()
122 LWP 9789 "maintenance_sch" 0x00007fb842754fb9 in ?? ()
Thread 122 (LWP 9789):
#0 0x00007fb842754fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000333 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007fb7f803f550 in ?? ()
#6 0x0000000000000666 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 9787):
#0 0x00007fb842754fb9 in ?? ()
#1 0x81afaaa989cdd16b in ?? ()
#2 0x00000000000000cc in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007fb7f88575b0 in ?? ()
#6 0x0000000000000198 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 9786):
#0 0x00007fb83b4c70c7 in ?? ()
#1 0x00006140000218a8 in ?? ()
#2 0x00007fb7f907ed50 in ?? ()
#3 0x00007fb7f907ed80 in ?? ()
#4 0x00007fb7f907ee80 in ?? ()
#5 0x00007fb7f907ed70 in ?? ()
#6 0x00007fb7f907ede0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007fb83f34bef0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 9785):
#0 0x00007fb842754fb9 in ?? ()
#1 0x00007fb7f9898f00 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffdc65748b0 in ?? ()
#5 0x00007fb7f9898ef0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 9784):
#0 0x00007fb842754fb9 in ?? ()
#1 0x00007fb7fa0b10c0 in ?? ()
#2 0x00000000000000cd in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007fb7fa0b10b0 in ?? ()
#6 0x000000000000019a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 9783):
#0 0x00007fb842754fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 9782):
#0 0x00007fb842754ad3 in ?? ()
#1 0x000000000000001f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007fb7fb0e0e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb7fb0e0e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007fb842754770 in ?? ()
#10 0x00007fb7fb0e0e70 in ?? ()
#11 0x00000ff6ff61c1d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 9781):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000029 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007fb7fb8f8e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb7fb8f8e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007fb842754770 in ?? ()
#10 0x00007fb7fb8f8e70 in ?? ()
#11 0x00000ff6ff71f1d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 9780):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 9779):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 9778):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 9777):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 9776):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 9775):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 9774):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 9773):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 9772):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 9771):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 9770):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 9769):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 9768):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 9767):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 9766):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 9765):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 9764):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 9763):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 9762):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 9761):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 9760):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 9759):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 9758):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 9757):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 9756):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 9755):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 9754):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 9753):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 9752):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 9751):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 9750):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 9749):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 9748):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 9747):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 9746):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 9745):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 9744):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 9743):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 9742):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007fb80f4b1e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb80f4b1e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 9741):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 9740):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 9739):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 9738):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 9737):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 9736):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 9734):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 9733):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 9732):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 9731):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 9730):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 9729):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 9728):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 9727):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 9726):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 9725):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 9724):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 9723):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 9722):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 9721):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007fb819692e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb819692e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 9720):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 9719):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 9718):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 9717):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 9716):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 9715):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 9714):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 9713):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 9712):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 9711):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 9710):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 9709):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 9708):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 9707):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 9706):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 9705):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 9704):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 9703):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 9702):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 9701):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007fb82305be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb82305be70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 9700):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 9699):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 9698):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 9697):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 9696):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 9695):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 9694):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 9693):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 9692):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 9691):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 9690):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 9689):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 9688):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 9687):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 9686):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 9685):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 9684):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 9683):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 9682):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 9681):
#0 0x00007fb842754fb9 in ?? ()
#1 0x00007fb82d0c3c80 in ?? ()
#2 0x0000000000000014 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001b360 in ?? ()
#5 0x00007fb82d0c3c70 in ?? ()
#6 0x0000000000000028 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 9680):
#0 0x00007fb842754fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 9679):
#0 0x00007fb842754fb9 in ?? ()
#1 0x00007fb82e0c51c0 in ?? ()
#2 0x0000000000000014 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007fb82e0c51b0 in ?? ()
#6 0x0000000000000028 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 9678):
#0 0x00007fb842754ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 9677):
#0 0x00007fb83b4c5a47 in ?? ()
#1 0x00007fb82f0c82e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007fb82f0c82d0 in ?? ()
#4 0x00007fb82f0c84e0 in ?? ()
#5 0x00007fb82f0c8320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007fb82f0c83a0 in ?? ()
#8 0x00007fb83db2525d in ?? ()
#9 0x3fb96ad16b86e000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000071800000 in ?? ()
#14 0x00007fb800000000 in ?? ()
#15 0x41d9e21ced720497 in ?? ()
#16 0x00000ff785e11074 in ?? ()
#17 0x00007fb82f0c8380 in ?? ()
#18 0x00007fb83db29ba3 in ?? ()
#19 0x00007fb82f0c8350 in ?? ()
#20 0x3fb96ad16b86e000 in ?? ()
#21 0x000000002f0c83a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb96ad16b86e000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 9676):
#0 0x00007fb83b4c5a47 in ?? ()
#1 0x00007fb82f8c92e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007fb82f8c92d0 in ?? ()
#4 0x00007fb82f8c94e0 in ?? ()
#5 0x00007fb82f8c9320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007fb82f8c93a0 in ?? ()
#8 0x00007fb83db2525d in ?? ()
#9 0x3fb967c0435a2000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x0000000042000000 in ?? ()
#14 0x00007fb800000000 in ?? ()
#15 0x41d9e21ced720496 in ?? ()
#16 0x00000ff785f11274 in ?? ()
#17 0x00007fb82f8c9380 in ?? ()
#18 0x00007fb83db29ba3 in ?? ()
#19 0x00007fb82f8c9350 in ?? ()
#20 0x3fb967c0435a2000 in ?? ()
#21 0x000000002f8c93a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb967c0435a2000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 9675):
#0 0x00007fb83b4c5a47 in ?? ()
#1 0x00007fb8300ca2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007fb8300ca2d0 in ?? ()
#4 0x00007fb8300ca4e0 in ?? ()
#5 0x00007fb8300ca320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007fb8300ca3a0 in ?? ()
#8 0x00007fb83db2525d in ?? ()
#9 0x3fb964c70c18c000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000077800000 in ?? ()
#14 0x00007fb800000000 in ?? ()
#15 0x41d9e21ced720497 in ?? ()
#16 0x00000ff786011474 in ?? ()
#17 0x00007fb8300ca380 in ?? ()
#18 0x00007fb83db29ba3 in ?? ()
#19 0x00007fb8300ca350 in ?? ()
#20 0x3fb964c70c18c000 in ?? ()
#21 0x00000000300ca3a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb964c70c18c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 9674):
#0 0x00007fb83b4c5a47 in ?? ()
#1 0x00007fb831cb32e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007fb831cb32d0 in ?? ()
#4 0x00007fb831cb34e0 in ?? ()
#5 0x00007fb831cb3320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fb831cb33a0 in ?? ()
#8 0x00007fb83db2525d in ?? ()
#9 0x3fb98cf65dc88000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x000000001c800000 in ?? ()
#14 0x00007fb800000000 in ?? ()
#15 0x41d9e21ced72049a in ?? ()
#16 0x00000ff78638e674 in ?? ()
#17 0x00007fb831cb3380 in ?? ()
#18 0x00007fb83db29ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 9671):
#0 0x00007fb83b4b8cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fb833cb7788 in ?? ()
#3 0x0000602000017cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 9670):
#0 0x00007fb842754fb9 in ?? ()
#1 0x00007fb8334b6080 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007fb8334b6070 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 9669):
#0 0x00007fb8427589e2 in ?? ()
#1 0x00007fb8324b2b60 in ?? ()
#2 0x00007fb8324b2c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fb8324b2bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007fb8324b2c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007fb8324b2c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007fb84807cf0c in ?? ()
#12 0x00007fb84806c7b9 in ?? ()
#13 0x00007fb848070da6 in ?? ()
#14 0x00007fb8480757e0 in ?? ()
#15 0x00007fb848075609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007fb83f36c98e in ?? ()
#18 0x00007fb84274e6db in ?? ()
#19 0x00000ff78648e47c in ?? ()
#20 0x00007fb8324b2400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 9663):
#0 0x00007fb842754fb9 in ?? ()
#1 0x00007fb834cb8c40 in ?? ()
#2 0x0000000000000406 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007fb834cb8c30 in ?? ()
#6 0x000000000000080c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 9662):
#0 0x00007fb842754fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 9661):
#0 0x00007fb842754fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 9660):
#0 0x00007fb842754fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fb8364bc410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 9659):
#0 0x00007fb842758d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250116 02:58:08.360538 8081 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 6 with UUID dfb2cd26d0bb42c99dbd98c0c1a2f731 and pid 9795
************************ BEGIN STACKS **************************
[New LWP 9797]
[New LWP 9798]
[New LWP 9799]
[New LWP 9800]
[New LWP 9806]
[New LWP 9807]
[New LWP 9808]
[New LWP 9811]
[New LWP 9812]
[New LWP 9813]
[New LWP 9814]
[New LWP 9815]
[New LWP 9816]
[New LWP 9817]
[New LWP 9818]
[New LWP 9819]
[New LWP 9820]
[New LWP 9821]
[New LWP 9822]
[New LWP 9823]
[New LWP 9824]
[New LWP 9825]
[New LWP 9826]
[New LWP 9827]
[New LWP 9828]
[New LWP 9829]
[New LWP 9830]
[New LWP 9831]
[New LWP 9832]
[New LWP 9833]
[New LWP 9834]
[New LWP 9835]
[New LWP 9836]
[New LWP 9837]
[New LWP 9838]
[New LWP 9839]
[New LWP 9840]
[New LWP 9841]
[New LWP 9842]
[New LWP 9843]
[New LWP 9844]
[New LWP 9845]
[New LWP 9846]
[New LWP 9847]
[New LWP 9848]
[New LWP 9849]
[New LWP 9850]
[New LWP 9851]
[New LWP 9852]
[New LWP 9853]
[New LWP 9854]
[New LWP 9855]
[New LWP 9856]
[New LWP 9857]
[New LWP 9858]
[New LWP 9859]
[New LWP 9860]
[New LWP 9861]
[New LWP 9862]
[New LWP 9863]
[New LWP 9864]
[New LWP 9865]
[New LWP 9866]
[New LWP 9867]
[New LWP 9868]
[New LWP 9869]
[New LWP 9870]
[New LWP 9872]
[New LWP 9873]
[New LWP 9874]
[New LWP 9875]
[New LWP 9876]
[New LWP 9877]
[New LWP 9878]
[New LWP 9879]
[New LWP 9880]
[New LWP 9881]
[New LWP 9882]
[New LWP 9883]
[New LWP 9884]
[New LWP 9885]
[New LWP 9886]
[New LWP 9887]
[New LWP 9888]
[New LWP 9889]
[New LWP 9890]
[New LWP 9891]
[New LWP 9892]
[New LWP 9893]
[New LWP 9894]
[New LWP 9895]
[New LWP 9896]
[New LWP 9897]
[New LWP 9898]
[New LWP 9899]
[New LWP 9900]
[New LWP 9901]
[New LWP 9902]
[New LWP 9903]
[New LWP 9905]
[New LWP 9906]
[New LWP 9907]
[New LWP 9908]
[New LWP 9909]
[New LWP 9910]
[New LWP 9911]
[New LWP 9912]
[New LWP 9913]
[New LWP 9914]
[New LWP 9915]
[New LWP 9916]
[New LWP 9917]
[New LWP 9918]
[New LWP 9919]
[New LWP 9920]
[New LWP 9921]
[New LWP 9922]
[New LWP 9923]
[New LWP 9924]
[New LWP 9925]
[New LWP 9926]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007fa20bc2bd50 in ?? ()
Id Target Id Frame
* 1 LWP 9795 "kudu" 0x00007fa20bc2bd50 in ?? ()
2 LWP 9797 "kudu" 0x00007fa20bc27fb9 in ?? ()
3 LWP 9798 "kudu" 0x00007fa20bc27fb9 in ?? ()
4 LWP 9799 "kudu" 0x00007fa20bc27fb9 in ?? ()
5 LWP 9800 "kernel-watcher-" 0x00007fa20bc27fb9 in ?? ()
6 LWP 9806 "ntp client-9806" 0x00007fa20bc2b9e2 in ?? ()
7 LWP 9807 "file cache-evic" 0x00007fa20bc27fb9 in ?? ()
8 LWP 9808 "sq_acceptor" 0x00007fa20498bcb9 in ?? ()
9 LWP 9811 "rpc reactor-981" 0x00007fa204998a47 in ?? ()
10 LWP 9812 "rpc reactor-981" 0x00007fa204998a47 in ?? ()
11 LWP 9813 "rpc reactor-981" 0x00007fa204998a47 in ?? ()
12 LWP 9814 "rpc reactor-981" 0x00007fa204998a47 in ?? ()
13 LWP 9815 "MaintenanceMgr " 0x00007fa20bc27ad3 in ?? ()
14 LWP 9816 "txn-status-mana" 0x00007fa20bc27fb9 in ?? ()
15 LWP 9817 "collect_and_rem" 0x00007fa20bc27fb9 in ?? ()
16 LWP 9818 "tc-session-exp-" 0x00007fa20bc27fb9 in ?? ()
17 LWP 9819 "rpc worker-9819" 0x00007fa20bc27ad3 in ?? ()
18 LWP 9820 "rpc worker-9820" 0x00007fa20bc27ad3 in ?? ()
19 LWP 9821 "rpc worker-9821" 0x00007fa20bc27ad3 in ?? ()
20 LWP 9822 "rpc worker-9822" 0x00007fa20bc27ad3 in ?? ()
21 LWP 9823 "rpc worker-9823" 0x00007fa20bc27ad3 in ?? ()
22 LWP 9824 "rpc worker-9824" 0x00007fa20bc27ad3 in ?? ()
23 LWP 9825 "rpc worker-9825" 0x00007fa20bc27ad3 in ?? ()
24 LWP 9826 "rpc worker-9826" 0x00007fa20bc27ad3 in ?? ()
25 LWP 9827 "rpc worker-9827" 0x00007fa20bc27ad3 in ?? ()
26 LWP 9828 "rpc worker-9828" 0x00007fa20bc27ad3 in ?? ()
27 LWP 9829 "rpc worker-9829" 0x00007fa20bc27ad3 in ?? ()
28 LWP 9830 "rpc worker-9830" 0x00007fa20bc27ad3 in ?? ()
29 LWP 9831 "rpc worker-9831" 0x00007fa20bc27ad3 in ?? ()
30 LWP 9832 "rpc worker-9832" 0x00007fa20bc27ad3 in ?? ()
31 LWP 9833 "rpc worker-9833" 0x00007fa20bc27ad3 in ?? ()
32 LWP 9834 "rpc worker-9834" 0x00007fa20bc27ad3 in ?? ()
33 LWP 9835 "rpc worker-9835" 0x00007fa20bc27ad3 in ?? ()
34 LWP 9836 "rpc worker-9836" 0x00007fa20bc27ad3 in ?? ()
35 LWP 9837 "rpc worker-9837" 0x00007fa20bc27ad3 in ?? ()
36 LWP 9838 "rpc worker-9838" 0x00007fa20bc27ad3 in ?? ()
37 LWP 9839 "rpc worker-9839" 0x00007fa20bc27ad3 in ?? ()
38 LWP 9840 "rpc worker-9840" 0x00007fa20bc27ad3 in ?? ()
39 LWP 9841 "rpc worker-9841" 0x00007fa20bc27ad3 in ?? ()
40 LWP 9842 "rpc worker-9842" 0x00007fa20bc27ad3 in ?? ()
41 LWP 9843 "rpc worker-9843" 0x00007fa20bc27ad3 in ?? ()
42 LWP 9844 "rpc worker-9844" 0x00007fa20bc27ad3 in ?? ()
43 LWP 9845 "rpc worker-9845" 0x00007fa20bc27ad3 in ?? ()
44 LWP 9846 "rpc worker-9846" 0x00007fa20bc27ad3 in ?? ()
45 LWP 9847 "rpc worker-9847" 0x00007fa20bc27ad3 in ?? ()
46 LWP 9848 "rpc worker-9848" 0x00007fa20bc27ad3 in ?? ()
47 LWP 9849 "rpc worker-9849" 0x00007fa20bc27ad3 in ?? ()
48 LWP 9850 "rpc worker-9850" 0x00007fa20bc27ad3 in ?? ()
49 LWP 9851 "rpc worker-9851" 0x00007fa20bc27ad3 in ?? ()
50 LWP 9852 "rpc worker-9852" 0x00007fa20bc27ad3 in ?? ()
51 LWP 9853 "rpc worker-9853" 0x00007fa20bc27ad3 in ?? ()
52 LWP 9854 "rpc worker-9854" 0x00007fa20bc27ad3 in ?? ()
53 LWP 9855 "rpc worker-9855" 0x00007fa20bc27ad3 in ?? ()
54 LWP 9856 "rpc worker-9856" 0x00007fa20bc27ad3 in ?? ()
55 LWP 9857 "rpc worker-9857" 0x00007fa20bc27ad3 in ?? ()
56 LWP 9858 "rpc worker-9858" 0x00007fa20bc27ad3 in ?? ()
57 LWP 9859 "rpc worker-9859" 0x00007fa20bc27ad3 in ?? ()
58 LWP 9860 "rpc worker-9860" 0x00007fa20bc27ad3 in ?? ()
59 LWP 9861 "rpc worker-9861" 0x00007fa20bc27ad3 in ?? ()
60 LWP 9862 "rpc worker-9862" 0x00007fa20bc27ad3 in ?? ()
61 LWP 9863 "rpc worker-9863" 0x00007fa20bc27ad3 in ?? ()
62 LWP 9864 "rpc worker-9864" 0x00007fa20bc27ad3 in ?? ()
63 LWP 9865 "rpc worker-9865" 0x00007fa20bc27ad3 in ?? ()
64 LWP 9866 "rpc worker-9866" 0x00007fa20bc27ad3 in ?? ()
65 LWP 9867 "rpc worker-9867" 0x00007fa20bc27ad3 in ?? ()
66 LWP 9868 "rpc worker-9868" 0x00007fa20bc27ad3 in ?? ()
67 LWP 9869 "rpc worker-9869" 0x00007fa20bc27ad3 in ?? ()
68 LWP 9870 "rpc worker-9870" 0x00007fa20bc27ad3 in ?? ()
69 LWP 9872 "rpc worker-9872" 0x00007fa20bc27ad3 in ?? ()
70 LWP 9873 "rpc worker-9873" 0x00007fa20bc27ad3 in ?? ()
71 LWP 9874 "rpc worker-9874" 0x00007fa20bc27ad3 in ?? ()
72 LWP 9875 "rpc worker-9875" 0x00007fa20bc27ad3 in ?? ()
73 LWP 9876 "rpc worker-9876" 0x00007fa20bc27ad3 in ?? ()
74 LWP 9877 "rpc worker-9877" 0x00007fa20bc27ad3 in ?? ()
75 LWP 9878 "rpc worker-9878" 0x00007fa20bc27ad3 in ?? ()
76 LWP 9879 "rpc worker-9879" 0x00007fa20bc27ad3 in ?? ()
77 LWP 9880 "rpc worker-9880" 0x00007fa20bc27ad3 in ?? ()
78 LWP 9881 "rpc worker-9881" 0x00007fa20bc27ad3 in ?? ()
79 LWP 9882 "rpc worker-9882" 0x00007fa20bc27ad3 in ?? ()
80 LWP 9883 "rpc worker-9883" 0x00007fa20bc27ad3 in ?? ()
81 LWP 9884 "rpc worker-9884" 0x00007fa20bc27ad3 in ?? ()
82 LWP 9885 "rpc worker-9885" 0x00007fa20bc27ad3 in ?? ()
83 LWP 9886 "rpc worker-9886" 0x00007fa20bc27ad3 in ?? ()
84 LWP 9887 "rpc worker-9887" 0x00007fa20bc27ad3 in ?? ()
85 LWP 9888 "rpc worker-9888" 0x00007fa20bc27ad3 in ?? ()
86 LWP 9889 "rpc worker-9889" 0x00007fa20bc27ad3 in ?? ()
87 LWP 9890 "rpc worker-9890" 0x00007fa20bc27ad3 in ?? ()
88 LWP 9891 "rpc worker-9891" 0x00007fa20bc27ad3 in ?? ()
89 LWP 9892 "rpc worker-9892" 0x00007fa20bc27ad3 in ?? ()
90 LWP 9893 "rpc worker-9893" 0x00007fa20bc27ad3 in ?? ()
91 LWP 9894 "rpc worker-9894" 0x00007fa20bc27ad3 in ?? ()
92 LWP 9895 "rpc worker-9895" 0x00007fa20bc27ad3 in ?? ()
93 LWP 9896 "rpc worker-9896" 0x00007fa20bc27ad3 in ?? ()
94 LWP 9897 "rpc worker-9897" 0x00007fa20bc27ad3 in ?? ()
95 LWP 9898 "rpc worker-9898" 0x00007fa20bc27ad3 in ?? ()
96 LWP 9899 "rpc worker-9899" 0x00007fa20bc27ad3 in ?? ()
97 LWP 9900 "rpc worker-9900" 0x00007fa20bc27ad3 in ?? ()
98 LWP 9901 "rpc worker-9901" 0x00007fa20bc27ad3 in ?? ()
99 LWP 9902 "rpc worker-9902" 0x00007fa20bc27ad3 in ?? ()
100 LWP 9903 "rpc worker-9903" 0x00007fa20bc27ad3 in ?? ()
101 LWP 9905 "rpc worker-9905" 0x00007fa20bc27ad3 in ?? ()
102 LWP 9906 "rpc worker-9906" 0x00007fa20bc27ad3 in ?? ()
103 LWP 9907 "rpc worker-9907" 0x00007fa20bc27ad3 in ?? ()
104 LWP 9908 "rpc worker-9908" 0x00007fa20bc27ad3 in ?? ()
105 LWP 9909 "rpc worker-9909" 0x00007fa20bc27ad3 in ?? ()
106 LWP 9910 "rpc worker-9910" 0x00007fa20bc27ad3 in ?? ()
107 LWP 9911 "rpc worker-9911" 0x00007fa20bc27ad3 in ?? ()
108 LWP 9912 "rpc worker-9912" 0x00007fa20bc27ad3 in ?? ()
109 LWP 9913 "rpc worker-9913" 0x00007fa20bc27ad3 in ?? ()
110 LWP 9914 "rpc worker-9914" 0x00007fa20bc27ad3 in ?? ()
111 LWP 9915 "rpc worker-9915" 0x00007fa20bc27ad3 in ?? ()
112 LWP 9916 "rpc worker-9916" 0x00007fa20bc27ad3 in ?? ()
113 LWP 9917 "rpc worker-9917" 0x00007fa20bc27ad3 in ?? ()
114 LWP 9918 "rpc worker-9918" 0x00007fa20bc27ad3 in ?? ()
115 LWP 9919 "rpc worker-9919" 0x00007fa20bc27ad3 in ?? ()
116 LWP 9920 "rpc worker-9920" 0x00007fa20bc27ad3 in ?? ()
117 LWP 9921 "diag-logger-992" 0x00007fa20bc27fb9 in ?? ()
118 LWP 9922 "result-tracker-" 0x00007fa20bc27fb9 in ?? ()
119 LWP 9923 "excess-log-dele" 0x00007fa20bc27fb9 in ?? ()
120 LWP 9924 "acceptor-9924" 0x00007fa20499a0c7 in ?? ()
121 LWP 9925 "heartbeat-9925" 0x00007fa20bc27fb9 in ?? ()
122 LWP 9926 "maintenance_sch" 0x00007fa20bc27fb9 in ?? ()
Thread 122 (LWP 9926):
#0 0x00007fa20bc27fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000032e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007fa1c1512550 in ?? ()
#6 0x000000000000065c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 9925):
#0 0x00007fa20bc27fb9 in ?? ()
#1 0x81afaaa989cdd16b in ?? ()
#2 0x00000000000000cb in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007fa1c1d2a5b0 in ?? ()
#6 0x0000000000000196 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 9924):
#0 0x00007fa20499a0c7 in ?? ()
#1 0x00006140000216a8 in ?? ()
#2 0x00007fa1c2551d50 in ?? ()
#3 0x00007fa1c2551d80 in ?? ()
#4 0x00007fa1c2551e80 in ?? ()
#5 0x00007fa1c2551d70 in ?? ()
#6 0x00007fa1c2551de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007fa20881eef0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 9923):
#0 0x00007fa20bc27fb9 in ?? ()
#1 0x00007fa1c2d6bf00 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff23559410 in ?? ()
#5 0x00007fa1c2d6bef0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 9922):
#0 0x00007fa20bc27fb9 in ?? ()
#1 0x00007fa1c35840c0 in ?? ()
#2 0x00000000000000cb in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007fa1c35840b0 in ?? ()
#6 0x0000000000000196 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 9921):
#0 0x00007fa20bc27fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 9920):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x000000000000001c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007fa1c45b3e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa1c45b3e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 9919):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x000000000000002c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002a6858 in ?? ()
#4 0x00007fa1c4dcbe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa1c4dcbe70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 9918):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 9917):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 9916):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 9915):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 9914):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 9913):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 9912):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 9911):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 9910):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 9909):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 9908):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 9907):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 9906):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 9905):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 9903):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 9902):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 9901):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 9900):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 9899):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 9898):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 9897):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 9896):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 9895):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 9894):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 9893):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 9892):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 9891):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 9890):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 9889):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 9888):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 9887):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 9886):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 9885):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 9884):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 9883):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 9882):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 9881):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 9880):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 9879):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007fa1d8984e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa1d8984e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 9878):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 9877):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 9876):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 9875):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 9874):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 9873):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 9872):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 9870):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 9869):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 9868):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 9867):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 9866):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 9865):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 9864):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 9863):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 9862):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 9861):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 9860):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 9859):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 9858):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007fa1e234de50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa1e234de70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 9857):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 9856):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 9855):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 9854):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 9853):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 9852):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 9851):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 9850):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 9849):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 9848):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 9847):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 9846):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 9845):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 9844):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 9843):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 9842):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 9841):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 9840):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 9839):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 9838):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007fa1ec52ee50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa1ec52ee70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 9837):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 9836):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 9835):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 9834):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 9833):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 9832):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 9831):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 9830):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 9829):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 9828):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 9827):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 9826):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 9825):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 9824):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 9823):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 9822):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 9821):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 9820):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 9819):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 9818):
#0 0x00007fa20bc27fb9 in ?? ()
#1 0x00007fa1f65c3c80 in ?? ()
#2 0x0000000000000014 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001b360 in ?? ()
#5 0x00007fa1f65c3c70 in ?? ()
#6 0x0000000000000028 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 9817):
#0 0x00007fa20bc27fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 9816):
#0 0x00007fa20bc27fb9 in ?? ()
#1 0x00007fa1f75c51c0 in ?? ()
#2 0x0000000000000014 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007fa1f75c51b0 in ?? ()
#6 0x0000000000000028 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 9815):
#0 0x00007fa20bc27ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 9814):
#0 0x00007fa204998a47 in ?? ()
#1 0x00007fa1f85c82e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007fa1f85c82d0 in ?? ()
#4 0x00007fa1f85c84e0 in ?? ()
#5 0x00007fa1f85c8320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007fa1f85c83a0 in ?? ()
#8 0x00007fa206ff825d in ?? ()
#9 0x3fb958fe1599c000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000079000000 in ?? ()
#14 0x00007fa100000000 in ?? ()
#15 0x41d9e21ced72049a in ?? ()
#16 0x00000ff4bf0b1074 in ?? ()
#17 0x00007fa1f85c8380 in ?? ()
#18 0x00007fa206ffcba3 in ?? ()
#19 0x00007fa1f85c8350 in ?? ()
#20 0x3fb958fe1599c000 in ?? ()
#21 0x00000000f85c83a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb958fe1599c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 9813):
#0 0x00007fa204998a47 in ?? ()
#1 0x00007fa1f8dc92e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007fa1f8dc92d0 in ?? ()
#4 0x00007fa1f8dc94e0 in ?? ()
#5 0x00007fa1f8dc9320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007fa1f8dc93a0 in ?? ()
#8 0x00007fa206ff825d in ?? ()
#9 0x3fb9571c9ca10000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x00000000dc000000 in ?? ()
#14 0x00007fa100000000 in ?? ()
#15 0x41d9e21ced720495 in ?? ()
#16 0x00000ff4bf1b1274 in ?? ()
#17 0x00007fa1f8dc9380 in ?? ()
#18 0x00007fa206ffcba3 in ?? ()
#19 0x00007fa1f8dc9350 in ?? ()
#20 0x3fb9571c9ca10000 in ?? ()
#21 0x00000000f8dc93a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb9571c9ca10000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 9812):
#0 0x00007fa204998a47 in ?? ()
#1 0x00007fa1f95ca2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007fa1f95ca2d0 in ?? ()
#4 0x00007fa1f95ca4e0 in ?? ()
#5 0x00007fa1f95ca320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007fa1f95ca3a0 in ?? ()
#8 0x00007fa206ff825d in ?? ()
#9 0x3fb95188afab0000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000056000000 in ?? ()
#14 0x00007fa100000000 in ?? ()
#15 0x41d9e21ced720495 in ?? ()
#16 0x00000ff4bf2b1474 in ?? ()
#17 0x00007fa1f95ca380 in ?? ()
#18 0x00007fa206ffcba3 in ?? ()
#19 0x00007fa1f95ca350 in ?? ()
#20 0x3fb95188afab0000 in ?? ()
#21 0x00000000f95ca3a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb95188afab0000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 9811):
#0 0x00007fa204998a47 in ?? ()
#1 0x00007fa1fb1b32e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007fa1fb1b32d0 in ?? ()
#4 0x00007fa1fb1b34e0 in ?? ()
#5 0x00007fa1fb1b3320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fa1fb1b33a0 in ?? ()
#8 0x00007fa206ff825d in ?? ()
#9 0x3fb96c9efa5ee000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x000000008d000000 in ?? ()
#14 0x00007fa100000000 in ?? ()
#15 0x41d9e21ced720496 in ?? ()
#16 0x00000ff4bf62e674 in ?? ()
#17 0x00007fa1fb1b3380 in ?? ()
#18 0x00007fa206ffcba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 9808):
#0 0x00007fa20498bcb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fa1fd1b7788 in ?? ()
#3 0x0000602000017cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 9807):
#0 0x00007fa20bc27fb9 in ?? ()
#1 0x00007fa1fc9b6080 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007fa1fc9b6070 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 9806):
#0 0x00007fa20bc2b9e2 in ?? ()
#1 0x00007fa1fb9b2b60 in ?? ()
#2 0x00007fa1fb9b2c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fa1fb9b2bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007fa1fb9b2c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007fa1fb9b2c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007fa21154ff0c in ?? ()
#12 0x00007fa21153f7b9 in ?? ()
#13 0x00007fa211543da6 in ?? ()
#14 0x00007fa2115487e0 in ?? ()
#15 0x00007fa211548609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007fa20883f98e in ?? ()
#18 0x00007fa20bc216db in ?? ()
#19 0x00000ff4bf72e47c in ?? ()
#20 0x00007fa1fb9b2400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 9800):
#0 0x00007fa20bc27fb9 in ?? ()
#1 0x00007fa1fe1b8c40 in ?? ()
#2 0x00000000000003fc in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007fa1fe1b8c30 in ?? ()
#6 0x00000000000007f8 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 9799):
#0 0x00007fa20bc27fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 9798):
#0 0x00007fa20bc27fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 9797):
#0 0x00007fa20bc27fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fa1ff9bc410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 9795):
#0 0x00007fa20bc2bd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250116 02:58:09.118683 8081 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 7 with UUID 9b5741c935d64e23af0c52c50e78420c and pid 11004
************************ BEGIN STACKS **************************
[New LWP 11006]
[New LWP 11007]
[New LWP 11008]
[New LWP 11009]
[New LWP 11015]
[New LWP 11016]
[New LWP 11017]
[New LWP 11020]
[New LWP 11021]
[New LWP 11022]
[New LWP 11023]
[New LWP 11024]
[New LWP 11025]
[New LWP 11026]
[New LWP 11028]
[New LWP 11029]
[New LWP 11030]
[New LWP 11032]
[New LWP 11033]
[New LWP 11034]
[New LWP 11035]
[New LWP 11036]
[New LWP 11037]
[New LWP 11038]
[New LWP 11039]
[New LWP 11040]
[New LWP 11041]
[New LWP 11042]
[New LWP 11043]
[New LWP 11044]
[New LWP 11045]
[New LWP 11046]
[New LWP 11047]
[New LWP 11048]
[New LWP 11049]
[New LWP 11050]
[New LWP 11051]
[New LWP 11052]
[New LWP 11053]
[New LWP 11054]
[New LWP 11055]
[New LWP 11056]
[New LWP 11058]
[New LWP 11059]
[New LWP 11060]
[New LWP 11061]
[New LWP 11062]
[New LWP 11063]
[New LWP 11064]
[New LWP 11065]
[New LWP 11066]
[New LWP 11067]
[New LWP 11068]
[New LWP 11069]
[New LWP 11070]
[New LWP 11071]
[New LWP 11072]
[New LWP 11073]
[New LWP 11074]
[New LWP 11075]
[New LWP 11076]
[New LWP 11077]
[New LWP 11078]
[New LWP 11079]
[New LWP 11080]
[New LWP 11081]
[New LWP 11082]
[New LWP 11083]
[New LWP 11084]
[New LWP 11085]
[New LWP 11086]
[New LWP 11087]
[New LWP 11088]
[New LWP 11089]
[New LWP 11090]
[New LWP 11091]
[New LWP 11092]
[New LWP 11093]
[New LWP 11094]
[New LWP 11095]
[New LWP 11096]
[New LWP 11097]
[New LWP 11098]
[New LWP 11099]
[New LWP 11100]
[New LWP 11101]
[New LWP 11102]
[New LWP 11103]
[New LWP 11104]
[New LWP 11105]
[New LWP 11106]
[New LWP 11107]
[New LWP 11108]
[New LWP 11109]
[New LWP 11110]
[New LWP 11111]
[New LWP 11112]
[New LWP 11113]
[New LWP 11114]
[New LWP 11115]
[New LWP 11116]
[New LWP 11117]
[New LWP 11118]
[New LWP 11119]
[New LWP 11120]
[New LWP 11121]
[New LWP 11122]
[New LWP 11123]
[New LWP 11124]
[New LWP 11125]
[New LWP 11126]
[New LWP 11127]
[New LWP 11128]
[New LWP 11129]
[New LWP 11130]
[New LWP 11131]
[New LWP 11132]
[New LWP 11133]
[New LWP 11134]
[New LWP 11135]
[New LWP 11138]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007fbdf9974d50 in ?? ()
Id Target Id Frame
* 1 LWP 11004 "kudu" 0x00007fbdf9974d50 in ?? ()
2 LWP 11006 "kudu" 0x00007fbdf9970fb9 in ?? ()
3 LWP 11007 "kudu" 0x00007fbdf9970fb9 in ?? ()
4 LWP 11008 "kudu" 0x00007fbdf9970fb9 in ?? ()
5 LWP 11009 "kernel-watcher-" 0x00007fbdf9970fb9 in ?? ()
6 LWP 11015 "ntp client-1101" 0x00007fbdf99749e2 in ?? ()
7 LWP 11016 "file cache-evic" 0x00007fbdf9970fb9 in ?? ()
8 LWP 11017 "sq_acceptor" 0x00007fbdf26d4cb9 in ?? ()
9 LWP 11020 "rpc reactor-110" 0x00007fbdf26e1a47 in ?? ()
10 LWP 11021 "rpc reactor-110" 0x00007fbdf26e1a47 in ?? ()
11 LWP 11022 "rpc reactor-110" 0x00007fbdf26e1a47 in ?? ()
12 LWP 11023 "rpc reactor-110" 0x00007fbdf26e1a47 in ?? ()
13 LWP 11024 "MaintenanceMgr " 0x00007fbdf9970ad3 in ?? ()
14 LWP 11025 "txn-status-mana" 0x00007fbdf9970fb9 in ?? ()
15 LWP 11026 "collect_and_rem" 0x00007fbdf9970fb9 in ?? ()
16 LWP 11028 "tc-session-exp-" 0x00007fbdf9970fb9 in ?? ()
17 LWP 11029 "rpc worker-1102" 0x00007fbdf9970ad3 in ?? ()
18 LWP 11030 "rpc worker-1103" 0x00007fbdf9970ad3 in ?? ()
19 LWP 11032 "rpc worker-1103" 0x00007fbdf9970ad3 in ?? ()
20 LWP 11033 "rpc worker-1103" 0x00007fbdf9970ad3 in ?? ()
21 LWP 11034 "rpc worker-1103" 0x00007fbdf9970ad3 in ?? ()
22 LWP 11035 "rpc worker-1103" 0x00007fbdf9970ad3 in ?? ()
23 LWP 11036 "rpc worker-1103" 0x00007fbdf9970ad3 in ?? ()
24 LWP 11037 "rpc worker-1103" 0x00007fbdf9970ad3 in ?? ()
25 LWP 11038 "rpc worker-1103" 0x00007fbdf9970ad3 in ?? ()
26 LWP 11039 "rpc worker-1103" 0x00007fbdf9970ad3 in ?? ()
27 LWP 11040 "rpc worker-1104" 0x00007fbdf9970ad3 in ?? ()
28 LWP 11041 "rpc worker-1104" 0x00007fbdf9970ad3 in ?? ()
29 LWP 11042 "rpc worker-1104" 0x00007fbdf9970ad3 in ?? ()
30 LWP 11043 "rpc worker-1104" 0x00007fbdf9970ad3 in ?? ()
31 LWP 11044 "rpc worker-1104" 0x00007fbdf9970ad3 in ?? ()
32 LWP 11045 "rpc worker-1104" 0x00007fbdf9970ad3 in ?? ()
33 LWP 11046 "rpc worker-1104" 0x00007fbdf9970ad3 in ?? ()
34 LWP 11047 "rpc worker-1104" 0x00007fbdf9970ad3 in ?? ()
35 LWP 11048 "rpc worker-1104" 0x00007fbdf9970ad3 in ?? ()
36 LWP 11049 "rpc worker-1104" 0x00007fbdf9970ad3 in ?? ()
37 LWP 11050 "rpc worker-1105" 0x00007fbdf9970ad3 in ?? ()
38 LWP 11051 "rpc worker-1105" 0x00007fbdf9970ad3 in ?? ()
39 LWP 11052 "rpc worker-1105" 0x00007fbdf9970ad3 in ?? ()
40 LWP 11053 "rpc worker-1105" 0x00007fbdf9970ad3 in ?? ()
41 LWP 11054 "rpc worker-1105" 0x00007fbdf9970ad3 in ?? ()
42 LWP 11055 "rpc worker-1105" 0x00007fbdf9970ad3 in ?? ()
43 LWP 11056 "rpc worker-1105" 0x00007fbdf9970ad3 in ?? ()
44 LWP 11058 "rpc worker-1105" 0x00007fbdf9970ad3 in ?? ()
45 LWP 11059 "rpc worker-1105" 0x00007fbdf9970ad3 in ?? ()
46 LWP 11060 "rpc worker-1106" 0x00007fbdf9970ad3 in ?? ()
47 LWP 11061 "rpc worker-1106" 0x00007fbdf9970ad3 in ?? ()
48 LWP 11062 "rpc worker-1106" 0x00007fbdf9970ad3 in ?? ()
49 LWP 11063 "rpc worker-1106" 0x00007fbdf9970ad3 in ?? ()
50 LWP 11064 "rpc worker-1106" 0x00007fbdf9970ad3 in ?? ()
51 LWP 11065 "rpc worker-1106" 0x00007fbdf9970ad3 in ?? ()
52 LWP 11066 "rpc worker-1106" 0x00007fbdf9970ad3 in ?? ()
53 LWP 11067 "rpc worker-1106" 0x00007fbdf9970ad3 in ?? ()
54 LWP 11068 "rpc worker-1106" 0x00007fbdf9970ad3 in ?? ()
55 LWP 11069 "rpc worker-1106" 0x00007fbdf9970ad3 in ?? ()
56 LWP 11070 "rpc worker-1107" 0x00007fbdf9970ad3 in ?? ()
57 LWP 11071 "rpc worker-1107" 0x00007fbdf9970ad3 in ?? ()
58 LWP 11072 "rpc worker-1107" 0x00007fbdf9970ad3 in ?? ()
59 LWP 11073 "rpc worker-1107" 0x00007fbdf9970ad3 in ?? ()
60 LWP 11074 "rpc worker-1107" 0x00007fbdf9970ad3 in ?? ()
61 LWP 11075 "rpc worker-1107" 0x00007fbdf9970ad3 in ?? ()
62 LWP 11076 "rpc worker-1107" 0x00007fbdf9970ad3 in ?? ()
63 LWP 11077 "rpc worker-1107" 0x00007fbdf9970ad3 in ?? ()
64 LWP 11078 "rpc worker-1107" 0x00007fbdf9970ad3 in ?? ()
65 LWP 11079 "rpc worker-1107" 0x00007fbdf9970ad3 in ?? ()
66 LWP 11080 "rpc worker-1108" 0x00007fbdf9970ad3 in ?? ()
67 LWP 11081 "rpc worker-1108" 0x00007fbdf9970ad3 in ?? ()
68 LWP 11082 "rpc worker-1108" 0x00007fbdf9970ad3 in ?? ()
69 LWP 11083 "rpc worker-1108" 0x00007fbdf9970ad3 in ?? ()
70 LWP 11084 "rpc worker-1108" 0x00007fbdf9970ad3 in ?? ()
71 LWP 11085 "rpc worker-1108" 0x00007fbdf9970ad3 in ?? ()
72 LWP 11086 "rpc worker-1108" 0x00007fbdf9970ad3 in ?? ()
73 LWP 11087 "rpc worker-1108" 0x00007fbdf9970ad3 in ?? ()
74 LWP 11088 "rpc worker-1108" 0x00007fbdf9970ad3 in ?? ()
75 LWP 11089 "rpc worker-1108" 0x00007fbdf9970ad3 in ?? ()
76 LWP 11090 "rpc worker-1109" 0x00007fbdf9970ad3 in ?? ()
77 LWP 11091 "rpc worker-1109" 0x00007fbdf9970ad3 in ?? ()
78 LWP 11092 "rpc worker-1109" 0x00007fbdf9970ad3 in ?? ()
79 LWP 11093 "rpc worker-1109" 0x00007fbdf9970ad3 in ?? ()
80 LWP 11094 "rpc worker-1109" 0x00007fbdf9970ad3 in ?? ()
81 LWP 11095 "rpc worker-1109" 0x00007fbdf9970ad3 in ?? ()
82 LWP 11096 "rpc worker-1109" 0x00007fbdf9970ad3 in ?? ()
83 LWP 11097 "rpc worker-1109" 0x00007fbdf9970ad3 in ?? ()
84 LWP 11098 "rpc worker-1109" 0x00007fbdf9970ad3 in ?? ()
85 LWP 11099 "rpc worker-1109" 0x00007fbdf9970ad3 in ?? ()
86 LWP 11100 "rpc worker-1110" 0x00007fbdf9970ad3 in ?? ()
87 LWP 11101 "rpc worker-1110" 0x00007fbdf9970ad3 in ?? ()
88 LWP 11102 "rpc worker-1110" 0x00007fbdf9970ad3 in ?? ()
89 LWP 11103 "rpc worker-1110" 0x00007fbdf9970ad3 in ?? ()
90 LWP 11104 "rpc worker-1110" 0x00007fbdf9970ad3 in ?? ()
91 LWP 11105 "rpc worker-1110" 0x00007fbdf9970ad3 in ?? ()
92 LWP 11106 "rpc worker-1110" 0x00007fbdf9970ad3 in ?? ()
93 LWP 11107 "rpc worker-1110" 0x00007fbdf9970ad3 in ?? ()
94 LWP 11108 "rpc worker-1110" 0x00007fbdf9970ad3 in ?? ()
95 LWP 11109 "rpc worker-1110" 0x00007fbdf9970ad3 in ?? ()
96 LWP 11110 "rpc worker-1111" 0x00007fbdf9970ad3 in ?? ()
97 LWP 11111 "rpc worker-1111" 0x00007fbdf9970ad3 in ?? ()
98 LWP 11112 "rpc worker-1111" 0x00007fbdf9970ad3 in ?? ()
99 LWP 11113 "rpc worker-1111" 0x00007fbdf9970ad3 in ?? ()
100 LWP 11114 "rpc worker-1111" 0x00007fbdf9970ad3 in ?? ()
101 LWP 11115 "rpc worker-1111" 0x00007fbdf9970ad3 in ?? ()
102 LWP 11116 "rpc worker-1111" 0x00007fbdf9970ad3 in ?? ()
103 LWP 11117 "rpc worker-1111" 0x00007fbdf9970ad3 in ?? ()
104 LWP 11118 "rpc worker-1111" 0x00007fbdf9970ad3 in ?? ()
105 LWP 11119 "rpc worker-1111" 0x00007fbdf9970ad3 in ?? ()
106 LWP 11120 "rpc worker-1112" 0x00007fbdf9970ad3 in ?? ()
107 LWP 11121 "rpc worker-1112" 0x00007fbdf9970ad3 in ?? ()
108 LWP 11122 "rpc worker-1112" 0x00007fbdf9970ad3 in ?? ()
109 LWP 11123 "rpc worker-1112" 0x00007fbdf9970ad3 in ?? ()
110 LWP 11124 "rpc worker-1112" 0x00007fbdf9970ad3 in ?? ()
111 LWP 11125 "rpc worker-1112" 0x00007fbdf9970ad3 in ?? ()
112 LWP 11126 "rpc worker-1112" 0x00007fbdf9970ad3 in ?? ()
113 LWP 11127 "rpc worker-1112" 0x00007fbdf9970ad3 in ?? ()
114 LWP 11128 "rpc worker-1112" 0x00007fbdf9970ad3 in ?? ()
115 LWP 11129 "rpc worker-1112" 0x00007fbdf9970ad3 in ?? ()
116 LWP 11130 "rpc worker-1113" 0x00007fbdf9970ad3 in ?? ()
117 LWP 11131 "diag-logger-111" 0x00007fbdf9970fb9 in ?? ()
118 LWP 11132 "result-tracker-" 0x00007fbdf9970fb9 in ?? ()
119 LWP 11133 "excess-log-dele" 0x00007fbdf9970fb9 in ?? ()
120 LWP 11134 "acceptor-11134" 0x00007fbdf26e30c7 in ?? ()
121 LWP 11135 "heartbeat-11135" 0x00007fbdf9970fb9 in ?? ()
122 LWP 11138 "maintenance_sch" 0x00007fbdf9970fb9 in ?? ()
Thread 122 (LWP 11138):
#0 0x00007fbdf9970fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000022e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007fbdaf25b550 in ?? ()
#6 0x000000000000045c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 11135):
#0 0x00007fbdf9970fb9 in ?? ()
#1 0x81afaaa989cdd16b in ?? ()
#2 0x000000000000008b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007fbdafa735b0 in ?? ()
#6 0x0000000000000116 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 11134):
#0 0x00007fbdf26e30c7 in ?? ()
#1 0x0000614000021ca8 in ?? ()
#2 0x00007fbdb029ad50 in ?? ()
#3 0x00007fbdb029ad80 in ?? ()
#4 0x00007fbdb029ae80 in ?? ()
#5 0x00007fbdb029ad70 in ?? ()
#6 0x00007fbdb029ade0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 11133):
#0 0x00007fbdf9970fb9 in ?? ()
#1 0x00007fbdb0ab4f00 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffcb9f74f50 in ?? ()
#5 0x00007fbdb0ab4ef0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 11132):
#0 0x00007fbdf9970fb9 in ?? ()
#1 0x00007fbdb12cd0c0 in ?? ()
#2 0x000000000000008b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007fbdb12cd0b0 in ?? ()
#6 0x0000000000000116 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 11131):
#0 0x00007fbdf9970fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 11130):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x000000000000001f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007fbdb22fce50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fbdb22fce70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007fbdf9970770 in ?? ()
#10 0x00007fbdb22fce70 in ?? ()
#11 0x00000ff7b645f9d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 11129):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000019 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007fbdb2b14e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fbdb2b14e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007fbdf9970770 in ?? ()
#10 0x00007fbdb2b14e70 in ?? ()
#11 0x00000ff7b65629d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 11128):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 11127):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 11126):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 11125):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 11124):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 11123):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 11122):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 11121):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 11120):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 11119):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 11118):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 11117):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 11116):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 11115):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 11114):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 11113):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 11112):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 11111):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 11110):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 11109):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 11108):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 11107):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 11106):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 11105):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 11104):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 11103):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 11102):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 11101):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 11100):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 11099):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 11098):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 11097):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 11096):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 11095):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 11094):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 11093):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 11092):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 11091):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 11090):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x000000000000000e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007fbdc66cde50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fbdc66cde70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 11089):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 11088):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 11087):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 11086):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 11085):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 11084):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 11083):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 11082):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 11081):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 11080):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 11079):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 11078):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 11077):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 11076):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 11075):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 11074):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 11073):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 11072):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 11071):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 11070):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x000000000000000e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007fbdd08aee50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fbdd08aee70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 11069):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 11068):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 11067):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 11066):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 11065):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 11064):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 11063):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 11062):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 11061):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 11060):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 11059):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 11058):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 11056):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 11055):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 11054):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 11053):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 11052):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 11051):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 11050):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 11049):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x000000000000000e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007fbdda277e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fbdda277e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 11048):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 11047):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 11046):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 11045):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 11044):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 11043):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 11042):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 11041):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 11040):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 11039):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 11038):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 11037):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 11036):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 11035):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 11034):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 11033):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 11032):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 11030):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 11029):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 11028):
#0 0x00007fbdf9970fb9 in ?? ()
#1 0x00007fbde4479c80 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001b360 in ?? ()
#5 0x00007fbde4479c70 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 11026):
#0 0x00007fbdf9970fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 11025):
#0 0x00007fbdf9970fb9 in ?? ()
#1 0x00007fbde54af1c0 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007fbde54af1b0 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 11024):
#0 0x00007fbdf9970ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 11023):
#0 0x00007fbdf26e1a47 in ?? ()
#1 0x00007fbde64b22e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007fbde64b22d0 in ?? ()
#4 0x00007fbde64b24e0 in ?? ()
#5 0x00007fbde64b2320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007fbde64b23a0 in ?? ()
#8 0x00007fbdf4d4125d in ?? ()
#9 0x3fb970da915c8000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000040000000 in ?? ()
#14 0x00007fbd00000000 in ?? ()
#15 0x41d9e21ced720495 in ?? ()
#16 0x00000ff83cc8e474 in ?? ()
#17 0x00007fbde64b2380 in ?? ()
#18 0x00007fbdf4d45ba3 in ?? ()
#19 0x00007fbde64b2350 in ?? ()
#20 0x3fb970da915c8000 in ?? ()
#21 0x00000000e64b23a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb970da915c8000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 11022):
#0 0x00007fbdf26e1a47 in ?? ()
#1 0x00007fbde6cb32e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007fbde6cb32d0 in ?? ()
#4 0x00007fbde6cb34e0 in ?? ()
#5 0x00007fbde6cb3320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007fbde6cb33a0 in ?? ()
#8 0x00007fbdf4d4125d in ?? ()
#9 0x3fb96e61722e0000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x0000000038000000 in ?? ()
#14 0x00007fbd00000000 in ?? ()
#15 0x41d9e21ced720498 in ?? ()
#16 0x00000ff83cd8e674 in ?? ()
#17 0x00007fbde6cb3380 in ?? ()
#18 0x00007fbdf4d45ba3 in ?? ()
#19 0x00007fbde6cb3350 in ?? ()
#20 0x3fb96e61722e0000 in ?? ()
#21 0x00000000e6cb33a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb96e61722e0000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 11021):
#0 0x00007fbdf26e1a47 in ?? ()
#1 0x00007fbde74b42e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007fbde74b42d0 in ?? ()
#4 0x00007fbde74b44e0 in ?? ()
#5 0x00007fbde74b4320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007fbde74b43a0 in ?? ()
#8 0x00007fbdf4d4125d in ?? ()
#9 0x3fb95f3b9c4fc000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000030000000 in ?? ()
#14 0x00007fbd00000000 in ?? ()
#15 0x41d9e21ced720496 in ?? ()
#16 0x00000ff83ce8e874 in ?? ()
#17 0x00007fbde74b4380 in ?? ()
#18 0x00007fbdf4d45ba3 in ?? ()
#19 0x00007fbde74b4350 in ?? ()
#20 0x3fb95f3b9c4fc000 in ?? ()
#21 0x00000000e74b43a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb95f3b9c4fc000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 11020):
#0 0x00007fbdf26e1a47 in ?? ()
#1 0x00007fbde94ad2e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007fbde94ad2d0 in ?? ()
#4 0x00007fbde94ad4e0 in ?? ()
#5 0x00007fbde94ad320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fbde94ad3a0 in ?? ()
#8 0x00007fbdf4d4125d in ?? ()
#9 0x3fb98631366ae000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x0000000066800000 in ?? ()
#14 0x00007fbd00000000 in ?? ()
#15 0x41d9e21ced720499 in ?? ()
#16 0x00000ff83d28da74 in ?? ()
#17 0x00007fbde94ad380 in ?? ()
#18 0x00007fbdf4d45ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 11017):
#0 0x00007fbdf26d4cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fbdeaeb7788 in ?? ()
#3 0x0000602000017cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 11016):
#0 0x00007fbdf9970fb9 in ?? ()
#1 0x00007fbdea6b6080 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007fbdea6b6070 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 11015):
#0 0x00007fbdf99749e2 in ?? ()
#1 0x00007fbde9eb3b60 in ?? ()
#2 0x00007fbde9eb3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fbde9eb3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007fbde9eb3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007fbde9eb3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007fbdff298f0c in ?? ()
#12 0x00007fbdff2887b9 in ?? ()
#13 0x00007fbdff28cda6 in ?? ()
#14 0x00007fbdff2917e0 in ?? ()
#15 0x00007fbdff291609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007fbdf658898e in ?? ()
#18 0x00007fbdf996a6db in ?? ()
#19 0x00000ff83d3ce67c in ?? ()
#20 0x00007fbde9eb3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 11009):
#0 0x00007fbdf9970fb9 in ?? ()
#1 0x00007fbdebeb8c40 in ?? ()
#2 0x00000000000002bf in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007fbdebeb8c30 in ?? ()
#6 0x000000000000057e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 11008):
#0 0x00007fbdf9970fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 11007):
#0 0x00007fbdf9970fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 11006):
#0 0x00007fbdf9970fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fbded6bc410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 11004):
#0 0x00007fbdf9974d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250116 02:58:09.869143 8081 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 8 with UUID 497640d49e7242aaa63256055e91414e and pid 11145
************************ BEGIN STACKS **************************
[New LWP 11149]
[New LWP 11150]
[New LWP 11151]
[New LWP 11152]
[New LWP 11158]
[New LWP 11159]
[New LWP 11160]
[New LWP 11163]
[New LWP 11164]
[New LWP 11165]
[New LWP 11166]
[New LWP 11167]
[New LWP 11168]
[New LWP 11170]
[New LWP 11171]
[New LWP 11173]
[New LWP 11174]
[New LWP 11175]
[New LWP 11176]
[New LWP 11177]
[New LWP 11178]
[New LWP 11179]
[New LWP 11180]
[New LWP 11182]
[New LWP 11183]
[New LWP 11184]
[New LWP 11185]
[New LWP 11186]
[New LWP 11187]
[New LWP 11188]
[New LWP 11189]
[New LWP 11190]
[New LWP 11191]
[New LWP 11192]
[New LWP 11193]
[New LWP 11194]
[New LWP 11195]
[New LWP 11197]
[New LWP 11198]
[New LWP 11199]
[New LWP 11200]
[New LWP 11201]
[New LWP 11202]
[New LWP 11203]
[New LWP 11204]
[New LWP 11205]
[New LWP 11206]
[New LWP 11207]
[New LWP 11208]
[New LWP 11209]
[New LWP 11210]
[New LWP 11211]
[New LWP 11212]
[New LWP 11213]
[New LWP 11214]
[New LWP 11215]
[New LWP 11216]
[New LWP 11217]
[New LWP 11218]
[New LWP 11219]
[New LWP 11220]
[New LWP 11221]
[New LWP 11222]
[New LWP 11223]
[New LWP 11224]
[New LWP 11225]
[New LWP 11226]
[New LWP 11227]
[New LWP 11228]
[New LWP 11229]
[New LWP 11230]
[New LWP 11231]
[New LWP 11232]
[New LWP 11233]
[New LWP 11234]
[New LWP 11235]
[New LWP 11236]
[New LWP 11237]
[New LWP 11238]
[New LWP 11239]
[New LWP 11240]
[New LWP 11241]
[New LWP 11242]
[New LWP 11243]
[New LWP 11244]
[New LWP 11245]
[New LWP 11246]
[New LWP 11247]
[New LWP 11248]
[New LWP 11249]
[New LWP 11250]
[New LWP 11251]
[New LWP 11252]
[New LWP 11253]
[New LWP 11254]
[New LWP 11255]
[New LWP 11256]
[New LWP 11257]
[New LWP 11258]
[New LWP 11259]
[New LWP 11260]
[New LWP 11261]
[New LWP 11262]
[New LWP 11263]
[New LWP 11264]
[New LWP 11265]
[New LWP 11266]
[New LWP 11267]
[New LWP 11268]
[New LWP 11269]
[New LWP 11270]
[New LWP 11271]
[New LWP 11272]
[New LWP 11273]
[New LWP 11274]
[New LWP 11275]
[New LWP 11276]
[New LWP 11277]
[New LWP 11278]
[New LWP 11279]
[New LWP 11281]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007fb116dcad50 in ?? ()
Id Target Id Frame
* 1 LWP 11145 "kudu" 0x00007fb116dcad50 in ?? ()
2 LWP 11149 "kudu" 0x00007fb116dc6fb9 in ?? ()
3 LWP 11150 "kudu" 0x00007fb116dc6fb9 in ?? ()
4 LWP 11151 "kudu" 0x00007fb116dc6fb9 in ?? ()
5 LWP 11152 "kernel-watcher-" 0x00007fb116dc6fb9 in ?? ()
6 LWP 11158 "ntp client-1115" 0x00007fb116dca9e2 in ?? ()
7 LWP 11159 "file cache-evic" 0x00007fb116dc6fb9 in ?? ()
8 LWP 11160 "sq_acceptor" 0x00007fb10fb2acb9 in ?? ()
9 LWP 11163 "rpc reactor-111" 0x00007fb10fb37a47 in ?? ()
10 LWP 11164 "rpc reactor-111" 0x00007fb10fb37a47 in ?? ()
11 LWP 11165 "rpc reactor-111" 0x00007fb10fb37a47 in ?? ()
12 LWP 11166 "rpc reactor-111" 0x00007fb10fb37a47 in ?? ()
13 LWP 11167 "MaintenanceMgr " 0x00007fb116dc6ad3 in ?? ()
14 LWP 11168 "txn-status-mana" 0x00007fb116dc6fb9 in ?? ()
15 LWP 11170 "collect_and_rem" 0x00007fb116dc6fb9 in ?? ()
16 LWP 11171 "tc-session-exp-" 0x00007fb116dc6fb9 in ?? ()
17 LWP 11173 "rpc worker-1117" 0x00007fb116dc6ad3 in ?? ()
18 LWP 11174 "rpc worker-1117" 0x00007fb116dc6ad3 in ?? ()
19 LWP 11175 "rpc worker-1117" 0x00007fb116dc6ad3 in ?? ()
20 LWP 11176 "rpc worker-1117" 0x00007fb116dc6ad3 in ?? ()
21 LWP 11177 "rpc worker-1117" 0x00007fb116dc6ad3 in ?? ()
22 LWP 11178 "rpc worker-1117" 0x00007fb116dc6ad3 in ?? ()
23 LWP 11179 "rpc worker-1117" 0x00007fb116dc6ad3 in ?? ()
24 LWP 11180 "rpc worker-1118" 0x00007fb116dc6ad3 in ?? ()
25 LWP 11182 "rpc worker-1118" 0x00007fb116dc6ad3 in ?? ()
26 LWP 11183 "rpc worker-1118" 0x00007fb116dc6ad3 in ?? ()
27 LWP 11184 "rpc worker-1118" 0x00007fb116dc6ad3 in ?? ()
28 LWP 11185 "rpc worker-1118" 0x00007fb116dc6ad3 in ?? ()
29 LWP 11186 "rpc worker-1118" 0x00007fb116dc6ad3 in ?? ()
30 LWP 11187 "rpc worker-1118" 0x00007fb116dc6ad3 in ?? ()
31 LWP 11188 "rpc worker-1118" 0x00007fb116dc6ad3 in ?? ()
32 LWP 11189 "rpc worker-1118" 0x00007fb116dc6ad3 in ?? ()
33 LWP 11190 "rpc worker-1119" 0x00007fb116dc6ad3 in ?? ()
34 LWP 11191 "rpc worker-1119" 0x00007fb116dc6ad3 in ?? ()
35 LWP 11192 "rpc worker-1119" 0x00007fb116dc6ad3 in ?? ()
36 LWP 11193 "rpc worker-1119" 0x00007fb116dc6ad3 in ?? ()
37 LWP 11194 "rpc worker-1119" 0x00007fb116dc6ad3 in ?? ()
38 LWP 11195 "rpc worker-1119" 0x00007fb116dc6ad3 in ?? ()
39 LWP 11197 "rpc worker-1119" 0x00007fb116dc6ad3 in ?? ()
40 LWP 11198 "rpc worker-1119" 0x00007fb116dc6ad3 in ?? ()
41 LWP 11199 "rpc worker-1119" 0x00007fb116dc6ad3 in ?? ()
42 LWP 11200 "rpc worker-1120" 0x00007fb116dc6ad3 in ?? ()
43 LWP 11201 "rpc worker-1120" 0x00007fb116dc6ad3 in ?? ()
44 LWP 11202 "rpc worker-1120" 0x00007fb116dc6ad3 in ?? ()
45 LWP 11203 "rpc worker-1120" 0x00007fb116dc6ad3 in ?? ()
46 LWP 11204 "rpc worker-1120" 0x00007fb116dc6ad3 in ?? ()
47 LWP 11205 "rpc worker-1120" 0x00007fb116dc6ad3 in ?? ()
48 LWP 11206 "rpc worker-1120" 0x00007fb116dc6ad3 in ?? ()
49 LWP 11207 "rpc worker-1120" 0x00007fb116dc6ad3 in ?? ()
50 LWP 11208 "rpc worker-1120" 0x00007fb116dc6ad3 in ?? ()
51 LWP 11209 "rpc worker-1120" 0x00007fb116dc6ad3 in ?? ()
52 LWP 11210 "rpc worker-1121" 0x00007fb116dc6ad3 in ?? ()
53 LWP 11211 "rpc worker-1121" 0x00007fb116dc6ad3 in ?? ()
54 LWP 11212 "rpc worker-1121" 0x00007fb116dc6ad3 in ?? ()
55 LWP 11213 "rpc worker-1121" 0x00007fb116dc6ad3 in ?? ()
56 LWP 11214 "rpc worker-1121" 0x00007fb116dc6ad3 in ?? ()
57 LWP 11215 "rpc worker-1121" 0x00007fb116dc6ad3 in ?? ()
58 LWP 11216 "rpc worker-1121" 0x00007fb116dc6ad3 in ?? ()
59 LWP 11217 "rpc worker-1121" 0x00007fb116dc6ad3 in ?? ()
60 LWP 11218 "rpc worker-1121" 0x00007fb116dc6ad3 in ?? ()
61 LWP 11219 "rpc worker-1121" 0x00007fb116dc6ad3 in ?? ()
62 LWP 11220 "rpc worker-1122" 0x00007fb116dc6ad3 in ?? ()
63 LWP 11221 "rpc worker-1122" 0x00007fb116dc6ad3 in ?? ()
64 LWP 11222 "rpc worker-1122" 0x00007fb116dc6ad3 in ?? ()
65 LWP 11223 "rpc worker-1122" 0x00007fb116dc6ad3 in ?? ()
66 LWP 11224 "rpc worker-1122" 0x00007fb116dc6ad3 in ?? ()
67 LWP 11225 "rpc worker-1122" 0x00007fb116dc6ad3 in ?? ()
68 LWP 11226 "rpc worker-1122" 0x00007fb116dc6ad3 in ?? ()
69 LWP 11227 "rpc worker-1122" 0x00007fb116dc6ad3 in ?? ()
70 LWP 11228 "rpc worker-1122" 0x00007fb116dc6ad3 in ?? ()
71 LWP 11229 "rpc worker-1122" 0x00007fb116dc6ad3 in ?? ()
72 LWP 11230 "rpc worker-1123" 0x00007fb116dc6ad3 in ?? ()
73 LWP 11231 "rpc worker-1123" 0x00007fb116dc6ad3 in ?? ()
74 LWP 11232 "rpc worker-1123" 0x00007fb116dc6ad3 in ?? ()
75 LWP 11233 "rpc worker-1123" 0x00007fb116dc6ad3 in ?? ()
76 LWP 11234 "rpc worker-1123" 0x00007fb116dc6ad3 in ?? ()
77 LWP 11235 "rpc worker-1123" 0x00007fb116dc6ad3 in ?? ()
78 LWP 11236 "rpc worker-1123" 0x00007fb116dc6ad3 in ?? ()
79 LWP 11237 "rpc worker-1123" 0x00007fb116dc6ad3 in ?? ()
80 LWP 11238 "rpc worker-1123" 0x00007fb116dc6ad3 in ?? ()
81 LWP 11239 "rpc worker-1123" 0x00007fb116dc6ad3 in ?? ()
82 LWP 11240 "rpc worker-1124" 0x00007fb116dc6ad3 in ?? ()
83 LWP 11241 "rpc worker-1124" 0x00007fb116dc6ad3 in ?? ()
84 LWP 11242 "rpc worker-1124" 0x00007fb116dc6ad3 in ?? ()
85 LWP 11243 "rpc worker-1124" 0x00007fb116dc6ad3 in ?? ()
86 LWP 11244 "rpc worker-1124" 0x00007fb116dc6ad3 in ?? ()
87 LWP 11245 "rpc worker-1124" 0x00007fb116dc6ad3 in ?? ()
88 LWP 11246 "rpc worker-1124" 0x00007fb116dc6ad3 in ?? ()
89 LWP 11247 "rpc worker-1124" 0x00007fb116dc6ad3 in ?? ()
90 LWP 11248 "rpc worker-1124" 0x00007fb116dc6ad3 in ?? ()
91 LWP 11249 "rpc worker-1124" 0x00007fb116dc6ad3 in ?? ()
92 LWP 11250 "rpc worker-1125" 0x00007fb116dc6ad3 in ?? ()
93 LWP 11251 "rpc worker-1125" 0x00007fb116dc6ad3 in ?? ()
94 LWP 11252 "rpc worker-1125" 0x00007fb116dc6ad3 in ?? ()
95 LWP 11253 "rpc worker-1125" 0x00007fb116dc6ad3 in ?? ()
96 LWP 11254 "rpc worker-1125" 0x00007fb116dc6ad3 in ?? ()
97 LWP 11255 "rpc worker-1125" 0x00007fb116dc6ad3 in ?? ()
98 LWP 11256 "rpc worker-1125" 0x00007fb116dc6ad3 in ?? ()
99 LWP 11257 "rpc worker-1125" 0x00007fb116dc6ad3 in ?? ()
100 LWP 11258 "rpc worker-1125" 0x00007fb116dc6ad3 in ?? ()
101 LWP 11259 "rpc worker-1125" 0x00007fb116dc6ad3 in ?? ()
102 LWP 11260 "rpc worker-1126" 0x00007fb116dc6ad3 in ?? ()
103 LWP 11261 "rpc worker-1126" 0x00007fb116dc6ad3 in ?? ()
104 LWP 11262 "rpc worker-1126" 0x00007fb116dc6ad3 in ?? ()
105 LWP 11263 "rpc worker-1126" 0x00007fb116dc6ad3 in ?? ()
106 LWP 11264 "rpc worker-1126" 0x00007fb116dc6ad3 in ?? ()
107 LWP 11265 "rpc worker-1126" 0x00007fb116dc6ad3 in ?? ()
108 LWP 11266 "rpc worker-1126" 0x00007fb116dc6ad3 in ?? ()
109 LWP 11267 "rpc worker-1126" 0x00007fb116dc6ad3 in ?? ()
110 LWP 11268 "rpc worker-1126" 0x00007fb116dc6ad3 in ?? ()
111 LWP 11269 "rpc worker-1126" 0x00007fb116dc6ad3 in ?? ()
112 LWP 11270 "rpc worker-1127" 0x00007fb116dc6ad3 in ?? ()
113 LWP 11271 "rpc worker-1127" 0x00007fb116dc6ad3 in ?? ()
114 LWP 11272 "rpc worker-1127" 0x00007fb116dc6ad3 in ?? ()
115 LWP 11273 "rpc worker-1127" 0x00007fb116dc6ad3 in ?? ()
116 LWP 11274 "rpc worker-1127" 0x00007fb116dc6ad3 in ?? ()
117 LWP 11275 "diag-logger-112" 0x00007fb116dc6fb9 in ?? ()
118 LWP 11276 "result-tracker-" 0x00007fb116dc6fb9 in ?? ()
119 LWP 11277 "excess-log-dele" 0x00007fb116dc6fb9 in ?? ()
120 LWP 11278 "acceptor-11278" 0x00007fb10fb390c7 in ?? ()
121 LWP 11279 "heartbeat-11279" 0x00007fb116dc6fb9 in ?? ()
122 LWP 11281 "maintenance_sch" 0x00007fb116dc6fb9 in ?? ()
Thread 122 (LWP 11281):
#0 0x00007fb116dc6fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000226 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007fb0cb63f550 in ?? ()
#6 0x000000000000044c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 11279):
#0 0x00007fb116dc6fb9 in ?? ()
#1 0x81afaaa989cdd16b in ?? ()
#2 0x000000000000008a in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x000061300001aac4 in ?? ()
#5 0x00007fb0cc66f5b0 in ?? ()
#6 0x0000000000000115 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fb0cc66f5d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007fb0cc66f650 in ?? ()
#12 0x00000ff6198cdecc in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 120 (LWP 11278):
#0 0x00007fb10fb390c7 in ?? ()
#1 0x0000614000021ca8 in ?? ()
#2 0x00007fb0cce96d50 in ?? ()
#3 0x00007fb0cce96d80 in ?? ()
#4 0x00007fb0cce96e80 in ?? ()
#5 0x00007fb0cce96d70 in ?? ()
#6 0x00007fb0cce96de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007fb1139bdef0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 11277):
#0 0x00007fb116dc6fb9 in ?? ()
#1 0x00007fb0cd6aff00 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffd9b511b30 in ?? ()
#5 0x00007fb0cd6afef0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 11276):
#0 0x00007fb116dc6fb9 in ?? ()
#1 0x00007fb0cdec90c0 in ?? ()
#2 0x000000000000008a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007fb0cdec90b0 in ?? ()
#6 0x0000000000000114 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 11275):
#0 0x00007fb116dc6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 11274):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000027 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002d682c in ?? ()
#4 0x00007fb0ceef8e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb0ceef8e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002d67e0 in ?? ()
#9 0x00007fb116dc6770 in ?? ()
#10 0x00007fb0ceef8e70 in ?? ()
#11 0x00000ff619ddf1d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 11273):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002d002c in ?? ()
#4 0x00007fb0cf710e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb0cf710e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002cffe0 in ?? ()
#9 0x00007fb116dc6770 in ?? ()
#10 0x00007fb0cf710e70 in ?? ()
#11 0x00000ff619ee21d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 11272):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 11271):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 11270):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 11269):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 11268):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 11267):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 11266):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 11265):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 11264):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 11263):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 11262):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 11261):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 11260):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 11259):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 11258):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 11257):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 11256):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 11255):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 11254):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 11253):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 11252):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 11251):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 11250):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 11249):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 11248):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 11247):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 11246):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 11245):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 11244):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 11243):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 11242):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 11241):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 11240):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 11239):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 11238):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 11237):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 11236):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 11235):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 11234):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000001285 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001cd06c in ?? ()
#4 0x00007fb0e32c9e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb0e32c9e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001cd020 in ?? ()
#9 0x00007fb116dc6770 in ?? ()
#10 0x00007fb0e32c9e70 in ?? ()
#11 0x00000ff61c6593d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 11233):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000001312 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001c6868 in ?? ()
#4 0x00007fb0e3ae1e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb0e3ae1e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 11232):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000160 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001c0068 in ?? ()
#4 0x00007fb0e42f9e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb0e42f9e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 73 (LWP 11231):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 11230):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 11229):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 11228):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 11227):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 11226):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 11225):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 11224):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 11223):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 11222):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 11221):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 11220):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 11219):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 11218):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 11217):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 11216):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 11215):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 11214):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x000000000000000e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000140018 in ?? ()
#4 0x00007fb0ecc92e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb0ecc92e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 11213):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 11212):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 11211):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 11210):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 11209):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 11208):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 11207):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 11206):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 11205):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 11204):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 11203):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 11202):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 11201):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 11200):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 11199):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 11198):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 11197):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 11195):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 11194):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 11193):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000191 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000acfec in ?? ()
#4 0x00007fb0f6e73e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb0f6e73e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000acfa0 in ?? ()
#9 0x00007fb116dc6770 in ?? ()
#10 0x00007fb0f6e73e70 in ?? ()
#11 0x00000ff61edce7d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 11192):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x000000000000034d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007fb0f768be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb0f768be70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007fb116dc6770 in ?? ()
#10 0x00007fb0f768be70 in ?? ()
#11 0x00000ff61eed17d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 34 (LWP 11191):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x000000000000034c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00009ffe8 in ?? ()
#4 0x00007fb0f7ea3e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb0f7ea3e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 33 (LWP 11190):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000169 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000967fc in ?? ()
#4 0x00007fb0f86bbe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb0f86bbe70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000967b0 in ?? ()
#9 0x00007fb116dc6770 in ?? ()
#10 0x00007fb0f86bbe70 in ?? ()
#11 0x00000ff61f0d77d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 32 (LWP 11189):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000105 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00008fffc in ?? ()
#4 0x00007fb0f8ed3e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb0f8ed3e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00008ffb0 in ?? ()
#9 0x00007fb116dc6770 in ?? ()
#10 0x00007fb0f8ed3e70 in ?? ()
#11 0x00000ff61f1da7d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 31 (LWP 11188):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 11187):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 11186):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 11185):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 11184):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 11183):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 11182):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x00000000000003b0 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000066828 in ?? ()
#4 0x00007fb0fc77be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb0fc77be70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 24 (LWP 11180):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 11179):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 11178):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 11177):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 11176):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 11175):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 11174):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 11173):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 11171):
#0 0x00007fb116dc6fb9 in ?? ()
#1 0x00007fb101075c80 in ?? ()
#2 0x000000000000000d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ba60 in ?? ()
#5 0x00007fb101075c70 in ?? ()
#6 0x000000000000001a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 11170):
#0 0x00007fb116dc6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 11168):
#0 0x00007fb116dc6fb9 in ?? ()
#1 0x00007fb10290a1c0 in ?? ()
#2 0x000000000000000d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007fb10290a1b0 in ?? ()
#6 0x000000000000001a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 11167):
#0 0x00007fb116dc6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 11166):
#0 0x00007fb10fb37a47 in ?? ()
#1 0x00007fb1039592e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007fb1039592d0 in ?? ()
#4 0x00007fb1039594e0 in ?? ()
#5 0x00007fb103959320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007fb1039593a0 in ?? ()
#8 0x00007fb11219725d in ?? ()
#9 0x3fb9876103af0000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x00000000dc000000 in ?? ()
#14 0x00007fb100000000 in ?? ()
#15 0x41d9e21ced720498 in ?? ()
#16 0x00000ff6a0723274 in ?? ()
#17 0x00007fb103959380 in ?? ()
#18 0x00007fb11219bba3 in ?? ()
#19 0x00007fb103959350 in ?? ()
#20 0x3fb9876103af0000 in ?? ()
#21 0x00000000039593a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb9876103af0000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 11165):
#0 0x00007fb10fb37a47 in ?? ()
#1 0x00007fb1041702e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007fb1041702d0 in ?? ()
#4 0x00007fb1041704e0 in ?? ()
#5 0x00007fb104170320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007fb1041703a0 in ?? ()
#8 0x00007fb11219725d in ?? ()
#9 0x3fb9864ecebd0000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x0000000006000000 in ?? ()
#14 0x00007fb100000000 in ?? ()
#15 0x41d9e21ced720495 in ?? ()
#16 0x00000ff6a0826074 in ?? ()
#17 0x00007fb104170380 in ?? ()
#18 0x00007fb11219bba3 in ?? ()
#19 0x00007fb104170350 in ?? ()
#20 0x3fb9864ecebd0000 in ?? ()
#21 0x00000000041703a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb9864ecebd0000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 11164):
#0 0x00007fb10fb37a47 in ?? ()
#1 0x00007fb1049872e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007fb1049872d0 in ?? ()
#4 0x00007fb1049874e0 in ?? ()
#5 0x00007fb104987320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007fb1049873a0 in ?? ()
#8 0x00007fb11219725d in ?? ()
#9 0x3fb96831756e6000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x000000008e000000 in ?? ()
#14 0x00007fb100000000 in ?? ()
#15 0x41d9e21ced720499 in ?? ()
#16 0x00000ff6a0928e74 in ?? ()
#17 0x00007fb104987380 in ?? ()
#18 0x00007fb11219bba3 in ?? ()
#19 0x00007fb104987350 in ?? ()
#20 0x3fb96831756e6000 in ?? ()
#21 0x00000000049873a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb96831756e6000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 11163):
#0 0x00007fb10fb37a47 in ?? ()
#1 0x00007fb1069ad2e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007fb1069ad2d0 in ?? ()
#4 0x00007fb1069ad4e0 in ?? ()
#5 0x00007fb1069ad320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fb1069ad3a0 in ?? ()
#8 0x00007fb11219725d in ?? ()
#9 0x3fb98d29a3234000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x000000002e000000 in ?? ()
#14 0x00007fb100000000 in ?? ()
#15 0x41d9e21ced720497 in ?? ()
#16 0x00000ff6a0d2da74 in ?? ()
#17 0x00007fb1069ad380 in ?? ()
#18 0x00007fb11219bba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 11160):
#0 0x00007fb10fb2acb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fb1083b7788 in ?? ()
#3 0x0000602000015cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 11159):
#0 0x00007fb116dc6fb9 in ?? ()
#1 0x00007fb107bb6080 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007fb107bb6070 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 11158):
#0 0x00007fb116dca9e2 in ?? ()
#1 0x00007fb1073b3b60 in ?? ()
#2 0x00007fb1073b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fb1073b3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007fb1073b3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007fb1073b3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007fb11c6eef0c in ?? ()
#12 0x00007fb11c6de7b9 in ?? ()
#13 0x00007fb11c6e2da6 in ?? ()
#14 0x00007fb11c6e77e0 in ?? ()
#15 0x00007fb11c6e7609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007fb1139de98e in ?? ()
#18 0x00007fb116dc06db in ?? ()
#19 0x00000ff6a0e6e67c in ?? ()
#20 0x00007fb1073b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 11152):
#0 0x00007fb116dc6fb9 in ?? ()
#1 0x00007fb1093b8c40 in ?? ()
#2 0x00000000000002b5 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007fb1093b8c30 in ?? ()
#6 0x000000000000056a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 11151):
#0 0x00007fb116dc6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 11150):
#0 0x00007fb116dc6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 11149):
#0 0x00007fb116dc6fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fb10abbc410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 11145):
#0 0x00007fb116dcad50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250116 02:58:10.641213 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 10185
I20250116 02:58:10.648587 10309 generic_service.cc:196] Checking for leaks (request via RPC)
I20250116 02:58:10.776335 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 10185
W20250116 02:58:10.801461 8106 connection.cc:537] server connection from 127.7.228.65:36283 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250116 02:58:10.801795 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 10327
I20250116 02:58:10.808528 10455 generic_service.cc:196] Checking for leaks (request via RPC)
I20250116 02:58:11.026964 12546 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 429bf4133bf54797b117ef8fad632920)
I20250116 02:58:11.027173 12546 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:58:11.027841 12546 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 497640d49e7242aaa63256055e91414e (127.7.228.73:42313), 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805)
I20250116 02:58:11.028431 11233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "0791364971ba49f3b2813bdc36c11eda" candidate_term: 4 candidate_status { last_received { term: 3 index: 17582 } } ignore_live_leader: false dest_uuid: "497640d49e7242aaa63256055e91414e" is_pre_election: true
I20250116 02:58:11.095844 12547 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 429bf4133bf54797b117ef8fad632920)
I20250116 02:58:11.096048 12547 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:58:11.096845 12547 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805), 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:58:11.115366 11917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 4 candidate_status { last_received { term: 3 index: 17582 } } ignore_live_leader: false dest_uuid: "0791364971ba49f3b2813bdc36c11eda" is_pre_election: true
I20250116 02:58:11.115702 11917 raft_consensus.cc:2463] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 497640d49e7242aaa63256055e91414e in term 3.
I20250116 02:58:11.116171 11166 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0791364971ba49f3b2813bdc36c11eda, 497640d49e7242aaa63256055e91414e; no voters:
I20250116 02:58:11.116456 12547 raft_consensus.cc:2798] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250116 02:58:11.116549 12547 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 429bf4133bf54797b117ef8fad632920)
I20250116 02:58:11.116655 12547 raft_consensus.cc:3054] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:58:11.118806 12547 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:58:11.119442 12547 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 4 election: Requested vote from peers 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805), 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593)
I20250116 02:58:11.119922 11917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "497640d49e7242aaa63256055e91414e" candidate_term: 4 candidate_status { last_received { term: 3 index: 17582 } } ignore_live_leader: false dest_uuid: "0791364971ba49f3b2813bdc36c11eda"
I20250116 02:58:11.120137 11917 raft_consensus.cc:3054] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 3 FOLLOWER]: Advancing to term 4
I20250116 02:58:11.125088 11917 raft_consensus.cc:2463] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 497640d49e7242aaa63256055e91414e in term 4.
I20250116 02:58:11.125614 11166 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0791364971ba49f3b2813bdc36c11eda, 497640d49e7242aaa63256055e91414e; no voters:
I20250116 02:58:11.125903 12547 raft_consensus.cc:2798] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 4 FOLLOWER]: Leader election won for term 4
I20250116 02:58:11.126348 12547 raft_consensus.cc:695] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 4 LEADER]: Becoming Leader. State: Replica: 497640d49e7242aaa63256055e91414e, State: Running, Role: LEADER
I20250116 02:58:11.126829 12547 consensus_queue.cc:237] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17582, Committed index: 17582, Last appended: 3.17582, Last appended by leader: 17582, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:58:11.130187 8119 catalog_manager.cc:5526] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e reported cstate change: term changed from 3 to 4, leader changed from 429bf4133bf54797b117ef8fad632920 (127.7.228.66) to 497640d49e7242aaa63256055e91414e (127.7.228.73). New cstate: current_term: 4 leader_uuid: "497640d49e7242aaa63256055e91414e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } health_report { overall_health: UNKNOWN } } }
I20250116 02:58:11.182897 10414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "0791364971ba49f3b2813bdc36c11eda" candidate_term: 4 candidate_status { last_received { term: 3 index: 17582 } } ignore_live_leader: false dest_uuid: "429bf4133bf54797b117ef8fad632920" is_pre_election: true
I20250116 02:58:11.183463 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 10327
I20250116 02:58:11.183741 11843 leader_election.cc:304] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0791364971ba49f3b2813bdc36c11eda; no voters: 429bf4133bf54797b117ef8fad632920, 497640d49e7242aaa63256055e91414e
I20250116 02:58:11.184172 11233 raft_consensus.cc:1235] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [term 4 LEADER]: Rejecting Update request from peer 429bf4133bf54797b117ef8fad632920 for earlier term 3. Current term is 4. Ops: []
I20250116 02:58:11.185110 12546 raft_consensus.cc:2743] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250116 02:58:11.235147 11917 raft_consensus.cc:1270] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 4 FOLLOWER]: Refusing update from remote peer 497640d49e7242aaa63256055e91414e: Log matching property violated. Preceding OpId in replica: term: 3 index: 17582. Preceding OpId from leader: term: 4 index: 17583. (index mismatch)
I20250116 02:58:11.235898 12547 consensus_queue.cc:1035] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17583, Last known committed idx: 17582, Time since last communication: 0.000s
W20250116 02:58:11.251916 11166 connection.cc:537] server connection from 127.7.228.66:57799 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250116 02:58:11.252754 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 10469
W20250116 02:58:11.252985 12549 negotiation.cc:337] Failed RPC negotiation. Trace:
0116 02:58:11.097117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.7.228.66:41805 (local address 127.7.228.73:47089)
0116 02:58:11.108926 (+ 11809us) negotiation.cc:107] Waiting for socket to connect
0116 02:58:11.108949 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0116 02:58:11.109103 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0116 02:58:11.184232 (+ 75129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0116 02:58:11.184241 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0116 02:58:11.184280 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0116 02:58:11.184873 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0116 02:58:11.184890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0116 02:58:11.252045 (+ 67155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.7.228.66:41805: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":11707,"thread_start_us":12168,"threads_started":1}
W20250116 02:58:11.253296 11163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.228.66:41805: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250116 02:58:11.254819 11163 leader_election.cc:336] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Network error: Client connection negotiation failed: client connection to 127.7.228.66:41805: connect: Connection refused (error 111)
W20250116 02:58:11.255092 11163 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.66:41805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250116 02:58:11.255183 11163 leader_election.cc:336] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Network error: Client connection negotiation failed: client connection to 127.7.228.66:41805: connect: Connection refused (error 111)
I20250116 02:58:11.259975 10596 generic_service.cc:196] Checking for leaks (request via RPC)
I20250116 02:58:11.404925 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 10469
I20250116 02:58:11.430900 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 11823
I20250116 02:58:11.436647 11956 generic_service.cc:196] Checking for leaks (request via RPC)
W20250116 02:58:11.688418 11163 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.66:41805: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250116 02:58:11.876492 12546 raft_consensus.cc:491] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 497640d49e7242aaa63256055e91414e)
I20250116 02:58:11.876682 12546 raft_consensus.cc:513] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "497640d49e7242aaa63256055e91414e" member_type: VOTER last_known_addr { host: "127.7.228.73" port: 42313 } } peers { permanent_uuid: "429bf4133bf54797b117ef8fad632920" member_type: VOTER last_known_addr { host: "127.7.228.66" port: 41805 } } peers { permanent_uuid: "0791364971ba49f3b2813bdc36c11eda" member_type: VOTER last_known_addr { host: "127.7.228.68" port: 41593 } }
I20250116 02:58:11.877424 12546 leader_election.cc:290] T 29c42185024a48cda0403d61962cf176 P 0791364971ba49f3b2813bdc36c11eda [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 497640d49e7242aaa63256055e91414e (127.7.228.73:42313), 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805)
I20250116 02:58:11.878329 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 11823
I20250116 02:58:11.880357 11233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c42185024a48cda0403d61962cf176" candidate_uuid: "0791364971ba49f3b2813bdc36c11eda" candidate_term: 5 candidate_status { last_received { term: 4 index: 17583 } } ignore_live_leader: false dest_uuid: "497640d49e7242aaa63256055e91414e" is_pre_election: true
W20250116 02:58:11.937956 11166 connection.cc:537] server connection from 127.7.228.68:53829 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250116 02:58:11.938380 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 9520
W20250116 02:58:11.941684 11166 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.68:41593: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250116 02:58:11.958177 9645 generic_service.cc:196] Checking for leaks (request via RPC)
I20250116 02:58:12.086584 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 9520
I20250116 02:58:12.112957 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 9659
I20250116 02:58:12.119297 9781 generic_service.cc:196] Checking for leaks (request via RPC)
W20250116 02:58:12.207294 11163 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e -> Peer 429bf4133bf54797b117ef8fad632920 (127.7.228.66:41805): Couldn't send request to peer 429bf4133bf54797b117ef8fad632920. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.66:41805: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250116 02:58:12.247356 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 9659
I20250116 02:58:12.272418 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 9795
I20250116 02:58:12.279124 9920 generic_service.cc:196] Checking for leaks (request via RPC)
I20250116 02:58:12.412078 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 9795
I20250116 02:58:12.437588 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 11004
I20250116 02:58:12.443845 11129 generic_service.cc:196] Checking for leaks (request via RPC)
W20250116 02:58:12.450683 11166 consensus_peers.cc:487] T 29c42185024a48cda0403d61962cf176 P 497640d49e7242aaa63256055e91414e -> Peer 0791364971ba49f3b2813bdc36c11eda (127.7.228.68:41593): Couldn't send request to peer 0791364971ba49f3b2813bdc36c11eda. Status: Network error: Client connection negotiation failed: client connection to 127.7.228.68:41593: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250116 02:58:12.568296 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 11004
I20250116 02:58:12.592445 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 11145
I20250116 02:58:12.603510 11273 generic_service.cc:196] Checking for leaks (request via RPC)
I20250116 02:58:13.090860 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 11145
I20250116 02:58:13.161479 8081 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task8SxoCU/build/asan/bin/kudu pid 8089
I20250116 02:58:13.168680 8150 generic_service.cc:196] Checking for leaks (request via RPC)
I20250116 02:58:13.360586 8081 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task8SxoCU/build/asan/bin/kudu with pid 8089
2025-01-16T02:58:13Z chronyd exiting
I20250116 02:58:13.415416 8081 test_util.cc:181] -----------------------------------------------
I20250116 02:58:13.415518 8081 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-task8SxoCU/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736996066753478-8081-0
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3, where GetParam() = 800-byte object <01-00 00-00 01-61 73-61 09-00 00-00 BE-BE BE-BE A0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-1F 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 03-00 00-00 03-00 00-00 00-BE BE-BE BE-BE BE-BE> (226601 ms)
[----------] 1 test from RollingRestartArgs/RollingRestartITest (226601 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (226601 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3, where GetParam() = 800-byte object <01-00 00-00 01-61 73-61 09-00 00-00 BE-BE BE-BE A0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-1F 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 03-00 00-00 03-00 00-00 00-BE BE-BE BE-BE BE-BE>
1 FAILED TEST