Note: This is test shard 8 of 8.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RollingRestartArgs/RollingRestartITest
[ RUN ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3
2025-06-22T19:55:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-22T19:55:04Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250622 19:55:04.683804 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.248.62:37973
--webserver_interface=127.1.248.62
--webserver_port=0
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.248.62:37973
--location_mapping_cmd=/tmp/dist-test-taskmkqhcg/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/location-assignment.state --map /L0:3 --map /L1:4 --map /L2:4
--master_client_location_assignment_enabled=false with env {}
W20250622 19:55:05.005379 2024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:05.005728 2024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:05.005808 2024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:05.013542 2024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250622 19:55:05.013646 2024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:05.013701 2024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250622 19:55:05.013756 2024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250622 19:55:05.022965 2024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/wal
--location_mapping_cmd=/tmp/dist-test-taskmkqhcg/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/location-assignment.state --map /L0:3 --map /L1:4 --map /L2:4
--ipki_ca_key_size=768
--master_addresses=127.1.248.62:37973
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.248.62:37973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.248.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:05.024593 2024 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:05.026369 2024 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:55:05.037263 2032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:05.037343 2030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:05.037312 2029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:05.037961 2024 server_base.cc:1048] running on GCE node
I20250622 19:55:05.038774 2024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:05.040242 2024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:05.041441 2024 hybrid_clock.cc:648] HybridClock initialized: now 1750622105041363 us; error 59 us; skew 500 ppm
I20250622 19:55:05.041766 2024 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:05.048593 2024 webserver.cc:469] Webserver started at http://127.1.248.62:38207/ using document root <none> and password file <none>
I20250622 19:55:05.049427 2024 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:05.049535 2024 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:05.049923 2024 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 19:55:05.052553 2024 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/data/instance:
uuid: "e103efdfebd54e69b47694a7953f484c"
format_stamp: "Formatted at 2025-06-22 19:55:05 on dist-test-slave-t73b"
I20250622 19:55:05.053365 2024 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/wal/instance:
uuid: "e103efdfebd54e69b47694a7953f484c"
format_stamp: "Formatted at 2025-06-22 19:55:05 on dist-test-slave-t73b"
I20250622 19:55:05.058904 2024 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20250622 19:55:05.062372 2038 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:05.063745 2024 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250622 19:55:05.063916 2024 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/wal
uuid: "e103efdfebd54e69b47694a7953f484c"
format_stamp: "Formatted at 2025-06-22 19:55:05 on dist-test-slave-t73b"
I20250622 19:55:05.064121 2024 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:05.088908 2024 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:05.089788 2024 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:05.090077 2024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:05.110013 2024 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.62:37973
I20250622 19:55:05.110047 2089 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.62:37973 every 8 connection(s)
I20250622 19:55:05.111609 2024 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/data/info.pb
I20250622 19:55:05.115561 2090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 19:55:05.121722 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 2024
I20250622 19:55:05.122076 2016 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/wal/instance
I20250622 19:55:05.127054 2090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c: Bootstrap starting.
I20250622 19:55:05.131093 2090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c: Neither blocks nor log segments found. Creating new log.
I20250622 19:55:05.132488 2090 log.cc:826] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c: Log is configured to *not* fsync() on all Append() calls
I20250622 19:55:05.135347 2090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c: No bootstrap required, opened a new log
I20250622 19:55:05.141399 2090 raft_consensus.cc:357] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e103efdfebd54e69b47694a7953f484c" member_type: VOTER last_known_addr { host: "127.1.248.62" port: 37973 } }
I20250622 19:55:05.141808 2090 raft_consensus.cc:383] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 19:55:05.141952 2090 raft_consensus.cc:738] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e103efdfebd54e69b47694a7953f484c, State: Initialized, Role: FOLLOWER
I20250622 19:55:05.142624 2090 consensus_queue.cc:260] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e103efdfebd54e69b47694a7953f484c" member_type: VOTER last_known_addr { host: "127.1.248.62" port: 37973 } }
I20250622 19:55:05.142865 2090 raft_consensus.cc:397] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250622 19:55:05.142983 2090 raft_consensus.cc:491] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250622 19:55:05.143155 2090 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [term 0 FOLLOWER]: Advancing to term 1
I20250622 19:55:05.145295 2090 raft_consensus.cc:513] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e103efdfebd54e69b47694a7953f484c" member_type: VOTER last_known_addr { host: "127.1.248.62" port: 37973 } }
I20250622 19:55:05.145850 2090 leader_election.cc:304] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e103efdfebd54e69b47694a7953f484c; no voters:
I20250622 19:55:05.146646 2090 leader_election.cc:290] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250622 19:55:05.146777 2095 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [term 1 FOLLOWER]: Leader election won for term 1
I20250622 19:55:05.148139 2095 raft_consensus.cc:695] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [term 1 LEADER]: Becoming Leader. State: Replica: e103efdfebd54e69b47694a7953f484c, State: Running, Role: LEADER
I20250622 19:55:05.148768 2095 consensus_queue.cc:237] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e103efdfebd54e69b47694a7953f484c" member_type: VOTER last_known_addr { host: "127.1.248.62" port: 37973 } }
I20250622 19:55:05.149168 2090 sys_catalog.cc:564] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [sys.catalog]: configured and running, proceeding with master startup.
I20250622 19:55:05.152591 2097 sys_catalog.cc:455] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e103efdfebd54e69b47694a7953f484c. Latest consensus state: current_term: 1 leader_uuid: "e103efdfebd54e69b47694a7953f484c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e103efdfebd54e69b47694a7953f484c" member_type: VOTER last_known_addr { host: "127.1.248.62" port: 37973 } } }
I20250622 19:55:05.152987 2097 sys_catalog.cc:458] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [sys.catalog]: This master's current role is: LEADER
I20250622 19:55:05.154735 2101 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250622 19:55:05.154611 2096 sys_catalog.cc:455] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e103efdfebd54e69b47694a7953f484c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e103efdfebd54e69b47694a7953f484c" member_type: VOTER last_known_addr { host: "127.1.248.62" port: 37973 } } }
I20250622 19:55:05.155071 2096 sys_catalog.cc:458] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c [sys.catalog]: This master's current role is: LEADER
I20250622 19:55:05.160751 2101 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250622 19:55:05.170377 2101 catalog_manager.cc:1349] Generated new cluster ID: 13835fbceb59449580ea04d2622468d1
I20250622 19:55:05.170516 2101 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250622 19:55:05.192986 2101 catalog_manager.cc:1372] Generated new certificate authority record
I20250622 19:55:05.194285 2101 catalog_manager.cc:1506] Loading token signing keys...
I20250622 19:55:05.204216 2101 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e103efdfebd54e69b47694a7953f484c: Generated new TSK 0
I20250622 19:55:05.205083 2101 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250622 19:55:05.219158 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.1:0
--local_ip_for_outbound_sockets=127.1.248.1
--webserver_interface=127.1.248.1
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250622 19:55:05.551373 2114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:05.551720 2114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:05.551821 2114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:05.560009 2114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:05.560199 2114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.1
I20250622 19:55:05.569886 2114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.248.1
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.1
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:05.571564 2114 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:05.573459 2114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:55:05.585754 2119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:05.585673 2120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:05.585729 2122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:05.586697 2114 server_base.cc:1048] running on GCE node
I20250622 19:55:05.587308 2114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:05.588315 2114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:05.589514 2114 hybrid_clock.cc:648] HybridClock initialized: now 1750622105589451 us; error 57 us; skew 500 ppm
I20250622 19:55:05.589824 2114 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:05.592168 2114 webserver.cc:469] Webserver started at http://127.1.248.1:41843/ using document root <none> and password file <none>
I20250622 19:55:05.592944 2114 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:05.593055 2114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:05.593402 2114 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 19:55:05.595917 2114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data/instance:
uuid: "2db5cde30e5a47fba8d434cc04849edc"
format_stamp: "Formatted at 2025-06-22 19:55:05 on dist-test-slave-t73b"
I20250622 19:55:05.596714 2114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/wal/instance:
uuid: "2db5cde30e5a47fba8d434cc04849edc"
format_stamp: "Formatted at 2025-06-22 19:55:05 on dist-test-slave-t73b"
I20250622 19:55:05.602104 2114 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250622 19:55:05.605398 2128 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:05.606838 2114 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 19:55:05.606987 2114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/wal
uuid: "2db5cde30e5a47fba8d434cc04849edc"
format_stamp: "Formatted at 2025-06-22 19:55:05 on dist-test-slave-t73b"
I20250622 19:55:05.607198 2114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:05.620518 2114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:05.621363 2114 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:05.621618 2114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:05.622571 2114 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:05.624151 2114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:55:05.624248 2114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:05.624356 2114 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:55:05.624428 2114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:05.656526 2114 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.1:36357
I20250622 19:55:05.656585 2240 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.1:36357 every 8 connection(s)
I20250622 19:55:05.657995 2114 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data/info.pb
I20250622 19:55:05.658864 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 2114
I20250622 19:55:05.659138 2016 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/wal/instance
I20250622 19:55:05.662767 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.2:0
--local_ip_for_outbound_sockets=127.1.248.2
--webserver_interface=127.1.248.2
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250622 19:55:05.671922 2241 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:05.672272 2241 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:05.673138 2241 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:05.721206 2055 ts_manager.cc:194] Registered new tserver with Master: 2db5cde30e5a47fba8d434cc04849edc (127.1.248.1:36357)
I20250622 19:55:05.723814 2055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.1:45335
W20250622 19:55:05.988005 2245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:05.988334 2245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:05.988431 2245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:05.997056 2245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:05.997238 2245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.2
I20250622 19:55:06.006454 2245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.248.2
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.2
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:06.008019 2245 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:06.009804 2245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:55:06.021831 2251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:06.021860 2252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:06.021972 2254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:06.023105 2245 server_base.cc:1048] running on GCE node
I20250622 19:55:06.023741 2245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:06.024688 2245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:06.025880 2245 hybrid_clock.cc:648] HybridClock initialized: now 1750622106025832 us; error 37 us; skew 500 ppm
I20250622 19:55:06.026182 2245 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:06.028556 2245 webserver.cc:469] Webserver started at http://127.1.248.2:46661/ using document root <none> and password file <none>
I20250622 19:55:06.029315 2245 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:06.029419 2245 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:06.029771 2245 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 19:55:06.032240 2245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data/instance:
uuid: "07bc6e053fe04f6d8b0c0157ca553029"
format_stamp: "Formatted at 2025-06-22 19:55:06 on dist-test-slave-t73b"
I20250622 19:55:06.033021 2245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/wal/instance:
uuid: "07bc6e053fe04f6d8b0c0157ca553029"
format_stamp: "Formatted at 2025-06-22 19:55:06 on dist-test-slave-t73b"
I20250622 19:55:06.038552 2245 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20250622 19:55:06.041813 2260 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:06.043258 2245 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250622 19:55:06.043414 2245 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/wal
uuid: "07bc6e053fe04f6d8b0c0157ca553029"
format_stamp: "Formatted at 2025-06-22 19:55:06 on dist-test-slave-t73b"
I20250622 19:55:06.043617 2245 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:06.067489 2245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:06.068397 2245 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:06.068645 2245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:06.069636 2245 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:06.071368 2245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:55:06.071475 2245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:06.071583 2245 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:55:06.071657 2245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:06.104262 2245 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.2:35999
I20250622 19:55:06.104331 2372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.2:35999 every 8 connection(s)
I20250622 19:55:06.105770 2245 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data/info.pb
I20250622 19:55:06.111963 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 2245
I20250622 19:55:06.112134 2016 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/wal/instance
I20250622 19:55:06.116684 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.3:0
--local_ip_for_outbound_sockets=127.1.248.3
--webserver_interface=127.1.248.3
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250622 19:55:06.120260 2373 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:06.120630 2373 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:06.121719 2373 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:06.179044 2055 ts_manager.cc:194] Registered new tserver with Master: 07bc6e053fe04f6d8b0c0157ca553029 (127.1.248.2:35999)
I20250622 19:55:06.180049 2055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.2:49817
W20250622 19:55:06.465317 2376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:06.465636 2376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:06.465761 2376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:06.479936 2376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:06.480144 2376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.3
I20250622 19:55:06.489553 2376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.248.3
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.3
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:06.491297 2376 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:06.493126 2376 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:55:06.505232 2383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:06.505271 2385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:06.505355 2382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:06.506222 2376 server_base.cc:1048] running on GCE node
I20250622 19:55:06.506873 2376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:06.507887 2376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:06.509120 2376 hybrid_clock.cc:648] HybridClock initialized: now 1750622106509060 us; error 49 us; skew 500 ppm
I20250622 19:55:06.509464 2376 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:06.511991 2376 webserver.cc:469] Webserver started at http://127.1.248.3:45773/ using document root <none> and password file <none>
I20250622 19:55:06.512826 2376 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:06.512926 2376 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:06.513299 2376 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 19:55:06.516948 2376 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data/instance:
uuid: "54555a8ecf924b339db8e8306cbc8bb4"
format_stamp: "Formatted at 2025-06-22 19:55:06 on dist-test-slave-t73b"
I20250622 19:55:06.519296 2376 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/wal/instance:
uuid: "54555a8ecf924b339db8e8306cbc8bb4"
format_stamp: "Formatted at 2025-06-22 19:55:06 on dist-test-slave-t73b"
I20250622 19:55:06.524986 2376 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250622 19:55:06.528318 2391 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:06.529742 2376 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250622 19:55:06.529892 2376 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/wal
uuid: "54555a8ecf924b339db8e8306cbc8bb4"
format_stamp: "Formatted at 2025-06-22 19:55:06 on dist-test-slave-t73b"
I20250622 19:55:06.530097 2376 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:06.544920 2376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:06.545799 2376 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:06.546058 2376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:06.547098 2376 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:06.548906 2376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:55:06.548996 2376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:06.549091 2376 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:55:06.549163 2376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:06.584491 2376 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.3:41117
I20250622 19:55:06.584537 2503 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.3:41117 every 8 connection(s)
I20250622 19:55:06.585927 2376 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data/info.pb
I20250622 19:55:06.588387 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 2376
I20250622 19:55:06.588577 2016 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/wal/instance
I20250622 19:55:06.591774 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.4:0
--local_ip_for_outbound_sockets=127.1.248.4
--webserver_interface=127.1.248.4
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250622 19:55:06.600503 2504 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:06.600929 2504 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:06.602003 2504 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:06.649284 2055 ts_manager.cc:194] Registered new tserver with Master: 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117)
I20250622 19:55:06.650611 2055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.3:59061
I20250622 19:55:06.727538 2241 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
W20250622 19:55:06.936013 2507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:06.936350 2507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:06.936475 2507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:06.944923 2507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:06.945152 2507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.4
I20250622 19:55:06.954859 2507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.1.248.4
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.4
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:06.956543 2507 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:06.958465 2507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:55:06.970665 2514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:06.970702 2516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:06.970744 2513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:06.971707 2507 server_base.cc:1048] running on GCE node
I20250622 19:55:06.972340 2507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:06.973337 2507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:06.974531 2507 hybrid_clock.cc:648] HybridClock initialized: now 1750622106974479 us; error 45 us; skew 500 ppm
I20250622 19:55:06.974849 2507 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:06.977106 2507 webserver.cc:469] Webserver started at http://127.1.248.4:33405/ using document root <none> and password file <none>
I20250622 19:55:06.977855 2507 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:06.977967 2507 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:06.978353 2507 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 19:55:06.981298 2507 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data/instance:
uuid: "144888b25afe4c04b4de5350bddf224e"
format_stamp: "Formatted at 2025-06-22 19:55:06 on dist-test-slave-t73b"
I20250622 19:55:06.982085 2507 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/wal/instance:
uuid: "144888b25afe4c04b4de5350bddf224e"
format_stamp: "Formatted at 2025-06-22 19:55:06 on dist-test-slave-t73b"
I20250622 19:55:06.987406 2507 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20250622 19:55:06.990684 2522 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:06.992036 2507 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250622 19:55:06.992179 2507 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/wal
uuid: "144888b25afe4c04b4de5350bddf224e"
format_stamp: "Formatted at 2025-06-22 19:55:06 on dist-test-slave-t73b"
I20250622 19:55:06.992374 2507 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:07.016111 2507 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:07.016985 2507 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:07.017241 2507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:07.018195 2507 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:07.019865 2507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:55:07.019948 2507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:07.020052 2507 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:55:07.020121 2507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:07.056042 2507 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.4:40193
I20250622 19:55:07.056100 2634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.4:40193 every 8 connection(s)
I20250622 19:55:07.057621 2507 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data/info.pb
I20250622 19:55:07.061532 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 2507
I20250622 19:55:07.061723 2016 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/wal/instance
I20250622 19:55:07.064750 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.5:0
--local_ip_for_outbound_sockets=127.1.248.5
--webserver_interface=127.1.248.5
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250622 19:55:07.072042 2635 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:07.072448 2635 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:07.073572 2635 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:07.134691 2055 ts_manager.cc:194] Registered new tserver with Master: 144888b25afe4c04b4de5350bddf224e (127.1.248.4:40193)
I20250622 19:55:07.135735 2055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.4:50929
I20250622 19:55:07.184090 2373 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
W20250622 19:55:07.419752 2638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:07.420153 2638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:07.420292 2638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:07.428683 2638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:07.428871 2638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.5
I20250622 19:55:07.438283 2638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data/info.pb
--webserver_interface=127.1.248.5
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.5
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:07.439900 2638 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:07.441669 2638 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:55:07.454077 2647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:07.453980 2644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:07.454005 2645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:07.454861 2638 server_base.cc:1048] running on GCE node
I20250622 19:55:07.455461 2638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:07.456374 2638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:07.457566 2638 hybrid_clock.cc:648] HybridClock initialized: now 1750622107457480 us; error 75 us; skew 500 ppm
I20250622 19:55:07.457864 2638 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:07.460247 2638 webserver.cc:469] Webserver started at http://127.1.248.5:34689/ using document root <none> and password file <none>
I20250622 19:55:07.460992 2638 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:07.461601 2638 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:07.461959 2638 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 19:55:07.464501 2638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data/instance:
uuid: "9ece0d51c71b47a1865ba3b05920d999"
format_stamp: "Formatted at 2025-06-22 19:55:07 on dist-test-slave-t73b"
I20250622 19:55:07.465303 2638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/wal/instance:
uuid: "9ece0d51c71b47a1865ba3b05920d999"
format_stamp: "Formatted at 2025-06-22 19:55:07 on dist-test-slave-t73b"
I20250622 19:55:07.470674 2638 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20250622 19:55:07.474434 2653 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:07.475788 2638 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250622 19:55:07.475952 2638 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/wal
uuid: "9ece0d51c71b47a1865ba3b05920d999"
format_stamp: "Formatted at 2025-06-22 19:55:07 on dist-test-slave-t73b"
I20250622 19:55:07.476163 2638 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:07.492225 2638 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:07.493124 2638 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:07.493371 2638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:07.494313 2638 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:07.495909 2638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:55:07.496006 2638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:07.496119 2638 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:55:07.496188 2638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:07.532550 2638 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.5:43933
I20250622 19:55:07.532600 2765 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.5:43933 every 8 connection(s)
I20250622 19:55:07.534019 2638 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data/info.pb
I20250622 19:55:07.544145 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 2638
I20250622 19:55:07.544358 2016 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/wal/instance
I20250622 19:55:07.547489 2766 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:07.547852 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.6:0
--local_ip_for_outbound_sockets=127.1.248.6
--webserver_interface=127.1.248.6
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250622 19:55:07.547912 2766 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:07.548985 2766 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:07.593442 2055 ts_manager.cc:194] Registered new tserver with Master: 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
I20250622 19:55:07.594810 2055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.5:54741
I20250622 19:55:07.653903 2504 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
W20250622 19:55:07.899058 2769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:07.899374 2769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:07.899523 2769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:07.908102 2769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:07.908319 2769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.6
I20250622 19:55:07.917971 2769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data/info.pb
--webserver_interface=127.1.248.6
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.6
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:07.920202 2769 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:07.922011 2769 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:55:07.933686 2778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:07.933753 2776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:07.933723 2775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:07.934504 2769 server_base.cc:1048] running on GCE node
I20250622 19:55:07.935091 2769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:07.936096 2769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:07.937284 2769 hybrid_clock.cc:648] HybridClock initialized: now 1750622107937232 us; error 43 us; skew 500 ppm
I20250622 19:55:07.937577 2769 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:07.939976 2769 webserver.cc:469] Webserver started at http://127.1.248.6:36755/ using document root <none> and password file <none>
I20250622 19:55:07.940711 2769 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:07.940806 2769 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:07.941167 2769 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 19:55:07.943740 2769 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data/instance:
uuid: "e63bffd596554411a2e0489c19e78fff"
format_stamp: "Formatted at 2025-06-22 19:55:07 on dist-test-slave-t73b"
I20250622 19:55:07.944504 2769 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/wal/instance:
uuid: "e63bffd596554411a2e0489c19e78fff"
format_stamp: "Formatted at 2025-06-22 19:55:07 on dist-test-slave-t73b"
I20250622 19:55:07.950104 2769 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250622 19:55:07.953397 2784 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:07.954809 2769 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 19:55:07.954988 2769 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/wal
uuid: "e63bffd596554411a2e0489c19e78fff"
format_stamp: "Formatted at 2025-06-22 19:55:07 on dist-test-slave-t73b"
I20250622 19:55:07.955188 2769 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:07.969740 2769 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:07.970705 2769 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:07.970947 2769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:07.971858 2769 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:07.973402 2769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:55:07.973517 2769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:07.973623 2769 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:55:07.973704 2769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:08.012605 2769 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.6:38357
I20250622 19:55:08.012612 2896 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.6:38357 every 8 connection(s)
I20250622 19:55:08.014021 2769 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data/info.pb
I20250622 19:55:08.017746 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 2769
I20250622 19:55:08.017948 2016 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/wal/instance
I20250622 19:55:08.021131 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.7:0
--local_ip_for_outbound_sockets=127.1.248.7
--webserver_interface=127.1.248.7
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250622 19:55:08.027858 2897 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:08.028301 2897 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:08.029439 2897 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:08.078030 2055 ts_manager.cc:194] Registered new tserver with Master: e63bffd596554411a2e0489c19e78fff (127.1.248.6:38357)
I20250622 19:55:08.079015 2055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.6:52505
I20250622 19:55:08.139132 2635 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
W20250622 19:55:08.366482 2900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:08.366807 2900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:08.366935 2900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:08.375240 2900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:08.375464 2900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.7
I20250622 19:55:08.384735 2900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data/info.pb
--webserver_interface=127.1.248.7
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.7
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:08.386497 2900 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:08.388880 2900 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:55:08.400465 2909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:08.400570 2906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:08.400570 2907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:08.401721 2900 server_base.cc:1048] running on GCE node
I20250622 19:55:08.402443 2900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:08.403465 2900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:08.404681 2900 hybrid_clock.cc:648] HybridClock initialized: now 1750622108404626 us; error 49 us; skew 500 ppm
I20250622 19:55:08.405005 2900 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:08.407310 2900 webserver.cc:469] Webserver started at http://127.1.248.7:39729/ using document root <none> and password file <none>
I20250622 19:55:08.408547 2900 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:08.408658 2900 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:08.409009 2900 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 19:55:08.411643 2900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data/instance:
uuid: "07ce178dce43415a852dbcce80f47f3e"
format_stamp: "Formatted at 2025-06-22 19:55:08 on dist-test-slave-t73b"
I20250622 19:55:08.412473 2900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/wal/instance:
uuid: "07ce178dce43415a852dbcce80f47f3e"
format_stamp: "Formatted at 2025-06-22 19:55:08 on dist-test-slave-t73b"
I20250622 19:55:08.418167 2900 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250622 19:55:08.421559 2915 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:08.423030 2900 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250622 19:55:08.423184 2900 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/wal
uuid: "07ce178dce43415a852dbcce80f47f3e"
format_stamp: "Formatted at 2025-06-22 19:55:08 on dist-test-slave-t73b"
I20250622 19:55:08.423386 2900 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:08.453928 2900 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:08.454855 2900 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:08.455080 2900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:08.455935 2900 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:08.457504 2900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:55:08.457588 2900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:08.457664 2900 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:55:08.457720 2900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:08.493485 2900 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.7:35201
I20250622 19:55:08.493543 3027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.7:35201 every 8 connection(s)
I20250622 19:55:08.495081 2900 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data/info.pb
I20250622 19:55:08.501446 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 2900
I20250622 19:55:08.501626 2016 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/wal/instance
I20250622 19:55:08.504570 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.8:0
--local_ip_for_outbound_sockets=127.1.248.8
--webserver_interface=127.1.248.8
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250622 19:55:08.508539 3028 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:08.509025 3028 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:08.509852 3028 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:08.572929 2055 ts_manager.cc:194] Registered new tserver with Master: 07ce178dce43415a852dbcce80f47f3e (127.1.248.7:35201)
I20250622 19:55:08.574301 2055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.7:37323
I20250622 19:55:08.598774 2766 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
W20250622 19:55:08.853559 3031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:08.853863 3031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:08.854004 3031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:08.862768 3031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:08.862980 3031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.8
I20250622 19:55:08.873106 3031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data/info.pb
--webserver_interface=127.1.248.8
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.8
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:08.874900 3031 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:08.876740 3031 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:55:08.888818 3038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:08.889051 3040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:08.888800 3037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:08.890103 3031 server_base.cc:1048] running on GCE node
I20250622 19:55:08.890681 3031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:08.891588 3031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:08.892768 3031 hybrid_clock.cc:648] HybridClock initialized: now 1750622108892715 us; error 46 us; skew 500 ppm
I20250622 19:55:08.893110 3031 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:08.895301 3031 webserver.cc:469] Webserver started at http://127.1.248.8:41309/ using document root <none> and password file <none>
I20250622 19:55:08.896039 3031 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:08.896133 3031 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:08.896418 3031 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 19:55:08.898990 3031 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data/instance:
uuid: "7e22adb60d65484c9844ee7d2108a396"
format_stamp: "Formatted at 2025-06-22 19:55:08 on dist-test-slave-t73b"
I20250622 19:55:08.899683 3031 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/wal/instance:
uuid: "7e22adb60d65484c9844ee7d2108a396"
format_stamp: "Formatted at 2025-06-22 19:55:08 on dist-test-slave-t73b"
I20250622 19:55:08.904712 3031 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20250622 19:55:08.908049 3046 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:08.909523 3031 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250622 19:55:08.909665 3031 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/wal
uuid: "7e22adb60d65484c9844ee7d2108a396"
format_stamp: "Formatted at 2025-06-22 19:55:08 on dist-test-slave-t73b"
I20250622 19:55:08.909871 3031 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:08.934783 3031 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:08.935679 3031 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:08.935918 3031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:08.936872 3031 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:08.938547 3031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:55:08.938640 3031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:08.938742 3031 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:55:08.938808 3031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:08.975924 3031 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.8:39693
I20250622 19:55:08.976063 3158 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.8:39693 every 8 connection(s)
I20250622 19:55:08.977324 3031 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data/info.pb
I20250622 19:55:08.984890 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 3031
I20250622 19:55:08.985088 2016 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/wal/instance
I20250622 19:55:08.988749 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.9:0
--local_ip_for_outbound_sockets=127.1.248.9
--webserver_interface=127.1.248.9
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250622 19:55:08.991587 3159 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:08.991933 3159 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:08.993265 3159 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:09.044189 2055 ts_manager.cc:194] Registered new tserver with Master: 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693)
I20250622 19:55:09.045449 2055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.8:47263
I20250622 19:55:09.082158 2897 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
W20250622 19:55:09.340843 3162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:09.341176 3162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:09.341266 3162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:09.349128 3162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:09.349326 3162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.9
I20250622 19:55:09.359320 3162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.9:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data/info.pb
--webserver_interface=127.1.248.9
--webserver_port=0
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.9
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:09.360981 3162 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:09.362898 3162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:55:09.375284 3171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:09.375245 3168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:09.375213 3169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:09.376111 3162 server_base.cc:1048] running on GCE node
I20250622 19:55:09.376675 3162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:09.377539 3162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:09.378721 3162 hybrid_clock.cc:648] HybridClock initialized: now 1750622109378672 us; error 39 us; skew 500 ppm
I20250622 19:55:09.379036 3162 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:09.381268 3162 webserver.cc:469] Webserver started at http://127.1.248.9:38409/ using document root <none> and password file <none>
I20250622 19:55:09.381978 3162 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:09.382072 3162 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:09.382443 3162 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250622 19:55:09.384842 3162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data/instance:
uuid: "7520035c9f604041a00535b523c45c1b"
format_stamp: "Formatted at 2025-06-22 19:55:09 on dist-test-slave-t73b"
I20250622 19:55:09.385587 3162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/wal/instance:
uuid: "7520035c9f604041a00535b523c45c1b"
format_stamp: "Formatted at 2025-06-22 19:55:09 on dist-test-slave-t73b"
I20250622 19:55:09.391569 3162 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250622 19:55:09.394850 3177 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:09.396267 3162 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250622 19:55:09.396430 3162 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/wal
uuid: "7520035c9f604041a00535b523c45c1b"
format_stamp: "Formatted at 2025-06-22 19:55:09 on dist-test-slave-t73b"
I20250622 19:55:09.396616 3162 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:09.421034 3162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:09.421911 3162 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:09.422163 3162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:09.423135 3162 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:09.425377 3162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:55:09.425479 3162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:09.425592 3162 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:55:09.425675 3162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:09.460209 3162 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.9:44015
I20250622 19:55:09.460268 3289 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.9:44015 every 8 connection(s)
I20250622 19:55:09.461738 3162 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data/info.pb
I20250622 19:55:09.469913 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 3162
I20250622 19:55:09.470093 2016 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/wal/instance
I20250622 19:55:09.475365 3290 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:09.475728 3290 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:09.477169 3290 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:09.521596 2055 ts_manager.cc:194] Registered new tserver with Master: 7520035c9f604041a00535b523c45c1b (127.1.248.9:44015)
I20250622 19:55:09.522616 2055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.9:52291
I20250622 19:55:09.525877 2016 external_mini_cluster.cc:934] 9 TS(s) registered with all masters
I20250622 19:55:09.547641 2016 test_util.cc:276] Using random seed: 2029632591
I20250622 19:55:09.569658 2055 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35940:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250622 19:55:09.577077 3028 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
I20250622 19:55:09.612805 2701 tablet_service.cc:1468] Processing CreateTablet for tablet 03ebbbde6ff2448a9a4c52e4cf8d160d (DEFAULT_TABLE table=test-workload [id=17b9e895898f4cf99dcea9a8b2fc6e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 19:55:09.614444 2701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03ebbbde6ff2448a9a4c52e4cf8d160d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 19:55:09.614862 2439 tablet_service.cc:1468] Processing CreateTablet for tablet 03ebbbde6ff2448a9a4c52e4cf8d160d (DEFAULT_TABLE table=test-workload [id=17b9e895898f4cf99dcea9a8b2fc6e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 19:55:09.616354 2439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03ebbbde6ff2448a9a4c52e4cf8d160d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 19:55:09.616365 3094 tablet_service.cc:1468] Processing CreateTablet for tablet 03ebbbde6ff2448a9a4c52e4cf8d160d (DEFAULT_TABLE table=test-workload [id=17b9e895898f4cf99dcea9a8b2fc6e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250622 19:55:09.617758 3094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03ebbbde6ff2448a9a4c52e4cf8d160d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250622 19:55:09.626489 3314 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: Bootstrap starting.
I20250622 19:55:09.631043 3315 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Bootstrap starting.
I20250622 19:55:09.631924 3314 tablet_bootstrap.cc:654] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: Neither blocks nor log segments found. Creating new log.
I20250622 19:55:09.632956 3316 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap starting.
I20250622 19:55:09.633458 3314 log.cc:826] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: Log is configured to *not* fsync() on all Append() calls
I20250622 19:55:09.635300 3315 tablet_bootstrap.cc:654] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Neither blocks nor log segments found. Creating new log.
I20250622 19:55:09.636765 3316 tablet_bootstrap.cc:654] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Neither blocks nor log segments found. Creating new log.
I20250622 19:55:09.636818 3314 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: No bootstrap required, opened a new log
I20250622 19:55:09.636802 3315 log.cc:826] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Log is configured to *not* fsync() on all Append() calls
I20250622 19:55:09.637115 3314 ts_tablet_manager.cc:1397] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20250622 19:55:09.638027 3316 log.cc:826] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Log is configured to *not* fsync() on all Append() calls
I20250622 19:55:09.640270 3315 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: No bootstrap required, opened a new log
I20250622 19:55:09.640591 3315 ts_tablet_manager.cc:1397] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20250622 19:55:09.643872 3316 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: No bootstrap required, opened a new log
I20250622 19:55:09.644246 3316 ts_tablet_manager.cc:1397] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250622 19:55:09.644866 3314 raft_consensus.cc:357] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:09.645315 3314 raft_consensus.cc:383] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 19:55:09.645460 3314 raft_consensus.cc:738] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54555a8ecf924b339db8e8306cbc8bb4, State: Initialized, Role: FOLLOWER
I20250622 19:55:09.646205 3314 consensus_queue.cc:260] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:09.646308 3315 raft_consensus.cc:357] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:09.646636 3315 raft_consensus.cc:383] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 19:55:09.646744 3315 raft_consensus.cc:738] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ece0d51c71b47a1865ba3b05920d999, State: Initialized, Role: FOLLOWER
I20250622 19:55:09.647305 3315 consensus_queue.cc:260] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:09.649330 3315 ts_tablet_manager.cc:1428] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250622 19:55:09.650488 3322 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 19:55:09.650694 3322 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:09.660017 3322 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693), 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117)
I20250622 19:55:09.661937 3316 raft_consensus.cc:357] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:09.662525 3316 raft_consensus.cc:383] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250622 19:55:09.662732 3316 raft_consensus.cc:738] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e22adb60d65484c9844ee7d2108a396, State: Initialized, Role: FOLLOWER
I20250622 19:55:09.663671 3316 consensus_queue.cc:260] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:09.666553 3159 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
I20250622 19:55:09.668308 3316 ts_tablet_manager.cc:1428] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Time spent starting tablet: real 0.024s user 0.012s sys 0.000s
I20250622 19:55:09.669613 3314 ts_tablet_manager.cc:1428] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: Time spent starting tablet: real 0.032s user 0.020s sys 0.008s
I20250622 19:55:09.673465 3114 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "9ece0d51c71b47a1865ba3b05920d999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e22adb60d65484c9844ee7d2108a396" is_pre_election: true
I20250622 19:55:09.673995 3114 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ece0d51c71b47a1865ba3b05920d999 in term 0.
I20250622 19:55:09.675215 2655 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e22adb60d65484c9844ee7d2108a396, 9ece0d51c71b47a1865ba3b05920d999; no voters:
I20250622 19:55:09.675220 2459 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "9ece0d51c71b47a1865ba3b05920d999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54555a8ecf924b339db8e8306cbc8bb4" is_pre_election: true
I20250622 19:55:09.675592 3322 raft_consensus.cc:2802] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250622 19:55:09.675665 2459 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ece0d51c71b47a1865ba3b05920d999 in term 0.
I20250622 19:55:09.675737 3322 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 19:55:09.675835 3322 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 0 FOLLOWER]: Advancing to term 1
I20250622 19:55:09.678175 3322 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:09.678790 3322 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [CANDIDATE]: Term 1 election: Requested vote from peers 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693), 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117)
I20250622 19:55:09.679217 3114 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "9ece0d51c71b47a1865ba3b05920d999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e22adb60d65484c9844ee7d2108a396"
I20250622 19:55:09.679297 2459 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "9ece0d51c71b47a1865ba3b05920d999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54555a8ecf924b339db8e8306cbc8bb4"
I20250622 19:55:09.679390 3114 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 0 FOLLOWER]: Advancing to term 1
I20250622 19:55:09.679497 2459 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 0 FOLLOWER]: Advancing to term 1
I20250622 19:55:09.681573 3114 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ece0d51c71b47a1865ba3b05920d999 in term 1.
I20250622 19:55:09.681778 2459 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ece0d51c71b47a1865ba3b05920d999 in term 1.
I20250622 19:55:09.681991 2655 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e22adb60d65484c9844ee7d2108a396, 9ece0d51c71b47a1865ba3b05920d999; no voters:
I20250622 19:55:09.682340 3322 raft_consensus.cc:2802] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 1 FOLLOWER]: Leader election won for term 1
I20250622 19:55:09.682530 3322 raft_consensus.cc:695] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 1 LEADER]: Becoming Leader. State: Replica: 9ece0d51c71b47a1865ba3b05920d999, State: Running, Role: LEADER
I20250622 19:55:09.683014 3322 consensus_queue.cc:237] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:09.687544 2053 catalog_manager.cc:5582] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5). New cstate: current_term: 1 leader_uuid: "9ece0d51c71b47a1865ba3b05920d999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } health_report { overall_health: HEALTHY } } }
I20250622 19:55:09.697505 2016 maintenance_mode-itest.cc:745] Restarting batch of 3 tservers: e63bffd596554411a2e0489c19e78fff,9ece0d51c71b47a1865ba3b05920d999,07ce178dce43415a852dbcce80f47f3e
W20250622 19:55:09.729288 3160 tablet.cc:2378] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 19:55:09.779909 3322 consensus_queue.cc:1035] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250622 19:55:09.785015 2459 raft_consensus.cc:1273] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 1 FOLLOWER]: Refusing update from remote peer 9ece0d51c71b47a1865ba3b05920d999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250622 19:55:09.786182 3322 consensus_queue.cc:1035] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250622 19:55:09.787139 2767 tablet.cc:2378] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250622 19:55:09.844614 3342 mvcc.cc:204] Tried to move back new op lower bound from 7170548161669488640 to 7170548161265201152. Current Snapshot: MvccSnapshot[applied={T|T < 7170548161669488640 or (T in {7170548161669488640})}]
I20250622 19:55:10.525754 3290 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
I20250622 19:55:10.585860 2054 ts_manager.cc:295] Set tserver state for 9ece0d51c71b47a1865ba3b05920d999 to MAINTENANCE_MODE
I20250622 19:55:10.614125 2050 ts_manager.cc:295] Set tserver state for 07ce178dce43415a852dbcce80f47f3e to MAINTENANCE_MODE
I20250622 19:55:11.428959 2050 ts_manager.cc:295] Set tserver state for e63bffd596554411a2e0489c19e78fff to MAINTENANCE_MODE
I20250622 19:55:11.835881 2963 tablet_service.cc:1423] Tablet server 07ce178dce43415a852dbcce80f47f3e set to quiescing
I20250622 19:55:11.836161 2963 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:13.837785 2701 tablet_service.cc:1423] Tablet server 9ece0d51c71b47a1865ba3b05920d999 set to quiescing
I20250622 19:55:13.838155 2701 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 19:55:13.838497 3354 raft_consensus.cc:991] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: : Instructing follower 7e22adb60d65484c9844ee7d2108a396 to start an election
I20250622 19:55:13.838848 3354 raft_consensus.cc:1079] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 1 LEADER]: Signalling peer 7e22adb60d65484c9844ee7d2108a396 to start an election
I20250622 19:55:13.840929 3113 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d"
dest_uuid: "7e22adb60d65484c9844ee7d2108a396"
from {username='slave'} at 127.1.248.5:38917
I20250622 19:55:13.841333 3113 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 19:55:13.841562 3113 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 1 FOLLOWER]: Advancing to term 2
I20250622 19:55:13.853111 3354 raft_consensus.cc:991] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: : Instructing follower 54555a8ecf924b339db8e8306cbc8bb4 to start an election
I20250622 19:55:13.853351 3354 raft_consensus.cc:1079] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 1 LEADER]: Signalling peer 54555a8ecf924b339db8e8306cbc8bb4 to start an election
I20250622 19:55:13.859202 2459 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d"
dest_uuid: "54555a8ecf924b339db8e8306cbc8bb4"
from {username='slave'} at 127.1.248.5:37695
I20250622 19:55:13.859642 2459 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 19:55:13.859850 2459 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 1 FOLLOWER]: Advancing to term 2
I20250622 19:55:13.864353 2459 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:13.867699 2458 raft_consensus.cc:1238] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 2 FOLLOWER]: Rejecting Update request from peer 9ece0d51c71b47a1865ba3b05920d999 for earlier term 1. Current term is 2. Ops: []
I20250622 19:55:13.868963 3418 consensus_queue.cc:1046] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 }, Status: INVALID_TERM, Last received: 1.171, Next index: 172, Last known committed idx: 170, Time since last communication: 0.000s
I20250622 19:55:13.869271 3418 raft_consensus.cc:3053] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 1 LEADER]: Stepping down as leader of term 1
I20250622 19:55:13.869390 3418 raft_consensus.cc:738] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9ece0d51c71b47a1865ba3b05920d999, State: Running, Role: LEADER
I20250622 19:55:13.869746 3418 consensus_queue.cc:260] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 171, Committed index: 171, Last appended: 1.172, Last appended by leader: 172, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:13.870067 3418 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 1 FOLLOWER]: Advancing to term 2
I20250622 19:55:13.875002 2459 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [CANDIDATE]: Term 2 election: Requested vote from peers 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693), 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
I20250622 19:55:13.891338 3113 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:13.892277 3114 raft_consensus.cc:1238] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 2 FOLLOWER]: Rejecting Update request from peer 9ece0d51c71b47a1865ba3b05920d999 for earlier term 1. Current term is 2. Ops: [1.171-1.171]
I20250622 19:55:13.894662 3113 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 2 election: Requested vote from peers 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117), 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
W20250622 19:55:13.897280 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:13.905421 3074 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:13.909559 3071 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
I20250622 19:55:13.911114 2721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "54555a8ecf924b339db8e8306cbc8bb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 171 } } ignore_live_leader: true dest_uuid: "9ece0d51c71b47a1865ba3b05920d999"
I20250622 19:55:13.911610 2721 raft_consensus.cc:2408] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 54555a8ecf924b339db8e8306cbc8bb4 for term 2 because replica has last-logged OpId of term: 1 index: 172, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 171.
I20250622 19:55:13.939222 3113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "54555a8ecf924b339db8e8306cbc8bb4" candidate_term: 2 candidate_status { last_received { term: 1 index: 171 } } ignore_live_leader: true dest_uuid: "7e22adb60d65484c9844ee7d2108a396"
I20250622 19:55:13.939597 3113 raft_consensus.cc:2391] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 54555a8ecf924b339db8e8306cbc8bb4 in current term 2: Already voted for candidate 7e22adb60d65484c9844ee7d2108a396 in this term.
I20250622 19:55:13.947830 2393 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54555a8ecf924b339db8e8306cbc8bb4; no voters: 7e22adb60d65484c9844ee7d2108a396, 9ece0d51c71b47a1865ba3b05920d999
I20250622 19:55:13.951541 3445 raft_consensus.cc:2747] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250622 19:55:13.953018 2459 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "7e22adb60d65484c9844ee7d2108a396" candidate_term: 2 candidate_status { last_received { term: 1 index: 170 } } ignore_live_leader: true dest_uuid: "54555a8ecf924b339db8e8306cbc8bb4"
I20250622 19:55:13.956176 2459 raft_consensus.cc:2391] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e22adb60d65484c9844ee7d2108a396 in current term 2: Already voted for candidate 54555a8ecf924b339db8e8306cbc8bb4 in this term.
I20250622 19:55:13.968308 2721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "7e22adb60d65484c9844ee7d2108a396" candidate_term: 2 candidate_status { last_received { term: 1 index: 170 } } ignore_live_leader: true dest_uuid: "9ece0d51c71b47a1865ba3b05920d999"
I20250622 19:55:13.968744 2721 raft_consensus.cc:2408] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e22adb60d65484c9844ee7d2108a396 for term 2 because replica has last-logged OpId of term: 1 index: 172, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 170.
I20250622 19:55:13.969856 3047 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e22adb60d65484c9844ee7d2108a396; no voters: 54555a8ecf924b339db8e8306cbc8bb4, 9ece0d51c71b47a1865ba3b05920d999
I20250622 19:55:13.996376 3446 raft_consensus.cc:2747] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250622 19:55:13.997651 2418 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:13.999684 2419 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.005280 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.008770 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.016273 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.016819 3073 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.025534 2419 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.029573 3070 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.040020 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.043234 2419 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.050133 2419 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.053941 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.063923 3073 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.066892 3070 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.081300 2419 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.082821 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.095523 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.098220 3072 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.110369 3073 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.116109 2419 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.124742 2419 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.136469 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.141817 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.157698 3385 raft_consensus.cc:668] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: failed to trigger leader election: Illegal state: leader elections are disabled
W20250622 19:55:14.159849 3071 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.160880 3072 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.177981 2419 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.180135 2419 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.199466 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.202174 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.219925 3072 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.220674 3070 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.241396 2419 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.241461 2418 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.265295 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.265635 2680 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.287616 3070 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.290455 3070 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.316138 2419 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.317014 2418 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.347079 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.347143 2680 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.375658 3072 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.378094 3073 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
I20250622 19:55:14.396996 3445 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 19:55:14.397259 3445 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:14.398073 3445 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693), 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
I20250622 19:55:14.398730 3113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "54555a8ecf924b339db8e8306cbc8bb4" candidate_term: 3 candidate_status { last_received { term: 1 index: 171 } } ignore_live_leader: false dest_uuid: "7e22adb60d65484c9844ee7d2108a396" is_pre_election: true
I20250622 19:55:14.399025 3113 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54555a8ecf924b339db8e8306cbc8bb4 in term 2.
I20250622 19:55:14.399109 2721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "54555a8ecf924b339db8e8306cbc8bb4" candidate_term: 3 candidate_status { last_received { term: 1 index: 171 } } ignore_live_leader: false dest_uuid: "9ece0d51c71b47a1865ba3b05920d999" is_pre_election: true
I20250622 19:55:14.399425 2721 raft_consensus.cc:2408] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54555a8ecf924b339db8e8306cbc8bb4 for term 3 because replica has last-logged OpId of term: 1 index: 172, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 171.
I20250622 19:55:14.400148 2393 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54555a8ecf924b339db8e8306cbc8bb4, 7e22adb60d65484c9844ee7d2108a396; no voters:
I20250622 19:55:14.401104 3445 raft_consensus.cc:2802] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250622 19:55:14.401212 3445 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 19:55:14.401324 3445 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 2 FOLLOWER]: Advancing to term 3
W20250622 19:55:14.405462 2418 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.407594 2418 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
I20250622 19:55:14.409705 3445 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:14.412848 3113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "54555a8ecf924b339db8e8306cbc8bb4" candidate_term: 3 candidate_status { last_received { term: 1 index: 171 } } ignore_live_leader: false dest_uuid: "7e22adb60d65484c9844ee7d2108a396"
I20250622 19:55:14.413077 3113 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 2 FOLLOWER]: Advancing to term 3
I20250622 19:55:14.413352 2721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "54555a8ecf924b339db8e8306cbc8bb4" candidate_term: 3 candidate_status { last_received { term: 1 index: 171 } } ignore_live_leader: false dest_uuid: "9ece0d51c71b47a1865ba3b05920d999"
I20250622 19:55:14.413569 2721 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 2 FOLLOWER]: Advancing to term 3
I20250622 19:55:14.415345 3113 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54555a8ecf924b339db8e8306cbc8bb4 in term 3.
I20250622 19:55:14.416801 2721 raft_consensus.cc:2408] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 54555a8ecf924b339db8e8306cbc8bb4 for term 3 because replica has last-logged OpId of term: 1 index: 172, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 171.
I20250622 19:55:14.417645 2393 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54555a8ecf924b339db8e8306cbc8bb4, 7e22adb60d65484c9844ee7d2108a396; no voters: 9ece0d51c71b47a1865ba3b05920d999
I20250622 19:55:14.420379 3445 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [CANDIDATE]: Term 3 election: Requested vote from peers 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693), 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
I20250622 19:55:14.422753 3445 raft_consensus.cc:2802] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 3 FOLLOWER]: Leader election won for term 3
I20250622 19:55:14.423280 3445 raft_consensus.cc:695] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 3 LEADER]: Becoming Leader. State: Replica: 54555a8ecf924b339db8e8306cbc8bb4, State: Running, Role: LEADER
I20250622 19:55:14.424082 3445 consensus_queue.cc:237] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 170, Committed index: 170, Last appended: 1.171, Last appended by leader: 171, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:14.436384 2053 catalog_manager.cc:5582] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 reported cstate change: term changed from 1 to 3, leader changed from 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5) to 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3). New cstate: current_term: 3 leader_uuid: "54555a8ecf924b339db8e8306cbc8bb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } health_report { overall_health: UNKNOWN } } }
W20250622 19:55:14.442721 2680 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.443324 2681 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.471135 3070 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:14.474023 3073 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
I20250622 19:55:14.512369 3113 raft_consensus.cc:1273] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 3 FOLLOWER]: Refusing update from remote peer 54555a8ecf924b339db8e8306cbc8bb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 3 index: 172. (index mismatch)
I20250622 19:55:14.515574 3449 consensus_queue.cc:1035] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 170, Time since last communication: 0.000s
I20250622 19:55:14.546741 2721 raft_consensus.cc:1273] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 3 FOLLOWER]: Refusing update from remote peer 54555a8ecf924b339db8e8306cbc8bb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 172. Preceding OpId from leader: term: 3 index: 172. (term mismatch)
I20250622 19:55:14.547009 2721 pending_rounds.cc:85] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Aborting all ops after (but not including) 171
I20250622 19:55:14.547102 2721 pending_rounds.cc:107] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Aborting uncommitted WRITE_OP operation due to leader change: 1.172
W20250622 19:55:14.547586 2721 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:59178: Aborted: Op aborted by new leader
I20250622 19:55:14.549811 3449 consensus_queue.cc:1035] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 172, Last known committed idx: 171, Time since last communication: 0.000s
I20250622 19:55:14.580153 3459 mvcc.cc:204] Tried to move back new op lower bound from 7170548181118873600 to 7170548180710830080. Current Snapshot: MvccSnapshot[applied={T|T < 7170548181118873600}]
I20250622 19:55:15.041993 2832 tablet_service.cc:1423] Tablet server e63bffd596554411a2e0489c19e78fff set to quiescing
I20250622 19:55:15.042282 2832 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:17.414703 2701 tablet_service.cc:1423] Tablet server 9ece0d51c71b47a1865ba3b05920d999 set to quiescing
I20250622 19:55:17.414994 2701 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:18.895241 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 2769
I20250622 19:55:18.919674 2892 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:55:19.099226 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 2769
I20250622 19:55:19.129674 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.6:38357
--local_ip_for_outbound_sockets=127.1.248.6
--tserver_master_addrs=127.1.248.62:37973
--webserver_port=36755
--webserver_interface=127.1.248.6
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250622 19:55:19.840878 3510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:19.841467 3510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:19.841694 3510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:19.867502 3510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:19.867779 3510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.6
I20250622 19:55:19.890172 3510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.6:38357
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data/info.pb
--webserver_interface=127.1.248.6
--webserver_port=36755
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.6
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:19.893457 3510 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:19.896139 3510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:55:19.930624 3519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:19.933347 3510 server_base.cc:1048] running on GCE node
W20250622 19:55:19.935753 3521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:19.940348 3518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:19.941532 3510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:19.947036 3510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:19.948474 3510 hybrid_clock.cc:648] HybridClock initialized: now 1750622119948390 us; error 63 us; skew 500 ppm
I20250622 19:55:19.949007 3510 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:20.053664 3510 webserver.cc:469] Webserver started at http://127.1.248.6:36755/ using document root <none> and password file <none>
I20250622 19:55:20.055090 3510 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:20.055343 3510 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:20.065366 3510 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.006s sys 0.001s
I20250622 19:55:20.073228 3527 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:20.075320 3510 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.000s sys 0.003s
I20250622 19:55:20.075637 3510 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/wal
uuid: "e63bffd596554411a2e0489c19e78fff"
format_stamp: "Formatted at 2025-06-22 19:55:07 on dist-test-slave-t73b"
I20250622 19:55:20.076728 3510 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:20.135250 3510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:20.136382 3510 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:20.136804 3510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:20.146703 3510 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:20.155658 3510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:55:20.155879 3510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250622 19:55:20.156118 3510 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:55:20.156244 3510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:21.052332 3641 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.6:38357 every 8 connection(s)
I20250622 19:55:21.052289 3510 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.6:38357
I20250622 19:55:21.054968 3510 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data/info.pb
I20250622 19:55:21.059633 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 3510
I20250622 19:55:21.059864 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 2638
I20250622 19:55:21.118784 2761 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:55:21.130577 3642 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:21.130999 3642 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:21.132177 3642 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:21.135877 2048 ts_manager.cc:194] Re-registered known tserver with Master: e63bffd596554411a2e0489c19e78fff (127.1.248.6:38357)
I20250622 19:55:21.137104 2048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.6:51697
I20250622 19:55:21.365279 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 2638
W20250622 19:55:21.405182 2392 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.248.5:43933 (error 108)
I20250622 19:55:21.406967 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.5:43933
--local_ip_for_outbound_sockets=127.1.248.5
--tserver_master_addrs=127.1.248.62:37973
--webserver_port=34689
--webserver_interface=127.1.248.5
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250622 19:55:21.428323 2392 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933): Couldn't send request to peer 9ece0d51c71b47a1865ba3b05920d999. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.5:43933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250622 19:55:21.853665 2392 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933): Couldn't send request to peer 9ece0d51c71b47a1865ba3b05920d999. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.5:43933: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250622 19:55:22.110436 3653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:22.110819 3653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:22.110967 3653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:22.119731 3653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:22.119951 3653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.5
I20250622 19:55:22.133615 3653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.5:43933
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data/info.pb
--webserver_interface=127.1.248.5
--webserver_port=34689
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.5
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:22.135950 3653 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:22.137867 3653 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250622 19:55:22.141592 3642 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
W20250622 19:55:22.157963 3659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:22.160717 3660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:22.171501 3662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:22.173530 3653 server_base.cc:1048] running on GCE node
I20250622 19:55:22.176308 3653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:22.178143 3653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:22.191318 3653 hybrid_clock.cc:648] HybridClock initialized: now 1750622122189711 us; error 1589 us; skew 500 ppm
I20250622 19:55:22.191695 3653 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:22.197190 3653 webserver.cc:469] Webserver started at http://127.1.248.5:34689/ using document root <none> and password file <none>
I20250622 19:55:22.198321 3653 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:22.198462 3653 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:22.217377 3653 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.005s sys 0.005s
I20250622 19:55:22.230885 3668 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:22.234783 3653 fs_manager.cc:730] Time spent opening block manager: real 0.014s user 0.005s sys 0.004s
I20250622 19:55:22.236336 3653 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/wal
uuid: "9ece0d51c71b47a1865ba3b05920d999"
format_stamp: "Formatted at 2025-06-22 19:55:07 on dist-test-slave-t73b"
I20250622 19:55:22.237437 3653 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:22.284793 3653 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:22.285974 3653 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:22.286293 3653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:22.301651 3653 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:22.315351 3675 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 19:55:22.331281 3653 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 19:55:22.331416 3653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.019s user 0.002s sys 0.001s
I20250622 19:55:22.331590 3653 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 19:55:22.341434 3653 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 19:55:22.341634 3653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s user 0.005s sys 0.000s
I20250622 19:55:22.350725 3675 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Bootstrap starting.
W20250622 19:55:22.418361 2392 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933): Couldn't send request to peer 9ece0d51c71b47a1865ba3b05920d999. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.5:43933: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250622 19:55:22.804754 3675 log.cc:826] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Log is configured to *not* fsync() on all Append() calls
W20250622 19:55:22.894049 2392 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933): Couldn't send request to peer 9ece0d51c71b47a1865ba3b05920d999. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.5:43933: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250622 19:55:23.309101 3783 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.5:43933 every 8 connection(s)
I20250622 19:55:23.309458 3653 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.5:43933
I20250622 19:55:23.311391 3653 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data/info.pb
I20250622 19:55:23.318390 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 3653
I20250622 19:55:23.318643 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 2900
I20250622 19:55:23.381878 3023 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:55:23.399113 3784 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:23.399586 3784 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:23.401036 3784 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:23.427312 2048 ts_manager.cc:194] Re-registered known tserver with Master: 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
I20250622 19:55:23.428932 2048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.5:48891
W20250622 19:55:23.464735 2392 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933): Couldn't send request to peer 9ece0d51c71b47a1865ba3b05920d999. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
I20250622 19:55:23.632508 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 2900
I20250622 19:55:23.690194 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.7:35201
--local_ip_for_outbound_sockets=127.1.248.7
--tserver_master_addrs=127.1.248.62:37973
--webserver_port=39729
--webserver_interface=127.1.248.7
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250622 19:55:23.938465 2392 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933): Couldn't send request to peer 9ece0d51c71b47a1865ba3b05920d999. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
W20250622 19:55:24.344305 3792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:24.344798 3792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:24.344982 3792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:24.371999 3792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:24.372327 3792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.7
I20250622 19:55:24.397408 3792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.7:35201
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data/info.pb
--webserver_interface=127.1.248.7
--webserver_port=39729
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.7
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:24.401105 3792 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:24.404521 3792 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250622 19:55:24.432911 3784 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
W20250622 19:55:24.445415 2392 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933): Couldn't send request to peer 9ece0d51c71b47a1865ba3b05920d999. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
W20250622 19:55:24.458959 3800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:24.459488 3799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:24.471657 3802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:24.484269 3792 server_base.cc:1048] running on GCE node
I20250622 19:55:24.485342 3792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:24.504956 3792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:24.506371 3792 hybrid_clock.cc:648] HybridClock initialized: now 1750622124506285 us; error 77 us; skew 500 ppm
I20250622 19:55:24.507277 3792 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:24.535769 3792 webserver.cc:469] Webserver started at http://127.1.248.7:39729/ using document root <none> and password file <none>
I20250622 19:55:24.537279 3792 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:24.537526 3792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:24.552474 3792 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.006s sys 0.005s
I20250622 19:55:24.566330 3808 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:24.569860 3792 fs_manager.cc:730] Time spent opening block manager: real 0.015s user 0.003s sys 0.001s
I20250622 19:55:24.572034 3792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/wal
uuid: "07ce178dce43415a852dbcce80f47f3e"
format_stamp: "Formatted at 2025-06-22 19:55:08 on dist-test-slave-t73b"
I20250622 19:55:24.574334 3792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:24.678462 3792 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:24.684020 3792 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:24.684432 3792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:24.695317 3792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:24.707937 3792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:55:24.708078 3792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:24.708245 3792 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:55:24.708307 3792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20250622 19:55:25.009502 2392 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933): Couldn't send request to peer 9ece0d51c71b47a1865ba3b05920d999. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20250622 19:55:25.491397 3792 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.7:35201
I20250622 19:55:25.491448 3923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.7:35201 every 8 connection(s)
I20250622 19:55:25.494344 3792 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data/info.pb
I20250622 19:55:25.499787 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 3792
I20250622 19:55:25.509019 3751 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 9ece0d51c71b47a1865ba3b05920d999 is lagging by at least 3 ops behind the committed index
I20250622 19:55:25.550868 3924 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:25.551379 3924 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:25.552572 3924 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:25.554505 2048 ts_manager.cc:194] Re-registered known tserver with Master: 07ce178dce43415a852dbcce80f47f3e (127.1.248.7:35201)
I20250622 19:55:25.555675 2048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.7:49747
W20250622 19:55:25.562430 2392 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933): Couldn't send request to peer 9ece0d51c71b47a1865ba3b05920d999. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20250622 19:55:26.049917 2392 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933): Couldn't send request to peer 9ece0d51c71b47a1865ba3b05920d999. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250622 19:55:26.558404 2392 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933): Couldn't send request to peer 9ece0d51c71b47a1865ba3b05920d999. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
I20250622 19:55:26.559710 3924 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
I20250622 19:55:26.945722 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:26.965749 2439 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 19:55:26.986791 2570 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:26.988701 2308 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:27.064129 3094 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250622 19:55:27.067749 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:27.099126 3225 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:27.111618 2176 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:27.134938 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 19:55:27.189504 2392 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933): Couldn't send request to peer 9ece0d51c71b47a1865ba3b05920d999. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
I20250622 19:55:27.568488 3675 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Bootstrap replayed 1/1 log segments. Stats: ops{read=756 overwritten=1 applied=753 ignored=0} inserts{seen=37550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250622 19:55:27.569434 3675 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Bootstrap complete.
I20250622 19:55:27.572943 3675 ts_tablet_manager.cc:1397] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Time spent bootstrapping tablet: real 5.230s user 3.266s sys 0.268s
I20250622 19:55:27.587718 3675 raft_consensus.cc:357] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 3 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:27.637984 3675 raft_consensus.cc:738] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ece0d51c71b47a1865ba3b05920d999, State: Initialized, Role: FOLLOWER
I20250622 19:55:27.665287 3675 consensus_queue.cc:260] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 753, Last appended: 3.755, Last appended by leader: 755, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:27.667825 3675 ts_tablet_manager.cc:1428] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Time spent starting tablet: real 0.095s user 0.021s sys 0.012s
I20250622 19:55:27.728682 3738 raft_consensus.cc:1215] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.753->[3.754-3.1178] Dedup: 3.755->[3.756-3.1178]
I20250622 19:55:28.160848 3977 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 54555a8ecf924b339db8e8306cbc8bb4)
I20250622 19:55:28.161150 3977 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:28.163443 3977 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693), 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117)
I20250622 19:55:28.204173 3113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "9ece0d51c71b47a1865ba3b05920d999" candidate_term: 4 candidate_status { last_received { term: 3 index: 1395 } } ignore_live_leader: false dest_uuid: "7e22adb60d65484c9844ee7d2108a396" is_pre_election: true
I20250622 19:55:28.212711 2459 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "9ece0d51c71b47a1865ba3b05920d999" candidate_term: 4 candidate_status { last_received { term: 3 index: 1395 } } ignore_live_leader: false dest_uuid: "54555a8ecf924b339db8e8306cbc8bb4" is_pre_election: true
I20250622 19:55:28.215184 3670 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ece0d51c71b47a1865ba3b05920d999; no voters: 54555a8ecf924b339db8e8306cbc8bb4, 7e22adb60d65484c9844ee7d2108a396
I20250622 19:55:28.216048 3977 raft_consensus.cc:2747] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250622 19:55:34.104331 2308 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:34.131857 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:34.198554 3225 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:34.201283 2176 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:34.253484 2439 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 19:55:34.265134 2570 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:34.323063 3094 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250622 19:55:34.352072 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:34.395056 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:35.763747 2053 ts_manager.cc:284] Unset tserver state for 9ece0d51c71b47a1865ba3b05920d999 from MAINTENANCE_MODE
I20250622 19:55:35.765038 3159 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:35.783802 2048 ts_manager.cc:284] Unset tserver state for 07ce178dce43415a852dbcce80f47f3e from MAINTENANCE_MODE
I20250622 19:55:35.814710 2241 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:35.855473 2050 ts_manager.cc:284] Unset tserver state for e63bffd596554411a2e0489c19e78fff from MAINTENANCE_MODE
I20250622 19:55:36.035380 3784 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:36.043006 2504 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:36.179132 3642 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:36.205331 2635 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:36.380688 2373 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:36.591467 3924 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:36.663861 3290 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:36.793450 3159 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:36.821619 2241 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:38.710107 2016 maintenance_mode-itest.cc:745] Restarting batch of 3 tservers: 2db5cde30e5a47fba8d434cc04849edc,07bc6e053fe04f6d8b0c0157ca553029,54555a8ecf924b339db8e8306cbc8bb4
I20250622 19:55:39.541049 2050 ts_manager.cc:295] Set tserver state for 07bc6e053fe04f6d8b0c0157ca553029 to MAINTENANCE_MODE
I20250622 19:55:40.079496 2050 ts_manager.cc:295] Set tserver state for 54555a8ecf924b339db8e8306cbc8bb4 to MAINTENANCE_MODE
I20250622 19:55:40.109688 2048 ts_manager.cc:295] Set tserver state for 2db5cde30e5a47fba8d434cc04849edc to MAINTENANCE_MODE
I20250622 19:55:41.885102 2176 tablet_service.cc:1423] Tablet server 2db5cde30e5a47fba8d434cc04849edc set to quiescing
I20250622 19:55:41.885505 2176 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:43.234263 2439 tablet_service.cc:1423] Tablet server 54555a8ecf924b339db8e8306cbc8bb4 set to quiescing
I20250622 19:55:43.234558 2439 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 19:55:43.238922 4094 raft_consensus.cc:991] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: : Instructing follower 9ece0d51c71b47a1865ba3b05920d999 to start an election
I20250622 19:55:43.239102 4094 raft_consensus.cc:1079] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 3 LEADER]: Signalling peer 9ece0d51c71b47a1865ba3b05920d999 to start an election
I20250622 19:55:43.243651 4153 raft_consensus.cc:991] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: : Instructing follower 9ece0d51c71b47a1865ba3b05920d999 to start an election
I20250622 19:55:43.243827 4153 raft_consensus.cc:1079] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 3 LEADER]: Signalling peer 9ece0d51c71b47a1865ba3b05920d999 to start an election
I20250622 19:55:43.251579 3738 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d"
dest_uuid: "9ece0d51c71b47a1865ba3b05920d999"
from {username='slave'} at 127.1.248.3:58273
I20250622 19:55:43.251883 3738 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 19:55:43.252025 3738 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 3 FOLLOWER]: Advancing to term 4
I20250622 19:55:43.258205 4102 raft_consensus.cc:991] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: : Instructing follower 7e22adb60d65484c9844ee7d2108a396 to start an election
I20250622 19:55:43.258123 3737 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d"
dest_uuid: "9ece0d51c71b47a1865ba3b05920d999"
from {username='slave'} at 127.1.248.3:58273
I20250622 19:55:43.258342 4102 raft_consensus.cc:1079] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 3 LEADER]: Signalling peer 7e22adb60d65484c9844ee7d2108a396 to start an election
I20250622 19:55:43.259443 3113 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d"
dest_uuid: "7e22adb60d65484c9844ee7d2108a396"
from {username='slave'} at 127.1.248.3:45135
I20250622 19:55:43.259847 3113 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 19:55:43.259984 3113 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 3 FOLLOWER]: Advancing to term 4
I20250622 19:55:43.262934 3113 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:43.264375 2459 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "7e22adb60d65484c9844ee7d2108a396" candidate_term: 4 candidate_status { last_received { term: 3 index: 2592 } } ignore_live_leader: true dest_uuid: "54555a8ecf924b339db8e8306cbc8bb4"
I20250622 19:55:43.264525 3113 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 4 election: Requested vote from peers 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117), 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
I20250622 19:55:43.264640 2459 raft_consensus.cc:3053] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 3 LEADER]: Stepping down as leader of term 3
I20250622 19:55:43.264739 2459 raft_consensus.cc:738] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 54555a8ecf924b339db8e8306cbc8bb4, State: Running, Role: LEADER
I20250622 19:55:43.265129 2459 consensus_queue.cc:260] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2592, Committed index: 2592, Last appended: 3.2592, Last appended by leader: 2592, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:43.265350 2459 logging.cc:424] LogThrottler TrackedPeer Lag: suppressed but not reported on 20 messages since previous log ~17 seconds ago
I20250622 19:55:43.265470 2459 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 3 FOLLOWER]: Advancing to term 4
W20250622 19:55:43.266114 2417 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
I20250622 19:55:43.268191 2459 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e22adb60d65484c9844ee7d2108a396 in term 4.
I20250622 19:55:43.268272 3738 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:43.269173 3736 raft_consensus.cc:1238] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 4 FOLLOWER]: Rejecting Update request from peer 54555a8ecf924b339db8e8306cbc8bb4 for earlier term 3. Current term is 4. Ops: [3.2592-3.2592]
I20250622 19:55:43.270642 3737 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 19:55:43.270792 3737 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 4 FOLLOWER]: Advancing to term 5
I20250622 19:55:43.271524 3048 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54555a8ecf924b339db8e8306cbc8bb4, 7e22adb60d65484c9844ee7d2108a396; no voters:
I20250622 19:55:43.272203 4188 raft_consensus.cc:2802] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 4 FOLLOWER]: Leader election won for term 4
I20250622 19:55:43.274770 3737 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:43.276458 3113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "9ece0d51c71b47a1865ba3b05920d999" candidate_term: 4 candidate_status { last_received { term: 3 index: 2591 } } ignore_live_leader: true dest_uuid: "7e22adb60d65484c9844ee7d2108a396"
I20250622 19:55:43.276769 2459 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "9ece0d51c71b47a1865ba3b05920d999" candidate_term: 4 candidate_status { last_received { term: 3 index: 2591 } } ignore_live_leader: true dest_uuid: "54555a8ecf924b339db8e8306cbc8bb4"
I20250622 19:55:43.277128 2459 raft_consensus.cc:2391] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 9ece0d51c71b47a1865ba3b05920d999 in current term 4: Already voted for candidate 7e22adb60d65484c9844ee7d2108a396 in this term.
I20250622 19:55:43.277981 3738 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [CANDIDATE]: Term 4 election: Requested vote from peers 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693), 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117)
I20250622 19:55:43.279484 3114 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "9ece0d51c71b47a1865ba3b05920d999" candidate_term: 5 candidate_status { last_received { term: 3 index: 2591 } } ignore_live_leader: true dest_uuid: "7e22adb60d65484c9844ee7d2108a396"
W20250622 19:55:43.283449 3074 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Service unavailable: leader is not yet ready
W20250622 19:55:43.283564 3070 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Service unavailable: leader is not yet ready
W20250622 19:55:43.284507 3069 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Service unavailable: leader is not yet ready
I20250622 19:55:43.285777 2459 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "9ece0d51c71b47a1865ba3b05920d999" candidate_term: 5 candidate_status { last_received { term: 3 index: 2591 } } ignore_live_leader: true dest_uuid: "54555a8ecf924b339db8e8306cbc8bb4"
I20250622 19:55:43.286099 2459 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 4 FOLLOWER]: Advancing to term 5
W20250622 19:55:43.290587 3073 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Service unavailable: leader is not yet ready
W20250622 19:55:43.291251 3071 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Service unavailable: leader is not yet ready
I20250622 19:55:43.294505 2459 raft_consensus.cc:2408] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 9ece0d51c71b47a1865ba3b05920d999 for term 5 because replica has last-logged OpId of term: 3 index: 2592, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2591.
I20250622 19:55:43.295117 3670 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ece0d51c71b47a1865ba3b05920d999; no voters: 54555a8ecf924b339db8e8306cbc8bb4, 7e22adb60d65484c9844ee7d2108a396
I20250622 19:55:43.297583 3737 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [CANDIDATE]: Term 5 election: Requested vote from peers 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693), 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117)
I20250622 19:55:43.303792 4190 raft_consensus.cc:2747] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250622 19:55:43.308112 3069 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Service unavailable: leader is not yet ready
W20250622 19:55:43.309633 3072 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Service unavailable: leader is not yet ready
I20250622 19:55:43.310458 4188 raft_consensus.cc:695] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 4 LEADER]: Becoming Leader. State: Replica: 7e22adb60d65484c9844ee7d2108a396, State: Running, Role: LEADER
W20250622 19:55:43.315001 3072 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Service unavailable: leader is not yet ready
W20250622 19:55:43.315137 3069 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Service unavailable: leader is not yet ready
I20250622 19:55:43.313724 4188 consensus_queue.cc:237] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2592, Committed index: 2592, Last appended: 3.2592, Last appended by leader: 2592, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:43.318936 3737 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "7e22adb60d65484c9844ee7d2108a396" candidate_term: 4 candidate_status { last_received { term: 3 index: 2592 } } ignore_live_leader: true dest_uuid: "9ece0d51c71b47a1865ba3b05920d999"
I20250622 19:55:43.319356 3737 raft_consensus.cc:2366] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e22adb60d65484c9844ee7d2108a396 for earlier term 4. Current term is 5.
I20250622 19:55:43.319914 3047 leader_election.cc:400] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 4 election: Vote denied by peer 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933) with higher term. Message: Invalid argument: T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 7e22adb60d65484c9844ee7d2108a396 for earlier term 4. Current term is 5.
W20250622 19:55:43.323617 3070 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Service unavailable: leader is not yet ready
W20250622 19:55:43.324206 3073 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Service unavailable: leader is not yet ready
W20250622 19:55:43.325249 3072 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Service unavailable: leader is not yet ready
I20250622 19:55:43.328938 3113 raft_consensus.cc:2391] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 4 LEADER]: Leader election vote request: Denying vote to candidate 9ece0d51c71b47a1865ba3b05920d999 in current term 4: Already voted for candidate 7e22adb60d65484c9844ee7d2108a396 in this term.
I20250622 19:55:43.330106 3670 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ece0d51c71b47a1865ba3b05920d999; no voters: 54555a8ecf924b339db8e8306cbc8bb4, 7e22adb60d65484c9844ee7d2108a396
I20250622 19:55:43.330619 4190 raft_consensus.cc:2747] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250622 19:55:43.332278 2048 catalog_manager.cc:5582] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 reported cstate change: term changed from 3 to 4, leader changed from 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3) to 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8). New cstate: current_term: 4 leader_uuid: "7e22adb60d65484c9844ee7d2108a396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } health_report { overall_health: UNKNOWN } } }
I20250622 19:55:43.360394 3737 raft_consensus.cc:1238] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 5 FOLLOWER]: Rejecting Update request from peer 7e22adb60d65484c9844ee7d2108a396 for earlier term 4. Current term is 5. Ops: []
I20250622 19:55:43.361378 4189 consensus_queue.cc:1046] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2593, Last known committed idx: 2590, Time since last communication: 0.000s
I20250622 19:55:43.362720 2459 raft_consensus.cc:1238] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 5 FOLLOWER]: Rejecting Update request from peer 7e22adb60d65484c9844ee7d2108a396 for earlier term 4. Current term is 5. Ops: []
I20250622 19:55:43.363425 4188 consensus_queue.cc:1046] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2593, Last known committed idx: 2592, Time since last communication: 0.000s
I20250622 19:55:43.364061 4189 raft_consensus.cc:3053] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 4 LEADER]: Stepping down as leader of term 4
I20250622 19:55:43.364202 4189 raft_consensus.cc:738] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7e22adb60d65484c9844ee7d2108a396, State: Running, Role: LEADER
I20250622 19:55:43.364519 4189 consensus_queue.cc:260] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2592, Committed index: 2592, Last appended: 4.2595, Last appended by leader: 2595, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:43.365041 4189 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 4 FOLLOWER]: Advancing to term 5
W20250622 19:55:43.374588 3458 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.013s ago.
W20250622 19:55:43.387805 2417 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:43.431960 3697 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35340: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:43.446147 3073 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:43.458362 2417 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:43.471742 3697 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35340: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:43.491025 3074 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:43.505756 2417 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:43.523497 3697 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35340: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:43.545032 3071 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:43.562392 2417 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:43.580454 3697 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35340: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:43.606489 3073 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
W20250622 19:55:43.627631 2417 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
W20250622 19:55:43.653245 3697 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35340: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
W20250622 19:55:43.677223 3072 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
I20250622 19:55:43.692217 4189 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250622 19:55:43.692430 4189 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:43.693169 4189 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117), 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
I20250622 19:55:43.693866 3737 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "7e22adb60d65484c9844ee7d2108a396" candidate_term: 6 candidate_status { last_received { term: 4 index: 2595 } } ignore_live_leader: false dest_uuid: "9ece0d51c71b47a1865ba3b05920d999" is_pre_election: true
I20250622 19:55:43.694139 3737 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e22adb60d65484c9844ee7d2108a396 in term 5.
I20250622 19:55:43.694274 2459 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "7e22adb60d65484c9844ee7d2108a396" candidate_term: 6 candidate_status { last_received { term: 4 index: 2595 } } ignore_live_leader: false dest_uuid: "54555a8ecf924b339db8e8306cbc8bb4" is_pre_election: true
I20250622 19:55:43.694510 2459 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e22adb60d65484c9844ee7d2108a396 in term 5.
I20250622 19:55:43.695132 3048 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54555a8ecf924b339db8e8306cbc8bb4, 7e22adb60d65484c9844ee7d2108a396; no voters:
I20250622 19:55:43.695640 4189 raft_consensus.cc:2802] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250622 19:55:43.695778 4189 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250622 19:55:43.695892 4189 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 5 FOLLOWER]: Advancing to term 6
I20250622 19:55:43.701006 4189 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:43.702728 2459 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "7e22adb60d65484c9844ee7d2108a396" candidate_term: 6 candidate_status { last_received { term: 4 index: 2595 } } ignore_live_leader: false dest_uuid: "54555a8ecf924b339db8e8306cbc8bb4"
I20250622 19:55:43.702955 2459 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 5 FOLLOWER]: Advancing to term 6
W20250622 19:55:43.704133 2417 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48532: Illegal state: replica 54555a8ecf924b339db8e8306cbc8bb4 is not leader of this config: current role FOLLOWER
I20250622 19:55:43.705962 2459 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e22adb60d65484c9844ee7d2108a396 in term 6.
I20250622 19:55:43.706470 3048 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54555a8ecf924b339db8e8306cbc8bb4, 7e22adb60d65484c9844ee7d2108a396; no voters:
I20250622 19:55:43.706947 4188 raft_consensus.cc:2802] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 6 FOLLOWER]: Leader election won for term 6
I20250622 19:55:43.712152 4189 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 6 election: Requested vote from peers 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117), 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
I20250622 19:55:43.715479 3737 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "7e22adb60d65484c9844ee7d2108a396" candidate_term: 6 candidate_status { last_received { term: 4 index: 2595 } } ignore_live_leader: false dest_uuid: "9ece0d51c71b47a1865ba3b05920d999"
I20250622 19:55:43.715829 3737 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 5 FOLLOWER]: Advancing to term 6
I20250622 19:55:43.720486 3737 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e22adb60d65484c9844ee7d2108a396 in term 6.
I20250622 19:55:43.726361 4188 raft_consensus.cc:695] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 6 LEADER]: Becoming Leader. State: Replica: 7e22adb60d65484c9844ee7d2108a396, State: Running, Role: LEADER
I20250622 19:55:43.727320 4188 consensus_queue.cc:237] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2592, Committed index: 2592, Last appended: 4.2595, Last appended by leader: 2595, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:55:43.731716 2048 catalog_manager.cc:5582] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "7e22adb60d65484c9844ee7d2108a396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } health_report { overall_health: UNKNOWN } } }
W20250622 19:55:43.734331 3697 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:35340: Illegal state: replica 9ece0d51c71b47a1865ba3b05920d999 is not leader of this config: current role FOLLOWER
I20250622 19:55:43.767017 3737 raft_consensus.cc:1273] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 6 FOLLOWER]: Refusing update from remote peer 7e22adb60d65484c9844ee7d2108a396: Log matching property violated. Preceding OpId in replica: term: 3 index: 2591. Preceding OpId from leader: term: 6 index: 2597. (index mismatch)
I20250622 19:55:43.767974 2459 raft_consensus.cc:1273] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 6 FOLLOWER]: Refusing update from remote peer 7e22adb60d65484c9844ee7d2108a396: Log matching property violated. Preceding OpId in replica: term: 3 index: 2592. Preceding OpId from leader: term: 6 index: 2597. (index mismatch)
I20250622 19:55:43.768328 4201 consensus_queue.cc:1035] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2596, Last known committed idx: 2590, Time since last communication: 0.000s
I20250622 19:55:43.769467 4201 consensus_queue.cc:1035] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2596, Last known committed idx: 2592, Time since last communication: 0.000s
I20250622 19:55:43.797973 3458 mvcc.cc:204] Tried to move back new op lower bound from 7170548300859764736 to 7170548299068604416. Current Snapshot: MvccSnapshot[applied={T|T < 7170548299136466944}]
I20250622 19:55:43.826001 2308 tablet_service.cc:1423] Tablet server 07bc6e053fe04f6d8b0c0157ca553029 set to quiescing
I20250622 19:55:43.826272 2308 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:43.851828 4211 mvcc.cc:204] Tried to move back new op lower bound from 7170548300859764736 to 7170548299068604416. Current Snapshot: MvccSnapshot[applied={T|T < 7170548300859764736}]
I20250622 19:55:45.257277 2439 tablet_service.cc:1423] Tablet server 54555a8ecf924b339db8e8306cbc8bb4 set to quiescing
I20250622 19:55:45.257548 2439 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:45.486935 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 2114
I20250622 19:55:45.515362 2236 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:55:45.738034 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 2114
I20250622 19:55:45.800326 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.1:36357
--local_ip_for_outbound_sockets=127.1.248.1
--tserver_master_addrs=127.1.248.62:37973
--webserver_port=41843
--webserver_interface=127.1.248.1
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250622 19:55:46.532755 4241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:46.533126 4241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:46.533268 4241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:46.544931 4241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:46.545246 4241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.1
I20250622 19:55:46.561941 4241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.1:36357
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.248.1
--webserver_port=41843
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.1
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:46.565039 4241 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:46.567618 4241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:55:46.587517 4249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:46.592048 4250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:46.597121 4241 server_base.cc:1048] running on GCE node
W20250622 19:55:46.597524 4252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:46.598232 4241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:46.599294 4241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:46.603596 4241 hybrid_clock.cc:648] HybridClock initialized: now 1750622146604330 us; error 1163 us; skew 500 ppm
I20250622 19:55:46.604001 4241 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:46.670661 4241 webserver.cc:469] Webserver started at http://127.1.248.1:41843/ using document root <none> and password file <none>
I20250622 19:55:46.671772 4241 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:46.671922 4241 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:46.695326 4241 fs_manager.cc:714] Time spent opening directory manager: real 0.021s user 0.001s sys 0.011s
I20250622 19:55:46.717479 4258 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:46.719830 4241 fs_manager.cc:730] Time spent opening block manager: real 0.023s user 0.007s sys 0.009s
I20250622 19:55:46.720149 4241 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/wal
uuid: "2db5cde30e5a47fba8d434cc04849edc"
format_stamp: "Formatted at 2025-06-22 19:55:05 on dist-test-slave-t73b"
I20250622 19:55:46.721304 4241 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:46.807910 4241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:46.809059 4241 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:46.809405 4241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:46.816859 4241 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:46.863509 4241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:55:46.863634 4241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:46.863778 4241 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:55:46.863843 4241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:47.421924 4241 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.1:36357
I20250622 19:55:47.424136 4241 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data/info.pb
I20250622 19:55:47.425206 4374 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.1:36357 every 8 connection(s)
I20250622 19:55:47.428067 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 4241
I20250622 19:55:47.428277 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 2245
I20250622 19:55:47.491613 2367 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:55:47.529022 4375 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:47.529474 4375 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:47.530691 4375 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:47.532943 2048 ts_manager.cc:194] Re-registered known tserver with Master: 2db5cde30e5a47fba8d434cc04849edc (127.1.248.1:36357)
I20250622 19:55:47.533996 2048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.1:55225
I20250622 19:55:47.743984 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 2245
I20250622 19:55:47.797731 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.2:35999
--local_ip_for_outbound_sockets=127.1.248.2
--tserver_master_addrs=127.1.248.62:37973
--webserver_port=46661
--webserver_interface=127.1.248.2
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250622 19:55:48.506836 4383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:48.507463 4383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:48.507716 4383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:48.520006 4383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:48.520361 4383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.2
I20250622 19:55:48.535001 4383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.2:35999
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.248.2
--webserver_port=46661
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.2
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:48.537451 4383 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:48.537493 4375 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
I20250622 19:55:48.540081 4383 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:55:48.566598 4391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:48.569514 4390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:48.571130 4393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:48.573493 4383 server_base.cc:1048] running on GCE node
I20250622 19:55:48.574399 4383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:48.585791 4383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:48.596920 4383 hybrid_clock.cc:648] HybridClock initialized: now 1750622148593675 us; error 3213 us; skew 500 ppm
I20250622 19:55:48.597522 4383 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:48.616714 4383 webserver.cc:469] Webserver started at http://127.1.248.2:46661/ using document root <none> and password file <none>
I20250622 19:55:48.623716 4383 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:48.623895 4383 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:48.640555 4383 fs_manager.cc:714] Time spent opening directory manager: real 0.014s user 0.009s sys 0.000s
I20250622 19:55:48.650569 4399 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:48.660307 4383 fs_manager.cc:730] Time spent opening block manager: real 0.018s user 0.005s sys 0.001s
I20250622 19:55:48.660740 4383 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/wal
uuid: "07bc6e053fe04f6d8b0c0157ca553029"
format_stamp: "Formatted at 2025-06-22 19:55:06 on dist-test-slave-t73b"
I20250622 19:55:48.661926 4383 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:48.731252 4383 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:48.732488 4383 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:48.732863 4383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:48.757885 4383 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:48.763255 4383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:55:48.763473 4383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250622 19:55:48.763692 4383 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:55:48.763803 4383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:49.601109 4383 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.2:35999
I20250622 19:55:49.601477 4516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.2:35999 every 8 connection(s)
I20250622 19:55:49.603137 4383 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data/info.pb
I20250622 19:55:49.607204 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 4383
I20250622 19:55:49.607384 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 2376
I20250622 19:55:49.662650 2498 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:55:49.724448 4517 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:49.724933 4517 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:49.726150 4517 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:49.728152 2048 ts_manager.cc:194] Re-registered known tserver with Master: 07bc6e053fe04f6d8b0c0157ca553029 (127.1.248.2:35999)
I20250622 19:55:49.729278 2048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.2:41031
I20250622 19:55:50.074863 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 2376
I20250622 19:55:50.166993 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.3:41117
--local_ip_for_outbound_sockets=127.1.248.3
--tserver_master_addrs=127.1.248.62:37973
--webserver_port=45773
--webserver_interface=127.1.248.3
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250622 19:55:50.166090 3048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.248.3:41117 (error 108)
W20250622 19:55:50.220364 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.3:41117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250622 19:55:50.729065 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.3:41117: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250622 19:55:50.733009 4517 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
W20250622 19:55:50.798964 4526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:55:50.799510 4526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:55:50.799685 4526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:55:50.812345 4526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:55:50.812686 4526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.3
I20250622 19:55:50.828001 4526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.3:41117
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.248.3
--webserver_port=45773
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.3
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:55:50.830418 4526 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:55:50.832962 4526 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:55:50.891178 4534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:55:50.891178 4535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:50.949381 4526 server_base.cc:1048] running on GCE node
W20250622 19:55:50.971212 4537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:55:50.982007 4526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:55:50.984243 4526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:55:50.989946 4526 hybrid_clock.cc:648] HybridClock initialized: now 1750622150989683 us; error 239 us; skew 500 ppm
I20250622 19:55:50.990463 4526 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:55:51.029089 4526 webserver.cc:469] Webserver started at http://127.1.248.3:45773/ using document root <none> and password file <none>
I20250622 19:55:51.030543 4526 fs_manager.cc:362] Metadata directory not provided
I20250622 19:55:51.030830 4526 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:55:51.076571 4526 fs_manager.cc:714] Time spent opening directory manager: real 0.038s user 0.007s sys 0.004s
I20250622 19:55:51.099896 4544 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:55:51.102669 4526 fs_manager.cc:730] Time spent opening block manager: real 0.018s user 0.005s sys 0.002s
I20250622 19:55:51.102977 4526 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/wal
uuid: "54555a8ecf924b339db8e8306cbc8bb4"
format_stamp: "Formatted at 2025-06-22 19:55:06 on dist-test-slave-t73b"
I20250622 19:55:51.104292 4526 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:55:51.228205 4526 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:55:51.229369 4526 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:55:51.229740 4526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:55:51.234658 4526 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:55:51.240218 4551 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250622 19:55:51.249250 4526 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 19:55:51.249364 4526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.001s sys 0.000s
I20250622 19:55:51.249498 4526 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 19:55:51.255349 4526 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 19:55:51.255492 4526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.004s sys 0.000s
I20250622 19:55:51.261449 4551 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: Bootstrap starting.
W20250622 19:55:51.295128 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.3:41117: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250622 19:55:51.835376 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.3:41117: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250622 19:55:52.000973 4659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.3:41117 every 8 connection(s)
I20250622 19:55:52.001319 4526 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.3:41117
I20250622 19:55:52.003502 4526 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data/info.pb
I20250622 19:55:52.010991 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 4526
I20250622 19:55:52.045521 4660 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:55:52.045955 4660 heartbeater.cc:461] Registering TS with master...
I20250622 19:55:52.047170 4660 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:55:52.050385 2048 ts_manager.cc:194] Re-registered known tserver with Master: 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117)
I20250622 19:55:52.051931 2048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.3:52319
W20250622 19:55:52.384995 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
W20250622 19:55:52.956816 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
I20250622 19:55:53.055878 4660 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
I20250622 19:55:53.313158 4451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:53.335959 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:53.361496 4551 log.cc:826] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: Log is configured to *not* fsync() on all Append() calls
I20250622 19:55:53.396231 2570 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:53.433576 4594 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:53.455181 3225 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:53.484500 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:53.499531 4309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:55:53.524590 3094 tablet_service.cc:1430] Tablet server has 1 leaders and 3 scanners
W20250622 19:55:53.623838 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250622 19:55:53.624706 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 19:55:54.117821 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20250622 19:55:54.628038 4188 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [LEADER]: Peer 54555a8ecf924b339db8e8306cbc8bb4 is lagging by at least 10 ops behind the committed index
W20250622 19:55:54.679288 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20250622 19:55:55.238055 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250622 19:55:55.705067 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
W20250622 19:55:56.222680 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
W20250622 19:55:56.803040 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
W20250622 19:55:57.297178 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
I20250622 19:55:57.694038 4720 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [LEADER]: Peer 54555a8ecf924b339db8e8306cbc8bb4 is lagging by at least 355 ops behind the committed index [suppressed 28 similar messages]
W20250622 19:55:57.825420 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
W20250622 19:55:58.332914 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
I20250622 19:55:58.389256 4551 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: Bootstrap replaying log segment 1/1 (1.64M/5.39M this segment, stats: ops{read=957 overwritten=0 applied=956 ignored=0} inserts{seen=47700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250622 19:55:58.852061 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250622 19:55:59.333971 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
W20250622 19:55:59.901391 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
W20250622 19:56:00.468422 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
I20250622 19:56:00.668493 4451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:00.696769 4730 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [LEADER]: Peer 54555a8ecf924b339db8e8306cbc8bb4 is lagging by at least 630 ops behind the committed index [suppressed 27 similar messages]
I20250622 19:56:00.706908 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:00.724601 2570 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:00.756438 4309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:00.840739 4594 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:00.847223 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:00.865689 3225 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:00.873167 3094 tablet_service.cc:1430] Tablet server has 1 leaders and 3 scanners
I20250622 19:56:00.875845 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 19:56:01.077603 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
W20250622 19:56:01.607903 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20250622 19:56:02.110458 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
W20250622 19:56:02.667562 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
W20250622 19:56:03.178385 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
I20250622 19:56:03.389415 4551 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: Bootstrap replaying log segment 1/1 (3.20M/5.39M this segment, stats: ops{read=1858 overwritten=0 applied=1857 ignored=0} inserts{seen=92750 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250622 19:56:03.714331 4730 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [LEADER]: Peer 54555a8ecf924b339db8e8306cbc8bb4 is lagging by at least 938 ops behind the committed index [suppressed 27 similar messages]
W20250622 19:56:03.734508 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
W20250622 19:56:04.257668 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
W20250622 19:56:04.733433 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
W20250622 19:56:05.246542 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
W20250622 19:56:05.767836 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
W20250622 19:56:06.437018 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
I20250622 19:56:06.783663 4730 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [LEADER]: Peer 54555a8ecf924b339db8e8306cbc8bb4 is lagging by at least 1239 ops behind the committed index [suppressed 28 similar messages]
W20250622 19:56:06.918463 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 156: this message will repeat every 5th retry.
W20250622 19:56:07.443480 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 161: this message will repeat every 5th retry.
I20250622 19:56:07.747936 4451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:07.761350 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:07.811743 4309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:07.827764 4594 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:07.835779 2570 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:07.933552 3225 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:08.002919 3094 tablet_service.cc:1430] Tablet server has 1 leaders and 3 scanners
I20250622 19:56:08.022637 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:08.034713 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 19:56:08.148921 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 166: this message will repeat every 5th retry.
I20250622 19:56:08.394057 4551 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: Bootstrap replaying log segment 1/1 (5.14M/5.39M this segment, stats: ops{read=2986 overwritten=0 applied=2986 ignored=0} inserts{seen=149100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250622 19:56:08.655193 3048 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 171: this message will repeat every 5th retry.
I20250622 19:56:09.129978 4551 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3136 overwritten=0 applied=3135 ignored=0} inserts{seen=156550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250622 19:56:09.131053 4551 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: Bootstrap complete.
I20250622 19:56:09.133402 4551 ts_tablet_manager.cc:1397] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: Time spent bootstrapping tablet: real 17.872s user 12.992s sys 0.604s
I20250622 19:56:09.138166 4551 raft_consensus.cc:357] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 6 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:56:09.150004 4551 raft_consensus.cc:738] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54555a8ecf924b339db8e8306cbc8bb4, State: Initialized, Role: FOLLOWER
I20250622 19:56:09.151080 4551 consensus_queue.cc:260] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3135, Last appended: 6.3136, Last appended by leader: 3136, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:56:09.197198 4551 ts_tablet_manager.cc:1428] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4: Time spent starting tablet: real 0.064s user 0.025s sys 0.035s
I20250622 19:56:09.625412 4839 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 7e22adb60d65484c9844ee7d2108a396)
I20250622 19:56:09.625710 4839 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:56:09.627616 4839 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693), 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
I20250622 19:56:09.651031 3737 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "54555a8ecf924b339db8e8306cbc8bb4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3561 } } ignore_live_leader: false dest_uuid: "9ece0d51c71b47a1865ba3b05920d999" is_pre_election: true
I20250622 19:56:09.694547 3113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "54555a8ecf924b339db8e8306cbc8bb4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3561 } } ignore_live_leader: false dest_uuid: "7e22adb60d65484c9844ee7d2108a396" is_pre_election: true
I20250622 19:56:09.695523 4546 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54555a8ecf924b339db8e8306cbc8bb4; no voters: 7e22adb60d65484c9844ee7d2108a396, 9ece0d51c71b47a1865ba3b05920d999
I20250622 19:56:10.022362 4839 raft_consensus.cc:2747] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250622 19:56:10.029655 4850 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [LEADER]: Peer 54555a8ecf924b339db8e8306cbc8bb4 is lagging by at least 668 ops behind the committed index [suppressed 22 similar messages]
I20250622 19:56:14.816754 4451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:14.843899 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:14.867311 2570 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:14.889545 4309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:14.916635 4594 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:14.975788 3094 tablet_service.cc:1430] Tablet server has 1 leaders and 3 scanners
I20250622 19:56:14.988675 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:15.011807 3225 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:15.045634 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:17.342964 2054 ts_manager.cc:284] Unset tserver state for 07bc6e053fe04f6d8b0c0157ca553029 from MAINTENANCE_MODE
I20250622 19:56:17.347126 2048 ts_manager.cc:284] Unset tserver state for 2db5cde30e5a47fba8d434cc04849edc from MAINTENANCE_MODE
I20250622 19:56:17.350322 2635 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:17.350821 3642 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:17.450452 2050 ts_manager.cc:284] Unset tserver state for 54555a8ecf924b339db8e8306cbc8bb4 from MAINTENANCE_MODE
I20250622 19:56:17.534451 4660 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:17.618233 4375 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:17.660856 3159 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:17.714955 3924 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:17.773350 3290 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:17.805409 4517 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:17.905448 3784 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:18.359575 2635 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:18.360900 3642 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:20.024371 2016 maintenance_mode-itest.cc:745] Restarting batch of 1 tservers: 144888b25afe4c04b4de5350bddf224e
I20250622 19:56:21.080938 2050 ts_manager.cc:295] Set tserver state for 144888b25afe4c04b4de5350bddf224e to MAINTENANCE_MODE
I20250622 19:56:23.713279 2570 tablet_service.cc:1423] Tablet server 144888b25afe4c04b4de5350bddf224e set to quiescing
I20250622 19:56:23.713541 2570 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:23.946295 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 2507
I20250622 19:56:23.990669 2629 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:56:24.275791 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 2507
I20250622 19:56:24.308269 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.4:40193
--local_ip_for_outbound_sockets=127.1.248.4
--tserver_master_addrs=127.1.248.62:37973
--webserver_port=33405
--webserver_interface=127.1.248.4
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250622 19:56:24.963940 5012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:56:24.964594 5012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:56:24.964866 5012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:56:24.977562 5012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:56:24.977959 5012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.4
I20250622 19:56:24.992849 5012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.4:40193
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.1.248.4
--webserver_port=33405
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.4
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:56:24.995071 5012 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:56:24.997804 5012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:56:25.029453 5019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:56:25.056824 5020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:56:25.063787 5024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:56:25.064388 5012 server_base.cc:1048] running on GCE node
I20250622 19:56:25.065389 5012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:56:25.094617 5012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:56:25.096031 5012 hybrid_clock.cc:648] HybridClock initialized: now 1750622185095969 us; error 80 us; skew 500 ppm
I20250622 19:56:25.096385 5012 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:56:25.108131 5012 webserver.cc:469] Webserver started at http://127.1.248.4:33405/ using document root <none> and password file <none>
I20250622 19:56:25.111675 5012 fs_manager.cc:362] Metadata directory not provided
I20250622 19:56:25.111912 5012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:56:25.125607 5012 fs_manager.cc:714] Time spent opening directory manager: real 0.011s user 0.002s sys 0.002s
I20250622 19:56:25.151751 5029 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:56:25.156515 5012 fs_manager.cc:730] Time spent opening block manager: real 0.025s user 0.002s sys 0.001s
I20250622 19:56:25.156733 5012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/wal
uuid: "144888b25afe4c04b4de5350bddf224e"
format_stamp: "Formatted at 2025-06-22 19:55:06 on dist-test-slave-t73b"
I20250622 19:56:25.157788 5012 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:56:25.416281 5012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:56:25.417873 5012 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:56:25.418376 5012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:56:25.431071 5012 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:56:25.445225 5012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:56:25.445437 5012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 19:56:25.445672 5012 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:56:25.445809 5012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 19:56:26.033556 5012 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.4:40193
I20250622 19:56:26.033588 5142 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.4:40193 every 8 connection(s)
I20250622 19:56:26.035571 5012 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data/info.pb
I20250622 19:56:26.043687 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 5012
I20250622 19:56:26.069417 5143 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:56:26.069837 5143 heartbeater.cc:461] Registering TS with master...
I20250622 19:56:26.070894 5143 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:26.072502 2053 ts_manager.cc:194] Re-registered known tserver with Master: 144888b25afe4c04b4de5350bddf224e (127.1.248.4:40193)
I20250622 19:56:26.073464 2053 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.4:43895
I20250622 19:56:27.076622 5143 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
I20250622 19:56:27.276293 5078 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:27.304872 4309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:27.305043 4451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:27.330824 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:27.365494 3225 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:27.445463 4594 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:27.445617 3094 tablet_service.cc:1430] Tablet server has 1 leaders and 3 scanners
I20250622 19:56:27.590634 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:27.607438 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:28.982862 2053 ts_manager.cc:284] Unset tserver state for 144888b25afe4c04b4de5350bddf224e from MAINTENANCE_MODE
I20250622 19:56:28.984107 3784 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:29.085341 5143 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:29.393471 3642 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:29.568492 4660 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:29.647723 4375 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:29.754422 3924 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:29.801494 3290 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:29.808039 3159 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:56:29.850109 4517 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
W20250622 19:56:30.330475 3327 meta_cache.cc:1261] Time spent looking up entry by key: real 0.055s user 0.001s sys 0.000s
I20250622 19:56:30.616037 2016 maintenance_mode-itest.cc:745] Restarting batch of 2 tservers: 7520035c9f604041a00535b523c45c1b,7e22adb60d65484c9844ee7d2108a396
I20250622 19:56:31.831625 2050 ts_manager.cc:295] Set tserver state for 7e22adb60d65484c9844ee7d2108a396 to MAINTENANCE_MODE
I20250622 19:56:31.969206 2054 ts_manager.cc:295] Set tserver state for 7520035c9f604041a00535b523c45c1b to MAINTENANCE_MODE
I20250622 19:56:33.842775 3225 tablet_service.cc:1423] Tablet server 7520035c9f604041a00535b523c45c1b set to quiescing
I20250622 19:56:33.843071 3225 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 19:56:34.049503 3300 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.248.8:39693, user_credentials={real_user=slave}} blocked reactor thread for 49483.6us
I20250622 19:56:34.687189 3094 tablet_service.cc:1423] Tablet server 7e22adb60d65484c9844ee7d2108a396 set to quiescing
I20250622 19:56:34.687458 3094 tablet_service.cc:1430] Tablet server has 1 leaders and 3 scanners
I20250622 19:56:34.724833 5257 raft_consensus.cc:991] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: : Instructing follower 54555a8ecf924b339db8e8306cbc8bb4 to start an election
I20250622 19:56:34.725045 5257 raft_consensus.cc:1079] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 6 LEADER]: Signalling peer 54555a8ecf924b339db8e8306cbc8bb4 to start an election
I20250622 19:56:34.729899 4614 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d"
dest_uuid: "54555a8ecf924b339db8e8306cbc8bb4"
from {username='slave'} at 127.1.248.8:52675
I20250622 19:56:34.730182 4614 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 19:56:34.730322 4614 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 6 FOLLOWER]: Advancing to term 7
I20250622 19:56:34.730491 5277 raft_consensus.cc:991] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: : Instructing follower 54555a8ecf924b339db8e8306cbc8bb4 to start an election
I20250622 19:56:34.730635 5277 raft_consensus.cc:1079] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 6 LEADER]: Signalling peer 54555a8ecf924b339db8e8306cbc8bb4 to start an election
I20250622 19:56:34.731284 4612 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d"
dest_uuid: "54555a8ecf924b339db8e8306cbc8bb4"
from {username='slave'} at 127.1.248.8:52675
I20250622 19:56:34.736603 4614 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:56:34.737093 4612 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250622 19:56:34.737208 4612 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 7 FOLLOWER]: Advancing to term 8
I20250622 19:56:34.741397 4612 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:56:34.743244 4612 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [CANDIDATE]: Term 8 election: Requested vote from peers 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693), 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
I20250622 19:56:34.744191 4614 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [CANDIDATE]: Term 7 election: Requested vote from peers 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693), 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
I20250622 19:56:34.744124 3738 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "54555a8ecf924b339db8e8306cbc8bb4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7069 } } ignore_live_leader: true dest_uuid: "9ece0d51c71b47a1865ba3b05920d999"
I20250622 19:56:34.744313 3738 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 6 FOLLOWER]: Advancing to term 8
I20250622 19:56:34.744690 3736 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "54555a8ecf924b339db8e8306cbc8bb4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7069 } } ignore_live_leader: true dest_uuid: "9ece0d51c71b47a1865ba3b05920d999"
I20250622 19:56:34.747164 3113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "54555a8ecf924b339db8e8306cbc8bb4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7069 } } ignore_live_leader: true dest_uuid: "7e22adb60d65484c9844ee7d2108a396"
I20250622 19:56:34.747489 3113 raft_consensus.cc:3053] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 6 LEADER]: Stepping down as leader of term 6
I20250622 19:56:34.747619 3113 raft_consensus.cc:738] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7e22adb60d65484c9844ee7d2108a396, State: Running, Role: LEADER
I20250622 19:56:34.748025 3113 consensus_queue.cc:260] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7069, Committed index: 7069, Last appended: 6.7069, Last appended by leader: 7069, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:56:34.748329 3113 logging.cc:424] LogThrottler TrackedPeer Lag: suppressed but not reported on 1 messages since previous log ~24 seconds ago
I20250622 19:56:34.748660 3113 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 6 FOLLOWER]: Advancing to term 8
I20250622 19:56:34.747164 3114 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "54555a8ecf924b339db8e8306cbc8bb4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7069 } } ignore_live_leader: true dest_uuid: "7e22adb60d65484c9844ee7d2108a396"
I20250622 19:56:34.751011 4546 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54555a8ecf924b339db8e8306cbc8bb4; no voters: 7e22adb60d65484c9844ee7d2108a396, 9ece0d51c71b47a1865ba3b05920d999
I20250622 19:56:34.751955 3738 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54555a8ecf924b339db8e8306cbc8bb4 in term 8.
W20250622 19:56:34.752518 3458 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.028s ago.
W20250622 19:56:34.753976 3458 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.029s ago.
I20250622 19:56:34.754407 3113 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54555a8ecf924b339db8e8306cbc8bb4 in term 8.
I20250622 19:56:34.754446 4545 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54555a8ecf924b339db8e8306cbc8bb4, 9ece0d51c71b47a1865ba3b05920d999; no voters:
I20250622 19:56:34.763557 5291 raft_consensus.cc:2747] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250622 19:56:34.763928 5291 raft_consensus.cc:2802] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 8 FOLLOWER]: Leader election won for term 8
W20250622 19:56:34.764025 3072 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:37314: Illegal state: replica 7e22adb60d65484c9844ee7d2108a396 is not leader of this config: current role FOLLOWER
I20250622 19:56:34.764258 5291 raft_consensus.cc:695] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [term 8 LEADER]: Becoming Leader. State: Replica: 54555a8ecf924b339db8e8306cbc8bb4, State: Running, Role: LEADER
I20250622 19:56:34.765071 5291 consensus_queue.cc:237] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7069, Committed index: 7069, Last appended: 6.7069, Last appended by leader: 7069, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:56:34.774813 2048 catalog_manager.cc:5582] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 reported cstate change: term changed from 6 to 8, leader changed from 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8) to 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3). New cstate: current_term: 8 leader_uuid: "54555a8ecf924b339db8e8306cbc8bb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } health_report { overall_health: UNKNOWN } } }
I20250622 19:56:34.812108 3738 raft_consensus.cc:1273] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 8 FOLLOWER]: Refusing update from remote peer 54555a8ecf924b339db8e8306cbc8bb4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7068. Preceding OpId from leader: term: 8 index: 7072. (index mismatch)
I20250622 19:56:34.812223 3113 raft_consensus.cc:1273] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 8 FOLLOWER]: Refusing update from remote peer 54555a8ecf924b339db8e8306cbc8bb4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7069. Preceding OpId from leader: term: 8 index: 7072. (index mismatch)
I20250622 19:56:34.813207 5290 consensus_queue.cc:1035] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7070, Last known committed idx: 7069, Time since last communication: 0.000s
I20250622 19:56:34.814772 5290 consensus_queue.cc:1035] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7070, Last known committed idx: 7067, Time since last communication: 0.000s
I20250622 19:56:34.852138 4838 mvcc.cc:204] Tried to move back new op lower bound from 7170548510084771840 to 7170548509767159808. Current Snapshot: MvccSnapshot[applied={T|T < 7170548510084771840}]
I20250622 19:56:38.076495 3094 tablet_service.cc:1423] Tablet server 7e22adb60d65484c9844ee7d2108a396 set to quiescing
I20250622 19:56:38.076771 3094 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250622 19:56:41.538934 3094 tablet_service.cc:1423] Tablet server 7e22adb60d65484c9844ee7d2108a396 set to quiescing
I20250622 19:56:41.539188 3094 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250622 19:56:44.977212 3094 tablet_service.cc:1423] Tablet server 7e22adb60d65484c9844ee7d2108a396 set to quiescing
I20250622 19:56:44.977461 3094 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250622 19:56:47.239028 3094 tablet_service.cc:1423] Tablet server 7e22adb60d65484c9844ee7d2108a396 set to quiescing
I20250622 19:56:47.239248 3094 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250622 19:56:50.812237 3094 tablet_service.cc:1423] Tablet server 7e22adb60d65484c9844ee7d2108a396 set to quiescing
I20250622 19:56:50.812525 3094 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250622 19:56:53.476078 3094 tablet_service.cc:1423] Tablet server 7e22adb60d65484c9844ee7d2108a396 set to quiescing
I20250622 19:56:53.476328 3094 tablet_service.cc:1430] Tablet server has 0 leaders and 1 scanners
I20250622 19:56:57.200098 3094 tablet_service.cc:1423] Tablet server 7e22adb60d65484c9844ee7d2108a396 set to quiescing
I20250622 19:56:57.200371 3094 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:56:59.096884 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 3162
I20250622 19:56:59.110579 3285 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:56:59.572804 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 3162
I20250622 19:56:59.603071 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.9:44015
--local_ip_for_outbound_sockets=127.1.248.9
--tserver_master_addrs=127.1.248.62:37973
--webserver_port=38409
--webserver_interface=127.1.248.9
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250622 19:57:00.279455 5465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:57:00.279946 5465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:57:00.280382 5465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:57:00.293099 5465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:57:00.293428 5465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.9
I20250622 19:57:00.310499 5465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.9:44015
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data/info.pb
--webserver_interface=127.1.248.9
--webserver_port=38409
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.9
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:57:00.313158 5465 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:57:00.316131 5465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:57:00.378338 5471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:57:00.395443 5470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:57:00.483776 5473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:57:00.493171 5465 server_base.cc:1048] running on GCE node
I20250622 19:57:00.494603 5465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:57:00.522482 5465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:57:00.528205 5465 hybrid_clock.cc:648] HybridClock initialized: now 1750622220527653 us; error 528 us; skew 500 ppm
I20250622 19:57:00.528811 5465 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:57:00.578989 5465 webserver.cc:469] Webserver started at http://127.1.248.9:38409/ using document root <none> and password file <none>
I20250622 19:57:00.580274 5465 fs_manager.cc:362] Metadata directory not provided
I20250622 19:57:00.580497 5465 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:57:00.620486 5465 fs_manager.cc:714] Time spent opening directory manager: real 0.037s user 0.013s sys 0.005s
I20250622 19:57:00.629630 5480 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:57:00.633512 5465 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.006s sys 0.000s
I20250622 19:57:00.633818 5465 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/wal
uuid: "7520035c9f604041a00535b523c45c1b"
format_stamp: "Formatted at 2025-06-22 19:55:09 on dist-test-slave-t73b"
I20250622 19:57:00.634944 5465 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:57:00.723701 5465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:57:00.734056 5465 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:57:00.734628 5465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:57:00.786585 5465 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:57:00.801270 5465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250622 19:57:00.801436 5465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250622 19:57:00.801584 5465 ts_tablet_manager.cc:610] Registered 0 tablets
I20250622 19:57:00.801673 5465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250622 19:57:02.262018 5600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.9:44015 every 8 connection(s)
I20250622 19:57:02.262008 5465 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.9:44015
I20250622 19:57:02.266894 5465 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data/info.pb
I20250622 19:57:02.269857 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 5465
I20250622 19:57:02.270080 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 3031
I20250622 19:57:02.413332 3153 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:57:02.435142 5601 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:57:02.435482 5601 heartbeater.cc:461] Registering TS with master...
I20250622 19:57:02.436529 5601 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:57:02.438927 2054 ts_manager.cc:194] Re-registered known tserver with Master: 7520035c9f604041a00535b523c45c1b (127.1.248.9:44015)
I20250622 19:57:02.440059 2054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.9:51935
I20250622 19:57:02.986711 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 3031
W20250622 19:57:03.104530 4546 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.248.8:39693 (error 108)
I20250622 19:57:03.119024 2016 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
/tmp/dist-test-taskmkqhcg/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.248.8:39693
--local_ip_for_outbound_sockets=127.1.248.8
--tserver_master_addrs=127.1.248.62:37973
--webserver_port=41309
--webserver_interface=127.1.248.8
--builtin_ntp_servers=127.1.248.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250622 19:57:03.154376 4546 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.248.8:39693, user_credentials={real_user=slave}} blocked reactor thread for 49890.7us
W20250622 19:57:03.171867 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.8:39693: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 19:57:03.445623 5601 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
W20250622 19:57:03.678880 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.8:39693: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250622 19:57:03.918928 5613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:57:03.919385 5613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250622 19:57:03.919559 5613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:57:03.933195 5613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250622 19:57:03.933523 5613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.248.8
I20250622 19:57:03.948336 5613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.248.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.248.8:39693
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data/info.pb
--webserver_interface=127.1.248.8
--webserver_port=41309
--tserver_master_addrs=127.1.248.62:37973
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.248.8
--log_dir=/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1550f239ea6e50bb8f84a51255fe1b8accb536aa
build type FASTDEBUG
built by None at 22 Jun 2025 19:43:22 UTC on 24a791456cd2
build id 6720
ASAN enabled
I20250622 19:57:03.951108 5613 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250622 19:57:03.953748 5613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250622 19:57:04.072763 5619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:57:04.124322 5613 server_base.cc:1048] running on GCE node
W20250622 19:57:04.125330 5622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250622 19:57:04.130787 5620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250622 19:57:04.131789 5613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250622 19:57:04.134407 5613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250622 19:57:04.137485 5613 hybrid_clock.cc:648] HybridClock initialized: now 1750622224137325 us; error 124 us; skew 500 ppm
I20250622 19:57:04.137926 5613 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250622 19:57:04.142114 5613 webserver.cc:469] Webserver started at http://127.1.248.8:41309/ using document root <none> and password file <none>
I20250622 19:57:04.143405 5613 fs_manager.cc:362] Metadata directory not provided
I20250622 19:57:04.143635 5613 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250622 19:57:04.158064 5613 fs_manager.cc:714] Time spent opening directory manager: real 0.011s user 0.004s sys 0.001s
I20250622 19:57:04.162176 5628 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250622 19:57:04.183374 5613 fs_manager.cc:730] Time spent opening block manager: real 0.023s user 0.001s sys 0.001s
I20250622 19:57:04.183820 5613 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data,/tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/wal
uuid: "7e22adb60d65484c9844ee7d2108a396"
format_stamp: "Formatted at 2025-06-22 19:55:08 on dist-test-slave-t73b"
I20250622 19:57:04.186748 5613 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/wal
metadata directory: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/wal
1 data directories: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250622 19:57:04.244763 5613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250622 19:57:04.245908 5613 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250622 19:57:04.246282 5613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250622 19:57:04.250486 5613 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250622 19:57:04.253772 5635 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20250622 19:57:04.268922 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.8:39693: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250622 19:57:04.290135 5613 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250622 19:57:04.290364 5613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.037s user 0.000s sys 0.000s
I20250622 19:57:04.290699 5613 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250622 19:57:04.296702 5613 ts_tablet_manager.cc:610] Registered 1 tablets
I20250622 19:57:04.296893 5613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.004s sys 0.000s
I20250622 19:57:04.298749 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap starting.
I20250622 19:57:04.541565 5635 log.cc:826] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Log is configured to *not* fsync() on all Append() calls
W20250622 19:57:04.811136 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.8:39693: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250622 19:57:05.291389 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.8:39693: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250622 19:57:05.373011 2044 maintenance_manager.cc:419] P e103efdfebd54e69b47694a7953f484c: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033396
I20250622 19:57:05.599154 2043 maintenance_manager.cc:643] P e103efdfebd54e69b47694a7953f484c: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.225s user 0.036s sys 0.004s Metrics: {"bytes_written":7320,"cfile_init":1,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":21875,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":14,"thread_start_us":14793,"threads_started":1,"wal-append.queue_time_us":14994}
I20250622 19:57:05.603621 2044 maintenance_manager.cc:419] P e103efdfebd54e69b47694a7953f484c: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 717 bytes on disk
I20250622 19:57:05.612468 2043 maintenance_manager.cc:643] P e103efdfebd54e69b47694a7953f484c: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4}
W20250622 19:57:05.789384 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.8:39693: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20250622 19:57:06.197209 5613 rpc_server.cc:307] RPC server started. Bound to: 127.1.248.8:39693
I20250622 19:57:06.199936 5613 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data/info.pb
I20250622 19:57:06.203672 2016 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu as pid 5613
I20250622 19:57:06.206938 5747 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.248.8:39693 every 8 connection(s)
I20250622 19:57:06.250331 5748 heartbeater.cc:344] Connected to a master server at 127.1.248.62:37973
I20250622 19:57:06.250773 5748 heartbeater.cc:461] Registering TS with master...
I20250622 19:57:06.251973 5748 heartbeater.cc:507] Master 127.1.248.62:37973 requested a full tablet report, sending...
I20250622 19:57:06.255364 2054 ts_manager.cc:194] Re-registered known tserver with Master: 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693)
I20250622 19:57:06.256916 2054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.248.8:35627
W20250622 19:57:06.414040 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250622 19:57:06.875874 5612 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 9 ops behind the committed index
W20250622 19:57:06.977552 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20250622 19:57:07.267102 5748 heartbeater.cc:499] Master 127.1.248.62:37973 was elected leader, sending a full tablet report...
W20250622 19:57:07.617810 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
I20250622 19:57:08.120318 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:08.166532 4309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:08.205782 4594 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 19:57:08.221998 5533 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:08.232010 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:08.234712 5678 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:08.256387 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250622 19:57:08.303414 5078 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:08.332906 4451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 19:57:08.394214 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250622 19:57:09.003011 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
I20250622 19:57:09.556037 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap replaying log segment 1/3 (1.22M/8.00M this segment, stats: ops{read=732 overwritten=0 applied=731 ignored=0} inserts{seen=36450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250622 19:57:09.628296 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
I20250622 19:57:09.884235 5462 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 228 ops behind the committed index [suppressed 22 similar messages]
W20250622 19:57:10.173767 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
W20250622 19:57:10.763461 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
W20250622 19:57:11.289983 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
W20250622 19:57:11.783951 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
W20250622 19:57:12.296618 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250622 19:57:12.809952 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
I20250622 19:57:12.892266 5610 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 522 ops behind the committed index [suppressed 27 similar messages]
W20250622 19:57:13.318413 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
W20250622 19:57:13.906419 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
W20250622 19:57:14.452067 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
I20250622 19:57:14.558382 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap replaying log segment 1/3 (2.85M/8.00M this segment, stats: ops{read=1698 overwritten=0 applied=1698 ignored=0} inserts{seen=84800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250622 19:57:15.013722 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20250622 19:57:15.586488 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
I20250622 19:57:15.932670 5831 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 786 ops behind the committed index [suppressed 27 similar messages]
W20250622 19:57:16.061566 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
I20250622 19:57:16.078514 4451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:16.110051 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:16.171223 4309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:16.239393 5078 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:16.363855 4594 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 19:57:16.371878 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250622 19:57:16.372224 5533 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:16.376709 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:16.435062 5678 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 19:57:16.692202 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
W20250622 19:57:17.205647 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
W20250622 19:57:17.790702 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
W20250622 19:57:18.392947 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
W20250622 19:57:18.867753 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
I20250622 19:57:19.025594 5831 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 1076 ops behind the committed index [suppressed 27 similar messages]
W20250622 19:57:19.333364 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
I20250622 19:57:19.561156 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap replaying log segment 1/3 (4.32M/8.00M this segment, stats: ops{read=2570 overwritten=0 applied=2570 ignored=0} inserts{seen=128400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250622 19:57:19.901096 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
W20250622 19:57:20.389710 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 156: this message will repeat every 5th retry.
W20250622 19:57:20.942902 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 161: this message will repeat every 5th retry.
W20250622 19:57:21.524364 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 166: this message will repeat every 5th retry.
W20250622 19:57:21.923705 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 171: this message will repeat every 5th retry.
I20250622 19:57:22.117051 5884 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 1370 ops behind the committed index [suppressed 29 similar messages]
W20250622 19:57:22.601642 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 176: this message will repeat every 5th retry.
W20250622 19:57:23.052595 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 181: this message will repeat every 5th retry.
W20250622 19:57:23.514458 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 186: this message will repeat every 5th retry.
I20250622 19:57:23.750213 4451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:23.863047 4309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:23.880647 5078 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:23.882910 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:23.947755 4594 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 19:57:23.965720 5533 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:23.982832 5678 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:24.008450 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
W20250622 19:57:24.078070 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 191: this message will repeat every 5th retry.
I20250622 19:57:24.106392 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:24.562382 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap replaying log segment 1/3 (5.85M/8.00M this segment, stats: ops{read=3457 overwritten=0 applied=3456 ignored=0} inserts{seen=172600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250622 19:57:24.619189 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 196: this message will repeat every 5th retry.
W20250622 19:57:25.144657 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 201: this message will repeat every 5th retry.
I20250622 19:57:25.234619 5829 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 1634 ops behind the committed index [suppressed 28 similar messages]
W20250622 19:57:25.672667 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 206: this message will repeat every 5th retry.
W20250622 19:57:26.234503 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 211: this message will repeat every 5th retry.
W20250622 19:57:26.707433 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 216: this message will repeat every 5th retry.
W20250622 19:57:27.215730 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 221: this message will repeat every 5th retry.
W20250622 19:57:27.730137 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 226: this message will repeat every 5th retry.
I20250622 19:57:27.877413 3787 maintenance_manager.cc:419] P 9ece0d51c71b47a1865ba3b05920d999: Scheduling FlushMRSOp(03ebbbde6ff2448a9a4c52e4cf8d160d): perf score=0.061004
I20250622 19:57:28.297413 5941 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 1914 ops behind the committed index [suppressed 28 similar messages]
W20250622 19:57:28.317667 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 231: this message will repeat every 5th retry.
W20250622 19:57:28.814363 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 236: this message will repeat every 5th retry.
W20250622 19:57:29.375936 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 241: this message will repeat every 5th retry.
I20250622 19:57:29.563320 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap replaying log segment 1/3 (7.67M/8.00M this segment, stats: ops{read=4504 overwritten=0 applied=4503 ignored=0} inserts{seen=224950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250622 19:57:29.890551 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 246: this message will repeat every 5th retry.
I20250622 19:57:30.567881 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap replayed 1/3 log segments. Stats: ops{read=4696 overwritten=0 applied=4694 ignored=0} inserts{seen=234500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
W20250622 19:57:30.667197 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 251: this message will repeat every 5th retry.
I20250622 19:57:31.312992 4451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 19:57:31.351064 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 256: this message will repeat every 5th retry.
I20250622 19:57:31.383529 5078 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:31.385519 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:31.439291 5978 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 2113 ops behind the committed index [suppressed 25 similar messages]
I20250622 19:57:31.454891 5678 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:31.468350 4309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:31.482656 5533 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:31.519596 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250622 19:57:31.524560 4594 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 19:57:31.562314 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 19:57:31.953547 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 261: this message will repeat every 5th retry.
W20250622 19:57:32.550801 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 266: this message will repeat every 5th retry.
W20250622 19:57:33.311453 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 271: this message will repeat every 5th retry.
W20250622 19:57:33.808301 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 276: this message will repeat every 5th retry.
W20250622 19:57:34.314610 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 281: this message will repeat every 5th retry.
I20250622 19:57:34.527947 6001 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 2305 ops behind the committed index [suppressed 25 similar messages]
I20250622 19:57:34.565266 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap replaying log segment 2/3 (968.2K/8.00M this segment, stats: ops{read=5242 overwritten=0 applied=5240 ignored=0} inserts{seen=261800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250622 19:57:35.016039 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 286: this message will repeat every 5th retry.
W20250622 19:57:35.634570 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 291: this message will repeat every 5th retry.
W20250622 19:57:36.224435 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 296: this message will repeat every 5th retry.
W20250622 19:57:36.942099 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 301: this message will repeat every 5th retry.
W20250622 19:57:37.535576 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 306: this message will repeat every 5th retry.
I20250622 19:57:37.590888 6006 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 2479 ops behind the committed index [suppressed 23 similar messages]
W20250622 19:57:38.092536 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 311: this message will repeat every 5th retry.
W20250622 19:57:38.495697 3328 meta_cache.cc:1261] Time spent looking up entry by key: real 0.069s user 0.000s sys 0.000s
W20250622 19:57:38.655503 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 316: this message will repeat every 5th retry.
W20250622 19:57:39.199110 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 321: this message will repeat every 5th retry.
I20250622 19:57:39.371002 4451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:39.498803 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:39.571405 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap replaying log segment 2/3 (2.39M/8.00M this segment, stats: ops{read=6078 overwritten=0 applied=6078 ignored=0} inserts{seen=303700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250622 19:57:39.590715 5078 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:39.622385 5678 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:39.626287 4309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:39.644667 4594 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 19:57:39.678272 5533 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:39.693008 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250622 19:57:39.703311 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 19:57:39.830667 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 326: this message will repeat every 5th retry.
W20250622 19:57:40.380497 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 331: this message will repeat every 5th retry.
I20250622 19:57:40.652303 6038 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 2677 ops behind the committed index [suppressed 26 similar messages]
W20250622 19:57:40.982455 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 336: this message will repeat every 5th retry.
W20250622 19:57:41.489876 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 341: this message will repeat every 5th retry.
W20250622 19:57:42.162125 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 346: this message will repeat every 5th retry.
W20250622 19:57:42.814422 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 351: this message will repeat every 5th retry.
W20250622 19:57:43.334471 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 356: this message will repeat every 5th retry.
I20250622 19:57:43.672112 6012 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 2864 ops behind the committed index [suppressed 23 similar messages]
W20250622 19:57:44.057163 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 361: this message will repeat every 5th retry.
W20250622 19:57:44.139102 3300 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.248.3:41117, user_credentials={real_user=slave}} blocked reactor thread for 73120us
I20250622 19:57:44.566893 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap replaying log segment 2/3 (3.63M/8.00M this segment, stats: ops{read=6794 overwritten=0 applied=6793 ignored=0} inserts{seen=339450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250622 19:57:44.574142 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 366: this message will repeat every 5th retry.
W20250622 19:57:45.171187 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 371: this message will repeat every 5th retry.
W20250622 19:57:45.636633 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 376: this message will repeat every 5th retry.
W20250622 19:57:46.285683 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 381: this message will repeat every 5th retry.
I20250622 19:57:46.827026 6078 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 3095 ops behind the committed index [suppressed 27 similar messages]
W20250622 19:57:46.866041 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 386: this message will repeat every 5th retry.
W20250622 19:57:47.378391 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 391: this message will repeat every 5th retry.
W20250622 19:57:47.908663 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 396: this message will repeat every 5th retry.
I20250622 19:57:47.988598 4451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:48.010648 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:48.060477 5078 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:48.067982 4309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:48.105707 4594 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 19:57:48.130226 5678 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:48.135969 5533 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:48.155016 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:48.169657 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
W20250622 19:57:48.420490 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 401: this message will repeat every 5th retry.
W20250622 19:57:48.984051 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 406: this message will repeat every 5th retry.
I20250622 19:57:49.568871 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap replaying log segment 2/3 (4.71M/8.00M this segment, stats: ops{read=7425 overwritten=0 applied=7425 ignored=0} inserts{seen=371000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250622 19:57:49.679831 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 411: this message will repeat every 5th retry.
I20250622 19:57:49.863044 6089 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 3281 ops behind the committed index [suppressed 26 similar messages]
W20250622 19:57:50.346446 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 416: this message will repeat every 5th retry.
W20250622 19:57:50.919938 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 421: this message will repeat every 5th retry.
W20250622 19:57:51.400827 3300 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.248.3:41117, user_credentials={real_user=slave}} blocked reactor thread for 106712us
W20250622 19:57:51.470958 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 426: this message will repeat every 5th retry.
W20250622 19:57:51.982597 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 431: this message will repeat every 5th retry.
W20250622 19:57:52.467327 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 436: this message will repeat every 5th retry.
I20250622 19:57:52.935074 6089 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 3533 ops behind the committed index [suppressed 27 similar messages]
W20250622 19:57:53.086925 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 441: this message will repeat every 5th retry.
W20250622 19:57:53.665450 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 446: this message will repeat every 5th retry.
W20250622 19:57:54.268818 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 451: this message will repeat every 5th retry.
W20250622 19:57:54.399051 3327 meta_cache.cc:1261] Time spent looking up entry by key: real 0.057s user 0.000s sys 0.000s
I20250622 19:57:54.575209 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap replaying log segment 2/3 (5.90M/8.00M this segment, stats: ops{read=8135 overwritten=0 applied=8135 ignored=0} inserts{seen=406500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250622 19:57:55.125064 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 456: this message will repeat every 5th retry.
I20250622 19:57:55.549177 4451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:55.582824 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:55.605165 4309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:55.621223 5078 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:55.633935 5533 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:55.660629 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250622 19:57:55.675387 5678 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:55.715535 4594 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 19:57:55.775913 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:57:55.947554 6155 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 3730 ops behind the committed index [suppressed 19 similar messages]
W20250622 19:57:56.012768 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 461: this message will repeat every 5th retry.
W20250622 19:57:56.644865 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 466: this message will repeat every 5th retry.
W20250622 19:57:57.166456 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 471: this message will repeat every 5th retry.
W20250622 19:57:57.665400 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 476: this message will repeat every 5th retry.
W20250622 19:57:58.225840 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 481: this message will repeat every 5th retry.
W20250622 19:57:58.857793 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 486: this message will repeat every 5th retry.
I20250622 19:57:58.983253 6158 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 3975 ops behind the committed index [suppressed 26 similar messages]
W20250622 19:57:59.366813 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 491: this message will repeat every 5th retry.
I20250622 19:57:59.592738 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap replaying log segment 2/3 (7.05M/8.00M this segment, stats: ops{read=8819 overwritten=0 applied=8819 ignored=0} inserts{seen=440700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250622 19:57:59.899056 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 496: this message will repeat every 5th retry.
W20250622 19:58:00.528702 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 501: this message will repeat every 5th retry.
W20250622 19:58:01.209306 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 506: this message will repeat every 5th retry.
W20250622 19:58:01.871973 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 511: this message will repeat every 5th retry.
I20250622 19:58:01.876912 3673 maintenance_manager.cc:643] P 9ece0d51c71b47a1865ba3b05920d999: FlushMRSOp(03ebbbde6ff2448a9a4c52e4cf8d160d) complete. Timing: real 33.999s user 18.841s sys 0.331s Metrics: {"bytes_written":16542407,"cfile_init":1,"dirs.queue_time_us":28741,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":1990,"drs_written":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4,"lbm_write_time_us":94445,"lbm_writes_1-10_ms":10,"lbm_writes_lt_1ms":715,"peak_mem_usage":0,"rows_written":595600,"thread_start_us":23411,"threads_started":1}
I20250622 19:58:01.881160 3787 maintenance_manager.cc:419] P 9ece0d51c71b47a1865ba3b05920d999: Scheduling UndoDeltaBlockGCOp(03ebbbde6ff2448a9a4c52e4cf8d160d): 6902498 bytes on disk
I20250622 19:58:01.882783 3673 maintenance_manager.cc:643] P 9ece0d51c71b47a1865ba3b05920d999: UndoDeltaBlockGCOp(03ebbbde6ff2448a9a4c52e4cf8d160d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20250622 19:58:02.035681 6158 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 4187 ops behind the committed index [suppressed 24 similar messages]
W20250622 19:58:02.544236 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 516: this message will repeat every 5th retry.
I20250622 19:58:02.590242 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap replayed 2/3 log segments. Stats: ops{read=9383 overwritten=0 applied=9381 ignored=0} inserts{seen=468800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250622 19:58:02.644683 3858 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:58:02.654809 4451 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:58:02.672768 5078 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:58:02.696597 4309 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:58:02.754016 4594 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250622 19:58:02.773497 5533 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:58:02.830659 5678 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250622 19:58:02.881466 3717 tablet_service.cc:1430] Tablet server has 0 leaders and 3 scanners
I20250622 19:58:02.959595 3576 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250622 19:58:03.213373 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 521: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/maintenance_mode-itest.cc:753: Failure
Failed
Bad status: Timed out: cluster didn't become healthy: Running 'cluster ksck 127.1.248.62:37973' did not succeed in 60 seconds: Runtime error: error running 'cluster ksck 127.1.248.62:37973': stdout: Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
e103efdfebd54e69b47694a7953f484c | 127.1.248.62:37973 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.248.20:38913 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server States
Server | State
----------------------------------+------------------
7520035c9f604041a00535b523c45c1b | MAINTENANCE_MODE
7e22adb60d65484c9844ee7d2108a396 | MAINTENANCE_MODE
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
07bc6e053fe04f6d8b0c0157ca553029 | 127.1.248.2:35999 | HEALTHY | /L0 | 0 | 0
07ce178dce43415a852dbcce80f47f3e | 127.1.248.7:35201 | HEALTHY | /L1 | 0 | 0
144888b25afe4c04b4de5350bddf224e | 127.1.248.4:40193 | HEALTHY | /L1 | 0 | 0
2db5cde30e5a47fba8d434cc04849edc | 127.1.248.1:36357 | HEALTHY | /L0 | 0 | 0
54555a8ecf924b339db8e8306cbc8bb4 | 127.1.248.3:41117 | HEALTHY | /L0 | 1 | 0
7520035c9f604041a00535b523c45c1b | 127.1.248.9:44015 | HEALTHY | /L2 | 0 | 0
7e22adb60d65484c9844ee7d2108a396 | 127.1.248.8:39693 | HEALTHY | /L2 | 0 | 0
9ece0d51c71b47a1865ba3b05920d999 | 127.1.248.5:43933 | HEALTHY | /L1 | 0 | 3
e63bffd596554411a2e0489c19e78fff | 127.1.248.6:38357 | HEALTHY | /L1 | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/L2 | 2
/L0 | 3
/L1 | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 9 server(s) checked
local_ip_for_outbound_sockets | 127.1.248.1 | experimental | 127.1.248.1:36357
local_ip_for_outbound_sockets | 127.1.248.2 | experimental | 127.1.248.2:35999
local_ip_for_outbound_sockets | 127.1.248.3 | experimental | 127.1.248.3:41117
local_ip_for_outbound_sockets | 127.1.248.4 | experimental | 127.1.248.4:40193
local_ip_for_outbound_sockets | 127.1.248.5 | experimental | 127.1.248.5:43933
local_ip_for_outbound_sockets | 127.1.248.6 | experimental | 127.1.248.6:38357
local_ip_for_outbound_sockets | 127.1.248.7 | experimental | 127.1.248.7:35201
local_ip_for_outbound_sockets | 127.1.248.8 | experimental | 127.1.248.8:39693
local_ip_for_outbound_sockets | 127.1.248.9 | experimental | 127.1.248.9:44015
never_fsync | true | unsafe,advanced | all 9 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 9 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked
server_dump_info_format | pb | hidden | all 9 server(s) checked
server_dump_info_path | /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-0/data/info.pb | hidden | 127.1.248.1:36357
server_dump_info_path | /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-1/data/info.pb | hidden | 127.1.248.2:35999
server_dump_info_path | /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-2/data/info.pb | hidden | 127.1.248.3:41117
server_dump_info_path | /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-3/data/info.pb | hidden | 127.1.248.4:40193
server_dump_info_path | /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/data/info.pb | hidden | 127.1.248.5:43933
server_dump_info_path | /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-5/data/info.pb | hidden | 127.1.248.6:38357
server_dump_info_path | /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-6/data/info.pb | hidden | 127.1.248.7:35201
server_dump_info_path | /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-7/data/info.pb | hidden | 127.1.248.8:39693
server_dump_info_path | /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-8/data/info.pb | hidden | 127.1.248.9:44015
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.248.20:38913 | all 9 server(s) checked
time_source | builtin | all 9 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------
1.18.0-SNAPSHOT | all 10 server(s) checked
Tablet Summary
Tablet 03ebbbde6ff2448a9a4c52e4cf8d160d of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): not running
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap replayed 2/3 log segments. Stats: ops{read=9383 overwritten=0 applied=9381 ignored=0} inserts{seen=468800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): RUNNING [LEADER]
9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933): RUNNING
All reported replicas are:
A = 7e22adb60d65484c9844ee7d2108a396
B = 54555a8ecf924b339db8e8306cbc8bb4
C = 9ece0d51c71b47a1865ba3b05920d999
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 8 | -1 | Yes
B | A B* C | 8 | -1 | Yes
C | A B* C | 8 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 9
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
, stderr: W20250622 19:58:01.990130 6226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250622 19:58:01.990732 6226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250622 19:58:02.003067 6226 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Runtime error: ksck discovered errors
: /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu: process exited with non-zero status 1
W20250622 19:58:03.752003 4546 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 -> Peer 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693): Couldn't send request to peer 7e22adb60d65484c9844ee7d2108a396. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 526: this message will repeat every 5th retry.
I20250622 19:58:04.013377 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap replayed 3/3 log segments. Stats: ops{read=9625 overwritten=0 applied=9619 ignored=0} inserts{seen=480700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 6 replicates
I20250622 19:58:04.014413 5635 tablet_bootstrap.cc:492] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Bootstrap complete.
I20250622 19:58:04.020419 5635 ts_tablet_manager.cc:1397] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Time spent bootstrapping tablet: real 59.722s user 37.861s sys 1.297s
I20250622 19:58:04.025679 5635 raft_consensus.cc:357] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 8 FOLLOWER]: Replica starting. Triggering 6 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:58:04.031397 5635 raft_consensus.cc:738] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e22adb60d65484c9844ee7d2108a396, State: Initialized, Role: FOLLOWER
I20250622 19:58:04.032280 5635 consensus_queue.cc:260] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9619, Last appended: 8.9625, Last appended by leader: 9625, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:58:04.072468 5635 ts_tablet_manager.cc:1428] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396: Time spent starting tablet: real 0.052s user 0.025s sys 0.027s
I20250622 19:58:04.168632 5700 raft_consensus.cc:1215] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 8 FOLLOWER]: Deduplicated request from leader. Original: 8.9619->[8.9620-8.10044] Dedup: 8.9625->[8.9626-8.10044]
I20250622 19:58:04.723990 6277 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 54555a8ecf924b339db8e8306cbc8bb4)
I20250622 19:58:04.724319 6277 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:58:04.727231 6277 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117), 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
I20250622 19:58:04.768428 4614 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "7e22adb60d65484c9844ee7d2108a396" candidate_term: 9 candidate_status { last_received { term: 8 index: 10469 } } ignore_live_leader: false dest_uuid: "54555a8ecf924b339db8e8306cbc8bb4" is_pre_election: true
I20250622 19:58:04.773128 3738 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "7e22adb60d65484c9844ee7d2108a396" candidate_term: 9 candidate_status { last_received { term: 8 index: 10469 } } ignore_live_leader: false dest_uuid: "9ece0d51c71b47a1865ba3b05920d999" is_pre_election: true
I20250622 19:58:04.777570 5629 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e22adb60d65484c9844ee7d2108a396; no voters: 54555a8ecf924b339db8e8306cbc8bb4, 9ece0d51c71b47a1865ba3b05920d999
I20250622 19:58:04.778028 6277 raft_consensus.cc:2747] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250622 19:58:05.132239 6217 consensus_queue.cc:786] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 54555a8ecf924b339db8e8306cbc8bb4 [LEADER]: Peer 7e22adb60d65484c9844ee7d2108a396 is lagging by at least 2577 ops behind the committed index [suppressed 20 similar messages]
I20250622 19:58:07.256063 2016 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20250622 19:58:07.256273 2016 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 2db5cde30e5a47fba8d434cc04849edc and pid 4241
I20250622 19:58:07.406772 3787 maintenance_manager.cc:419] P 9ece0d51c71b47a1865ba3b05920d999: Scheduling LogGCOp(03ebbbde6ff2448a9a4c52e4cf8d160d): free 16779658 bytes of WAL
I20250622 19:58:07.407163 3673 log_reader.cc:385] T 03ebbbde6ff2448a9a4c52e4cf8d160d: removed 2 log segments from log reader
I20250622 19:58:07.407277 3673 log.cc:1079] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Deleting log segment in path: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/wal/wals/03ebbbde6ff2448a9a4c52e4cf8d160d/wal-000000001 (ops 1-4820)
I20250622 19:58:07.407404 3673 log.cc:1079] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999: Deleting log segment in path: /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0/minicluster-data/ts-4/wal/wals/03ebbbde6ff2448a9a4c52e4cf8d160d/wal-000000002 (ops 4821-9558)
I20250622 19:58:07.411752 3673 maintenance_manager.cc:643] P 9ece0d51c71b47a1865ba3b05920d999: LogGCOp(03ebbbde6ff2448a9a4c52e4cf8d160d) complete. Timing: real 0.005s user 0.000s sys 0.003s Metrics: {}
************************ BEGIN STACKS **************************
[New LWP 4245]
[New LWP 4246]
[New LWP 4247]
[New LWP 4248]
[New LWP 4254]
[New LWP 4255]
[New LWP 4256]
[New LWP 4259]
[New LWP 4260]
[New LWP 4261]
[New LWP 4262]
[New LWP 4263]
[New LWP 4264]
[New LWP 4265]
[New LWP 4266]
[New LWP 4267]
[New LWP 4268]
[New LWP 4269]
[New LWP 4271]
[New LWP 4272]
[New LWP 4273]
[New LWP 4274]
[New LWP 4276]
[New LWP 4277]
[New LWP 4278]
[New LWP 4279]
[New LWP 4280]
[New LWP 4281]
[New LWP 4282]
[New LWP 4283]
[New LWP 4284]
[New LWP 4285]
[New LWP 4286]
[New LWP 4287]
[New LWP 4288]
[New LWP 4289]
[New LWP 4290]
[New LWP 4292]
[New LWP 4293]
[New LWP 4294]
[New LWP 4295]
[New LWP 4296]
[New LWP 4297]
[New LWP 4298]
[New LWP 4299]
[New LWP 4300]
[New LWP 4301]
[New LWP 4302]
[New LWP 4303]
[New LWP 4304]
[New LWP 4305]
[New LWP 4306]
[New LWP 4307]
[New LWP 4308]
[New LWP 4309]
[New LWP 4310]
[New LWP 4311]
[New LWP 4312]
[New LWP 4313]
[New LWP 4314]
[New LWP 4315]
[New LWP 4316]
[New LWP 4317]
[New LWP 4318]
[New LWP 4319]
[New LWP 4320]
[New LWP 4321]
[New LWP 4322]
[New LWP 4323]
[New LWP 4324]
[New LWP 4325]
[New LWP 4326]
[New LWP 4327]
[New LWP 4329]
[New LWP 4330]
[New LWP 4331]
[New LWP 4332]
[New LWP 4333]
[New LWP 4334]
[New LWP 4335]
[New LWP 4336]
[New LWP 4337]
[New LWP 4338]
[New LWP 4339]
[New LWP 4340]
[New LWP 4341]
[New LWP 4342]
[New LWP 4343]
[New LWP 4344]
[New LWP 4345]
[New LWP 4346]
[New LWP 4347]
[New LWP 4348]
[New LWP 4349]
[New LWP 4350]
[New LWP 4351]
[New LWP 4352]
[New LWP 4353]
[New LWP 4354]
[New LWP 4355]
[New LWP 4356]
[New LWP 4357]
[New LWP 4358]
[New LWP 4359]
[New LWP 4360]
[New LWP 4361]
[New LWP 4362]
[New LWP 4363]
[New LWP 4364]
[New LWP 4365]
[New LWP 4366]
[New LWP 4367]
[New LWP 4368]
[New LWP 4369]
[New LWP 4370]
[New LWP 4371]
[New LWP 4372]
[New LWP 4373]
[New LWP 4374]
[New LWP 4375]
[New LWP 4378]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007f1f68fead50 in ?? ()
Id Target Id Frame
* 1 LWP 4241 "kudu" 0x00007f1f68fead50 in ?? ()
2 LWP 4245 "kudu" 0x00007f1f68fe6fb9 in ?? ()
3 LWP 4246 "kudu" 0x00007f1f68fe6fb9 in ?? ()
4 LWP 4247 "kudu" 0x00007f1f68fe6fb9 in ?? ()
5 LWP 4248 "kernel-watcher-" 0x00007f1f68fe6fb9 in ?? ()
6 LWP 4254 "ntp client-4254" 0x00007f1f68fea9e2 in ?? ()
7 LWP 4255 "file cache-evic" 0x00007f1f68fe6fb9 in ?? ()
8 LWP 4256 "sq_acceptor" 0x00007f1f61d0ccb9 in ?? ()
9 LWP 4259 "rpc reactor-425" 0x00007f1f61d19a47 in ?? ()
10 LWP 4260 "rpc reactor-426" 0x00007f1f61d19a47 in ?? ()
11 LWP 4261 "rpc reactor-426" 0x00007f1f61d19a47 in ?? ()
12 LWP 4262 "rpc reactor-426" 0x00007f1f61d19a47 in ?? ()
13 LWP 4263 "MaintenanceMgr " 0x00007f1f68fe6ad3 in ?? ()
14 LWP 4264 "txn-status-mana" 0x00007f1f68fe6fb9 in ?? ()
15 LWP 4265 "collect_and_rem" 0x00007f1f68fe6fb9 in ?? ()
16 LWP 4266 "tc-session-exp-" 0x00007f1f68fe6fb9 in ?? ()
17 LWP 4267 "rpc worker-4267" 0x00007f1f68fe6ad3 in ?? ()
18 LWP 4268 "rpc worker-4268" 0x00007f1f68fe6ad3 in ?? ()
19 LWP 4269 "rpc worker-4269" 0x00007f1f68fe6ad3 in ?? ()
20 LWP 4271 "rpc worker-4271" 0x00007f1f68fe6ad3 in ?? ()
21 LWP 4272 "rpc worker-4272" 0x00007f1f68fe6ad3 in ?? ()
22 LWP 4273 "rpc worker-4273" 0x00007f1f68fe6ad3 in ?? ()
23 LWP 4274 "rpc worker-4274" 0x00007f1f68fe6ad3 in ?? ()
24 LWP 4276 "rpc worker-4276" 0x00007f1f68fe6ad3 in ?? ()
25 LWP 4277 "rpc worker-4277" 0x00007f1f68fe6ad3 in ?? ()
26 LWP 4278 "rpc worker-4278" 0x00007f1f68fe6ad3 in ?? ()
27 LWP 4279 "rpc worker-4279" 0x00007f1f68fe6ad3 in ?? ()
28 LWP 4280 "rpc worker-4280" 0x00007f1f68fe6ad3 in ?? ()
29 LWP 4281 "rpc worker-4281" 0x00007f1f68fe6ad3 in ?? ()
30 LWP 4282 "rpc worker-4282" 0x00007f1f68fe6ad3 in ?? ()
31 LWP 4283 "rpc worker-4283" 0x00007f1f68fe6ad3 in ?? ()
32 LWP 4284 "rpc worker-4284" 0x00007f1f68fe6ad3 in ?? ()
33 LWP 4285 "rpc worker-4285" 0x00007f1f68fe6ad3 in ?? ()
34 LWP 4286 "rpc worker-4286" 0x00007f1f68fe6ad3 in ?? ()
35 LWP 4287 "rpc worker-4287" 0x00007f1f68fe6ad3 in ?? ()
36 LWP 4288 "rpc worker-4288" 0x00007f1f68fe6ad3 in ?? ()
37 LWP 4289 "rpc worker-4289" 0x00007f1f68fe6ad3 in ?? ()
38 LWP 4290 "rpc worker-4290" 0x00007f1f68fe6ad3 in ?? ()
39 LWP 4292 "rpc worker-4292" 0x00007f1f68fe6ad3 in ?? ()
40 LWP 4293 "rpc worker-4293" 0x00007f1f68fe6ad3 in ?? ()
41 LWP 4294 "rpc worker-4294" 0x00007f1f68fe6ad3 in ?? ()
42 LWP 4295 "rpc worker-4295" 0x00007f1f68fe6ad3 in ?? ()
43 LWP 4296 "rpc worker-4296" 0x00007f1f68fe6ad3 in ?? ()
44 LWP 4297 "rpc worker-4297" 0x00007f1f68fe6ad3 in ?? ()
45 LWP 4298 "rpc worker-4298" 0x00007f1f68fe6ad3 in ?? ()
46 LWP 4299 "rpc worker-4299" 0x00007f1f68fe6ad3 in ?? ()
47 LWP 4300 "rpc worker-4300" 0x00007f1f68fe6ad3 in ?? ()
48 LWP 4301 "rpc worker-4301" 0x00007f1f68fe6ad3 in ?? ()
49 LWP 4302 "rpc worker-4302" 0x00007f1f68fe6ad3 in ?? ()
50 LWP 4303 "rpc worker-4303" 0x00007f1f68fe6ad3 in ?? ()
51 LWP 4304 "rpc worker-4304" 0x00007f1f68fe6ad3 in ?? ()
52 LWP 4305 "rpc worker-4305" 0x00007f1f68fe6ad3 in ?? ()
53 LWP 4306 "rpc worker-4306" 0x00007f1f68fe6ad3 in ?? ()
54 LWP 4307 "rpc worker-4307" 0x00007f1f68fe6ad3 in ?? ()
55 LWP 4308 "rpc worker-4308" 0x00007f1f68fe6ad3 in ?? ()
56 LWP 4309 "rpc worker-4309" 0x00007f1f68fe6ad3 in ?? ()
57 LWP 4310 "rpc worker-4310" 0x00007f1f68fe6ad3 in ?? ()
58 LWP 4311 "rpc worker-4311" 0x00007f1f68fe6ad3 in ?? ()
59 LWP 4312 "rpc worker-4312" 0x00007f1f68fe6ad3 in ?? ()
60 LWP 4313 "rpc worker-4313" 0x00007f1f68fe6ad3 in ?? ()
61 LWP 4314 "rpc worker-4314" 0x00007f1f68fe6ad3 in ?? ()
62 LWP 4315 "rpc worker-4315" 0x00007f1f68fe6ad3 in ?? ()
63 LWP 4316 "rpc worker-4316" 0x00007f1f68fe6ad3 in ?? ()
64 LWP 4317 "rpc worker-4317" 0x00007f1f68fe6ad3 in ?? ()
65 LWP 4318 "rpc worker-4318" 0x00007f1f68fe6ad3 in ?? ()
66 LWP 4319 "rpc worker-4319" 0x00007f1f68fe6ad3 in ?? ()
67 LWP 4320 "rpc worker-4320" 0x00007f1f68fe6ad3 in ?? ()
68 LWP 4321 "rpc worker-4321" 0x00007f1f68fe6ad3 in ?? ()
69 LWP 4322 "rpc worker-4322" 0x00007f1f68fe6ad3 in ?? ()
70 LWP 4323 "rpc worker-4323" 0x00007f1f68fe6ad3 in ?? ()
71 LWP 4324 "rpc worker-4324" 0x00007f1f68fe6ad3 in ?? ()
72 LWP 4325 "rpc worker-4325" 0x00007f1f68fe6ad3 in ?? ()
73 LWP 4326 "rpc worker-4326" 0x00007f1f68fe6ad3 in ?? ()
74 LWP 4327 "rpc worker-4327" 0x00007f1f68fe6ad3 in ?? ()
75 LWP 4329 "rpc worker-4329" 0x00007f1f68fe6ad3 in ?? ()
76 LWP 4330 "rpc worker-4330" 0x00007f1f68fe6ad3 in ?? ()
77 LWP 4331 "rpc worker-4331" 0x00007f1f68fe6ad3 in ?? ()
78 LWP 4332 "rpc worker-4332" 0x00007f1f68fe6ad3 in ?? ()
79 LWP 4333 "rpc worker-4333" 0x00007f1f68fe6ad3 in ?? ()
80 LWP 4334 "rpc worker-4334" 0x00007f1f68fe6ad3 in ?? ()
81 LWP 4335 "rpc worker-4335" 0x00007f1f68fe6ad3 in ?? ()
82 LWP 4336 "rpc worker-4336" 0x00007f1f68fe6ad3 in ?? ()
83 LWP 4337 "rpc worker-4337" 0x00007f1f68fe6ad3 in ?? ()
84 LWP 4338 "rpc worker-4338" 0x00007f1f68fe6ad3 in ?? ()
85 LWP 4339 "rpc worker-4339" 0x00007f1f68fe6ad3 in ?? ()
86 LWP 4340 "rpc worker-4340" 0x00007f1f68fe6ad3 in ?? ()
87 LWP 4341 "rpc worker-4341" 0x00007f1f68fe6ad3 in ?? ()
88 LWP 4342 "rpc worker-4342" 0x00007f1f68fe6ad3 in ?? ()
89 LWP 4343 "rpc worker-4343" 0x00007f1f68fe6ad3 in ?? ()
90 LWP 4344 "rpc worker-4344" 0x00007f1f68fe6ad3 in ?? ()
91 LWP 4345 "rpc worker-4345" 0x00007f1f68fe6ad3 in ?? ()
92 LWP 4346 "rpc worker-4346" 0x00007f1f68fe6ad3 in ?? ()
93 LWP 4347 "rpc worker-4347" 0x00007f1f68fe6ad3 in ?? ()
94 LWP 4348 "rpc worker-4348" 0x00007f1f68fe6ad3 in ?? ()
95 LWP 4349 "rpc worker-4349" 0x00007f1f68fe6ad3 in ?? ()
96 LWP 4350 "rpc worker-4350" 0x00007f1f68fe6ad3 in ?? ()
97 LWP 4351 "rpc worker-4351" 0x00007f1f68fe6ad3 in ?? ()
98 LWP 4352 "rpc worker-4352" 0x00007f1f68fe6ad3 in ?? ()
99 LWP 4353 "rpc worker-4353" 0x00007f1f68fe6ad3 in ?? ()
100 LWP 4354 "rpc worker-4354" 0x00007f1f68fe6ad3 in ?? ()
101 LWP 4355 "rpc worker-4355" 0x00007f1f68fe6ad3 in ?? ()
102 LWP 4356 "rpc worker-4356" 0x00007f1f68fe6ad3 in ?? ()
103 LWP 4357 "rpc worker-4357" 0x00007f1f68fe6ad3 in ?? ()
104 LWP 4358 "rpc worker-4358" 0x00007f1f68fe6ad3 in ?? ()
105 LWP 4359 "rpc worker-4359" 0x00007f1f68fe6ad3 in ?? ()
106 LWP 4360 "rpc worker-4360" 0x00007f1f68fe6ad3 in ?? ()
107 LWP 4361 "rpc worker-4361" 0x00007f1f68fe6ad3 in ?? ()
108 LWP 4362 "rpc worker-4362" 0x00007f1f68fe6ad3 in ?? ()
109 LWP 4363 "rpc worker-4363" 0x00007f1f68fe6ad3 in ?? ()
110 LWP 4364 "rpc worker-4364" 0x00007f1f68fe6ad3 in ?? ()
111 LWP 4365 "rpc worker-4365" 0x00007f1f68fe6ad3 in ?? ()
112 LWP 4366 "rpc worker-4366" 0x00007f1f68fe6ad3 in ?? ()
113 LWP 4367 "rpc worker-4367" 0x00007f1f68fe6ad3 in ?? ()
114 LWP 4368 "rpc worker-4368" 0x00007f1f68fe6ad3 in ?? ()
115 LWP 4369 "rpc worker-4369" 0x00007f1f68fe6ad3 in ?? ()
116 LWP 4370 "rpc worker-4370" 0x00007f1f68fe6ad3 in ?? ()
117 LWP 4371 "diag-logger-437" 0x00007f1f68fe6fb9 in ?? ()
118 LWP 4372 "result-tracker-" 0x00007f1f68fe6fb9 in ?? ()
119 LWP 4373 "excess-log-dele" 0x00007f1f68fe6fb9 in ?? ()
120 LWP 4374 "acceptor-4374" 0x00007f1f61d1b0c7 in ?? ()
121 LWP 4375 "heartbeat-4375" 0x00007f1f68fe6fb9 in ?? ()
122 LWP 4378 "maintenance_sch" 0x00007f1f68fe6fb9 in ?? ()
Thread 122 (LWP 4378):
#0 0x00007f1f68fe6fb9 in ?? ()
#1 0x00007f1f1e88e6e0 in ?? ()
#2 0x0000000000000232 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f1f1e88e730 in ?? ()
#6 0x0000000000000464 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 4375):
#0 0x00007f1f68fe6fb9 in ?? ()
#1 0x8ea7c1ad18c34923 in ?? ()
#2 0x000000000000008c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019400 in ?? ()
#5 0x00007f1f1f0a65f0 in ?? ()
#6 0x0000000000000118 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 4374):
#0 0x00007f1f61d1b0c7 in ?? ()
#1 0x00006140000214a8 in ?? ()
#2 0x00007f1f1f8cdd90 in ?? ()
#3 0x00007f1f1f8cddc0 in ?? ()
#4 0x00007f1f1f8cdec0 in ?? ()
#5 0x00007f1f1f8cddb0 in ?? ()
#6 0x00007f1f1f8cde20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 4373):
#0 0x00007f1f68fe6fb9 in ?? ()
#1 0x00007f1f200e7f40 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffe42f55b10 in ?? ()
#5 0x00007f1f200e7f30 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 4372):
#0 0x00007f1f68fe6fb9 in ?? ()
#1 0x00007f1f20900100 in ?? ()
#2 0x000000000000008c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f1f209000f0 in ?? ()
#6 0x0000000000000118 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 4371):
#0 0x00007f1f68fe6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 4370):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007f1f2192fe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1f2192feb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 4369):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x000000000000001c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002a6858 in ?? ()
#4 0x00007f1f22147e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1f22147eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 4368):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 4367):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 4366):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 4365):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 4364):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 4363):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 4362):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 4361):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 4360):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 4359):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 4358):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 4357):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 4356):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 4355):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 4354):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 4353):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 4352):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 4351):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 4350):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 4349):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 4348):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 4347):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 4346):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 4345):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 4344):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 4343):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 4342):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 4341):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 4340):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 4339):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 4338):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 4337):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 4336):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 4335):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 4334):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 4333):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 4332):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 4331):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 4330):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x000000000000000d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a688c in ?? ()
#4 0x00007f1f35d00e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1f35d00eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a6840 in ?? ()
#9 0x00007f1f68fe6770 in ?? ()
#10 0x00007f1f35d00eb0 in ?? ()
#11 0x00000fe3e6ba01d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 4329):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 4327):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 4326):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 4325):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 4324):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 4323):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 4322):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 4321):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 4320):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 4319):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 4318):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 4317):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 4316):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 4315):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 4314):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 4313):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 4312):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 4311):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 4310):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 4309):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x000000000000000d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012003c in ?? ()
#4 0x00007f1f3f6c9e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1f3f6c9eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00011fff0 in ?? ()
#9 0x00007f1f68fe6770 in ?? ()
#10 0x00007f1f3f6c9eb0 in ?? ()
#11 0x00000fe3e7ed93d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 4308):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 4307):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 4306):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 4305):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 4304):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 4303):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 4302):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 4301):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 4300):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 4299):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 4298):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 4297):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 4296):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 4295):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 4294):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 4293):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 4292):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 4290):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 4289):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 4288):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x000000000000000d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007f1f498aae90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1f498aaeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007f1f68fe6770 in ?? ()
#10 0x00007f1f498aaeb0 in ?? ()
#11 0x00000fe3e93155d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 4287):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 4286):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 4285):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 4284):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 4283):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 4282):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 4281):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 4280):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 4279):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 4278):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 4277):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 4276):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 4274):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 4273):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 4272):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 4271):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 4269):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 4268):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 4267):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 4266):
#0 0x00007f1f68fe6fb9 in ?? ()
#1 0x00007f1f53aadcc0 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019ca0 in ?? ()
#5 0x00007f1f53aadcb0 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 4265):
#0 0x00007f1f68fe6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 4264):
#0 0x00007f1f68fe6fb9 in ?? ()
#1 0x00007f1f54aaf240 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007f1f54aaf230 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 4263):
#0 0x00007f1f68fe6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 4262):
#0 0x00007f1f61d19a47 in ?? ()
#1 0x00007f1f55ab2320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f1f55ab2310 in ?? ()
#4 0x00007f1f55ab2520 in ?? ()
#5 0x00007f1f55ab2360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f1f55ab23e0 in ?? ()
#8 0x00007f1f6438c25d in ?? ()
#9 0x3fb96722cf056000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x0000000006800000 in ?? ()
#14 0x00007f1f00000000 in ?? ()
#15 0x41da16179c5dd6b6 in ?? ()
#16 0x00000fe46ab4e47c in ?? ()
#17 0x00007f1f55ab23c0 in ?? ()
#18 0x00007f1f64390ba3 in ?? ()
#19 0x00007f1f55ab2390 in ?? ()
#20 0x3fb96722cf056000 in ?? ()
#21 0x0000000055ab23e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb96722cf056000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 4261):
#0 0x00007f1f61d19a47 in ?? ()
#1 0x00007f1f562b3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f1f562b3310 in ?? ()
#4 0x00007f1f562b3520 in ?? ()
#5 0x00007f1f562b3360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f1f562b33e0 in ?? ()
#8 0x00007f1f6438c25d in ?? ()
#9 0x3fb963ca43c2c000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x0000000038000000 in ?? ()
#14 0x00007f1f00000000 in ?? ()
#15 0x41da16179c5dd6b5 in ?? ()
#16 0x00000fe46ac4e67c in ?? ()
#17 0x00007f1f562b33c0 in ?? ()
#18 0x00007f1f64390ba3 in ?? ()
#19 0x00007f1f562b3390 in ?? ()
#20 0x3fb963ca43c2c000 in ?? ()
#21 0x00000000562b33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb963ca43c2c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 4260):
#0 0x00007f1f61d19a47 in ?? ()
#1 0x00007f1f56ab4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f1f56ab4310 in ?? ()
#4 0x00007f1f56ab4520 in ?? ()
#5 0x00007f1f56ab4360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f1f56ab43e0 in ?? ()
#8 0x00007f1f6438c25d in ?? ()
#9 0x3fb95bc522d7c000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x0000000088000000 in ?? ()
#14 0x00007f1f00000000 in ?? ()
#15 0x41da16179c5dd6b5 in ?? ()
#16 0x00000fe46ad4e87c in ?? ()
#17 0x00007f1f56ab43c0 in ?? ()
#18 0x00007f1f64390ba3 in ?? ()
#19 0x00007f1f56ab4390 in ?? ()
#20 0x3fb95bc522d7c000 in ?? ()
#21 0x0000000056ab43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb95bc522d7c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 4259):
#0 0x00007f1f61d19a47 in ?? ()
#1 0x00007f1f58aad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f1f58aad310 in ?? ()
#4 0x00007f1f58aad520 in ?? ()
#5 0x00007f1f58aad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007f1f58aad3e0 in ?? ()
#8 0x00007f1f6438c25d in ?? ()
#9 0x3fb969a3ae7a1000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x0000000030c00000 in ?? ()
#14 0x00007f1f00000000 in ?? ()
#15 0x41da16179c5dd6b6 in ?? ()
#16 0x00000fe46b14da7c in ?? ()
#17 0x00007f1f58aad3c0 in ?? ()
#18 0x00007f1f64390ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 4256):
#0 0x00007f1f61d0ccb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f1f5a4b77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 4255):
#0 0x00007f1f68fe6fb9 in ?? ()
#1 0x00007f1f59cb60c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007f1f59cb60b0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 4254):
#0 0x00007f1f68fea9e2 in ?? ()
#1 0x00007f1f594b3ba0 in ?? ()
#2 0x00007f1f594b3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f1f594b3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007f1f594b3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f1f594b3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007f1f6e8fdedc in ?? ()
#12 0x00007f1f6e8ed8d9 in ?? ()
#13 0x00007f1f6e8f1ec6 in ?? ()
#14 0x00007f1f6e8f6900 in ?? ()
#15 0x00007f1f6e8f6729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007f1f65bc24fe in ?? ()
#18 0x00007f1f68fe06db in ?? ()
#19 0x00000fe46b28e684 in ?? ()
#20 0x00007f1f594b3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 4248):
#0 0x00007f1f68fe6fb9 in ?? ()
#1 0x00007f1f5b4b8c80 in ?? ()
#2 0x00000000000002c2 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007f1f5b4b8c70 in ?? ()
#6 0x0000000000000584 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 4247):
#0 0x00007f1f68fe6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 4246):
#0 0x00007f1f68fe6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 4245):
#0 0x00007f1f68fe6fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f1f5ccbc450 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 4241):
#0 0x00007f1f68fead50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250622 19:58:08.464187 2016 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID 07bc6e053fe04f6d8b0c0157ca553029 and pid 4383
************************ BEGIN STACKS **************************
I20250622 19:58:09.389922 4663 maintenance_manager.cc:419] P 54555a8ecf924b339db8e8306cbc8bb4: Scheduling FlushMRSOp(03ebbbde6ff2448a9a4c52e4cf8d160d): perf score=0.072632
[New LWP 4386]
[New LWP 4387]
[New LWP 4388]
[New LWP 4389]
[New LWP 4395]
[New LWP 4396]
[New LWP 4397]
[New LWP 4400]
[New LWP 4401]
[New LWP 4402]
[New LWP 4403]
[New LWP 4404]
[New LWP 4405]
[New LWP 4406]
[New LWP 4407]
[New LWP 4408]
[New LWP 4409]
[New LWP 4410]
[New LWP 4411]
[New LWP 4412]
[New LWP 4413]
[New LWP 4414]
[New LWP 4415]
[New LWP 4416]
[New LWP 4417]
[New LWP 4418]
[New LWP 4419]
[New LWP 4420]
[New LWP 4421]
[New LWP 4422]
[New LWP 4423]
[New LWP 4425]
[New LWP 4426]
[New LWP 4427]
[New LWP 4428]
[New LWP 4429]
[New LWP 4430]
[New LWP 4431]
[New LWP 4432]
[New LWP 4433]
[New LWP 4434]
[New LWP 4437]
[New LWP 4438]
[New LWP 4440]
[New LWP 4441]
[New LWP 4442]
[New LWP 4443]
[New LWP 4444]
[New LWP 4445]
[New LWP 4446]
[New LWP 4447]
[New LWP 4448]
[New LWP 4449]
[New LWP 4450]
[New LWP 4451]
[New LWP 4452]
[New LWP 4453]
[New LWP 4454]
[New LWP 4455]
[New LWP 4456]
[New LWP 4457]
[New LWP 4458]
[New LWP 4459]
[New LWP 4460]
[New LWP 4461]
[New LWP 4462]
[New LWP 4463]
[New LWP 4464]
[New LWP 4465]
[New LWP 4466]
[New LWP 4467]
[New LWP 4468]
[New LWP 4469]
[New LWP 4470]
[New LWP 4471]
[New LWP 4472]
[New LWP 4473]
[New LWP 4474]
[New LWP 4475]
[New LWP 4476]
[New LWP 4477]
[New LWP 4478]
[New LWP 4479]
[New LWP 4480]
[New LWP 4481]
[New LWP 4482]
[New LWP 4483]
[New LWP 4484]
[New LWP 4485]
[New LWP 4486]
[New LWP 4487]
[New LWP 4488]
[New LWP 4489]
[New LWP 4490]
[New LWP 4491]
[New LWP 4492]
[New LWP 4493]
[New LWP 4494]
[New LWP 4495]
[New LWP 4496]
[New LWP 4497]
[New LWP 4498]
[New LWP 4499]
[New LWP 4500]
[New LWP 4501]
[New LWP 4502]
[New LWP 4503]
[New LWP 4504]
[New LWP 4505]
[New LWP 4506]
[New LWP 4507]
[New LWP 4509]
[New LWP 4510]
[New LWP 4511]
[New LWP 4512]
[New LWP 4513]
[New LWP 4514]
[New LWP 4515]
[New LWP 4516]
[New LWP 4517]
[New LWP 4521]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007f4856a19d50 in ?? ()
Id Target Id Frame
* 1 LWP 4383 "kudu" 0x00007f4856a19d50 in ?? ()
2 LWP 4386 "kudu" 0x00007f4856a15fb9 in ?? ()
3 LWP 4387 "kudu" 0x00007f4856a15fb9 in ?? ()
4 LWP 4388 "kudu" 0x00007f4856a15fb9 in ?? ()
5 LWP 4389 "kernel-watcher-" 0x00007f4856a15fb9 in ?? ()
6 LWP 4395 "ntp client-4395" 0x00007f4856a199e2 in ?? ()
7 LWP 4396 "file cache-evic" 0x00007f4856a15fb9 in ?? ()
8 LWP 4397 "sq_acceptor" 0x00007f484f73bcb9 in ?? ()
9 LWP 4400 "rpc reactor-440" 0x00007f484f748a47 in ?? ()
10 LWP 4401 "rpc reactor-440" 0x00007f484f748a47 in ?? ()
11 LWP 4402 "rpc reactor-440" 0x00007f484f748a47 in ?? ()
12 LWP 4403 "rpc reactor-440" 0x00007f484f748a47 in ?? ()
13 LWP 4404 "MaintenanceMgr " 0x00007f4856a15ad3 in ?? ()
14 LWP 4405 "txn-status-mana" 0x00007f4856a15fb9 in ?? ()
15 LWP 4406 "collect_and_rem" 0x00007f4856a15fb9 in ?? ()
16 LWP 4407 "tc-session-exp-" 0x00007f4856a15fb9 in ?? ()
17 LWP 4408 "rpc worker-4408" 0x00007f4856a15ad3 in ?? ()
18 LWP 4409 "rpc worker-4409" 0x00007f4856a15ad3 in ?? ()
19 LWP 4410 "rpc worker-4410" 0x00007f4856a15ad3 in ?? ()
20 LWP 4411 "rpc worker-4411" 0x00007f4856a15ad3 in ?? ()
21 LWP 4412 "rpc worker-4412" 0x00007f4856a15ad3 in ?? ()
22 LWP 4413 "rpc worker-4413" 0x00007f4856a15ad3 in ?? ()
23 LWP 4414 "rpc worker-4414" 0x00007f4856a15ad3 in ?? ()
24 LWP 4415 "rpc worker-4415" 0x00007f4856a15ad3 in ?? ()
25 LWP 4416 "rpc worker-4416" 0x00007f4856a15ad3 in ?? ()
26 LWP 4417 "rpc worker-4417" 0x00007f4856a15ad3 in ?? ()
27 LWP 4418 "rpc worker-4418" 0x00007f4856a15ad3 in ?? ()
28 LWP 4419 "rpc worker-4419" 0x00007f4856a15ad3 in ?? ()
29 LWP 4420 "rpc worker-4420" 0x00007f4856a15ad3 in ?? ()
30 LWP 4421 "rpc worker-4421" 0x00007f4856a15ad3 in ?? ()
31 LWP 4422 "rpc worker-4422" 0x00007f4856a15ad3 in ?? ()
32 LWP 4423 "rpc worker-4423" 0x00007f4856a15ad3 in ?? ()
33 LWP 4425 "rpc worker-4425" 0x00007f4856a15ad3 in ?? ()
34 LWP 4426 "rpc worker-4426" 0x00007f4856a15ad3 in ?? ()
35 LWP 4427 "rpc worker-4427" 0x00007f4856a15ad3 in ?? ()
36 LWP 4428 "rpc worker-4428" 0x00007f4856a15ad3 in ?? ()
37 LWP 4429 "rpc worker-4429" 0x00007f4856a15ad3 in ?? ()
38 LWP 4430 "rpc worker-4430" 0x00007f4856a15ad3 in ?? ()
39 LWP 4431 "rpc worker-4431" 0x00007f4856a15ad3 in ?? ()
40 LWP 4432 "rpc worker-4432" 0x00007f4856a15ad3 in ?? ()
41 LWP 4433 "rpc worker-4433" 0x00007f4856a15ad3 in ?? ()
42 LWP 4434 "rpc worker-4434" 0x00007f4856a15ad3 in ?? ()
43 LWP 4437 "rpc worker-4437" 0x00007f4856a15ad3 in ?? ()
44 LWP 4438 "rpc worker-4438" 0x00007f4856a15ad3 in ?? ()
45 LWP 4440 "rpc worker-4440" 0x00007f4856a15ad3 in ?? ()
46 LWP 4441 "rpc worker-4441" 0x00007f4856a15ad3 in ?? ()
47 LWP 4442 "rpc worker-4442" 0x00007f4856a15ad3 in ?? ()
48 LWP 4443 "rpc worker-4443" 0x00007f4856a15ad3 in ?? ()
49 LWP 4444 "rpc worker-4444" 0x00007f4856a15ad3 in ?? ()
50 LWP 4445 "rpc worker-4445" 0x00007f4856a15ad3 in ?? ()
51 LWP 4446 "rpc worker-4446" 0x00007f4856a15ad3 in ?? ()
52 LWP 4447 "rpc worker-4447" 0x00007f4856a15ad3 in ?? ()
53 LWP 4448 "rpc worker-4448" 0x00007f4856a15ad3 in ?? ()
54 LWP 4449 "rpc worker-4449" 0x00007f4856a15ad3 in ?? ()
55 LWP 4450 "rpc worker-4450" 0x00007f4856a15ad3 in ?? ()
56 LWP 4451 "rpc worker-4451" 0x00007f4856a15ad3 in ?? ()
57 LWP 4452 "rpc worker-4452" 0x00007f4856a15ad3 in ?? ()
58 LWP 4453 "rpc worker-4453" 0x00007f4856a15ad3 in ?? ()
59 LWP 4454 "rpc worker-4454" 0x00007f4856a15ad3 in ?? ()
60 LWP 4455 "rpc worker-4455" 0x00007f4856a15ad3 in ?? ()
61 LWP 4456 "rpc worker-4456" 0x00007f4856a15ad3 in ?? ()
62 LWP 4457 "rpc worker-4457" 0x00007f4856a15ad3 in ?? ()
63 LWP 4458 "rpc worker-4458" 0x00007f4856a15ad3 in ?? ()
64 LWP 4459 "rpc worker-4459" 0x00007f4856a15ad3 in ?? ()
65 LWP 4460 "rpc worker-4460" 0x00007f4856a15ad3 in ?? ()
66 LWP 4461 "rpc worker-4461" 0x00007f4856a15ad3 in ?? ()
67 LWP 4462 "rpc worker-4462" 0x00007f4856a15ad3 in ?? ()
68 LWP 4463 "rpc worker-4463" 0x00007f4856a15ad3 in ?? ()
69 LWP 4464 "rpc worker-4464" 0x00007f4856a15ad3 in ?? ()
70 LWP 4465 "rpc worker-4465" 0x00007f4856a15ad3 in ?? ()
71 LWP 4466 "rpc worker-4466" 0x00007f4856a15ad3 in ?? ()
72 LWP 4467 "rpc worker-4467" 0x00007f4856a15ad3 in ?? ()
73 LWP 4468 "rpc worker-4468" 0x00007f4856a15ad3 in ?? ()
74 LWP 4469 "rpc worker-4469" 0x00007f4856a15ad3 in ?? ()
75 LWP 4470 "rpc worker-4470" 0x00007f4856a15ad3 in ?? ()
76 LWP 4471 "rpc worker-4471" 0x00007f4856a15ad3 in ?? ()
77 LWP 4472 "rpc worker-4472" 0x00007f4856a15ad3 in ?? ()
78 LWP 4473 "rpc worker-4473" 0x00007f4856a15ad3 in ?? ()
79 LWP 4474 "rpc worker-4474" 0x00007f4856a15ad3 in ?? ()
80 LWP 4475 "rpc worker-4475" 0x00007f4856a15ad3 in ?? ()
81 LWP 4476 "rpc worker-4476" 0x00007f4856a15ad3 in ?? ()
82 LWP 4477 "rpc worker-4477" 0x00007f4856a15ad3 in ?? ()
83 LWP 4478 "rpc worker-4478" 0x00007f4856a15ad3 in ?? ()
84 LWP 4479 "rpc worker-4479" 0x00007f4856a15ad3 in ?? ()
85 LWP 4480 "rpc worker-4480" 0x00007f4856a15ad3 in ?? ()
86 LWP 4481 "rpc worker-4481" 0x00007f4856a15ad3 in ?? ()
87 LWP 4482 "rpc worker-4482" 0x00007f4856a15ad3 in ?? ()
88 LWP 4483 "rpc worker-4483" 0x00007f4856a15ad3 in ?? ()
89 LWP 4484 "rpc worker-4484" 0x00007f4856a15ad3 in ?? ()
90 LWP 4485 "rpc worker-4485" 0x00007f4856a15ad3 in ?? ()
91 LWP 4486 "rpc worker-4486" 0x00007f4856a15ad3 in ?? ()
92 LWP 4487 "rpc worker-4487" 0x00007f4856a15ad3 in ?? ()
93 LWP 4488 "rpc worker-4488" 0x00007f4856a15ad3 in ?? ()
94 LWP 4489 "rpc worker-4489" 0x00007f4856a15ad3 in ?? ()
95 LWP 4490 "rpc worker-4490" 0x00007f4856a15ad3 in ?? ()
96 LWP 4491 "rpc worker-4491" 0x00007f4856a15ad3 in ?? ()
97 LWP 4492 "rpc worker-4492" 0x00007f4856a15ad3 in ?? ()
98 LWP 4493 "rpc worker-4493" 0x00007f4856a15ad3 in ?? ()
99 LWP 4494 "rpc worker-4494" 0x00007f4856a15ad3 in ?? ()
100 LWP 4495 "rpc worker-4495" 0x00007f4856a15ad3 in ?? ()
101 LWP 4496 "rpc worker-4496" 0x00007f4856a15ad3 in ?? ()
102 LWP 4497 "rpc worker-4497" 0x00007f4856a15ad3 in ?? ()
103 LWP 4498 "rpc worker-4498" 0x00007f4856a15ad3 in ?? ()
104 LWP 4499 "rpc worker-4499" 0x00007f4856a15ad3 in ?? ()
105 LWP 4500 "rpc worker-4500" 0x00007f4856a15ad3 in ?? ()
106 LWP 4501 "rpc worker-4501" 0x00007f4856a15ad3 in ?? ()
107 LWP 4502 "rpc worker-4502" 0x00007f4856a15ad3 in ?? ()
108 LWP 4503 "rpc worker-4503" 0x00007f4856a15ad3 in ?? ()
109 LWP 4504 "rpc worker-4504" 0x00007f4856a15ad3 in ?? ()
110 LWP 4505 "rpc worker-4505" 0x00007f4856a15ad3 in ?? ()
111 LWP 4506 "rpc worker-4506" 0x00007f4856a15ad3 in ?? ()
112 LWP 4507 "rpc worker-4507" 0x00007f4856a15ad3 in ?? ()
113 LWP 4509 "rpc worker-4509" 0x00007f4856a15ad3 in ?? ()
114 LWP 4510 "rpc worker-4510" 0x00007f4856a15ad3 in ?? ()
115 LWP 4511 "rpc worker-4511" 0x00007f4856a15ad3 in ?? ()
116 LWP 4512 "rpc worker-4512" 0x00007f4856a15ad3 in ?? ()
117 LWP 4513 "diag-logger-451" 0x00007f4856a15fb9 in ?? ()
118 LWP 4514 "result-tracker-" 0x00007f4856a15fb9 in ?? ()
119 LWP 4515 "excess-log-dele" 0x00007f4856a15fb9 in ?? ()
120 LWP 4516 "acceptor-4516" 0x00007f484f74a0c7 in ?? ()
121 LWP 4517 "heartbeat-4517" 0x00007f4856a15fb9 in ?? ()
122 LWP 4521 "maintenance_sch" 0x00007f4856a15fb9 in ?? ()
Thread 122 (LWP 4521):
#0 0x00007f4856a15fb9 in ?? ()
#1 0x00007f480c2bd6e0 in ?? ()
#2 0x000000000000022e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f480c2bd730 in ?? ()
#6 0x000000000000045c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 4517):
#0 0x00007f4856a15fb9 in ?? ()
#1 0x8ea7c1ad18c34923 in ?? ()
#2 0x000000000000008b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019400 in ?? ()
#5 0x00007f480cad55f0 in ?? ()
#6 0x0000000000000116 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 4516):
#0 0x00007f484f74a0c7 in ?? ()
#1 0x00006140000216a8 in ?? ()
#2 0x00007f480d2fcd90 in ?? ()
#3 0x00007f480d2fcdc0 in ?? ()
#4 0x00007f480d2fcec0 in ?? ()
#5 0x00007f480d2fcdb0 in ?? ()
#6 0x00007f480d2fce20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x00007f484be2ec88 in ?? ()
#10 0x000000007b800066 in ?? ()
#11 0x000000000000000d in ?? ()
#12 0x000000000000000d in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 119 (LWP 4515):
#0 0x00007f4856a15fb9 in ?? ()
#1 0x00007f480db16f40 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff7afd81b0 in ?? ()
#5 0x00007f480db16f30 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 4514):
#0 0x00007f4856a15fb9 in ?? ()
#1 0x00007f480e32f100 in ?? ()
#2 0x000000000000008b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f480e32f0f0 in ?? ()
#6 0x0000000000000116 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 4513):
#0 0x00007f4856a15fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 4512):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000013 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007f480f35ee90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f480f35eeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007f4856a15770 in ?? ()
#10 0x00007f480f35eeb0 in ?? ()
#11 0x00000fe901e6bdd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 4511):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000021 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007f480fb76e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f480fb76eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007f4856a15770 in ?? ()
#10 0x00007f480fb76eb0 in ?? ()
#11 0x00000fe901f6edd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 4510):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 4509):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 4507):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 4506):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 4505):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 4504):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 4503):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 4502):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 4501):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 4500):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 4499):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 4498):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 4497):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 4496):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 4495):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 4494):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 4493):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 4492):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 4491):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 4490):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 4489):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 4488):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 4487):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 4486):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 4485):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 4484):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 4483):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 4482):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 4481):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 4480):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 4479):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 4478):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 4477):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 4476):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 4475):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 4474):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 4473):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 4472):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 4471):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x000000000000000d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a688c in ?? ()
#4 0x00007f482372fe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f482372feb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a6840 in ?? ()
#9 0x00007f4856a15770 in ?? ()
#10 0x00007f482372feb0 in ?? ()
#11 0x00000fe9046e5fd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 4470):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 4469):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 4468):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 4467):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 4466):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 4465):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 4464):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 4463):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 4462):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 4461):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 4460):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 4459):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 4458):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 4457):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 4456):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 4455):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 4454):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 4453):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 4452):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 4451):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x000000000000000d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012003c in ?? ()
#4 0x00007f482d910e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f482d910eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00011fff0 in ?? ()
#9 0x00007f4856a15770 in ?? ()
#10 0x00007f482d910eb0 in ?? ()
#11 0x00000fe905b221d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 4450):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 4449):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 4448):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 4447):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 4446):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 4445):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 4444):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 4443):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 4442):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 4441):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 4440):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 4438):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 4437):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 4434):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 4433):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 4432):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 4431):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 4430):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 4429):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 4428):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x000000000000000d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007f48372d9e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f48372d9eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007f4856a15770 in ?? ()
#10 0x00007f48372d9eb0 in ?? ()
#11 0x00000fe906e5b3d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 4427):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 4426):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 4425):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 4423):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 4422):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 4421):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 4420):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 4419):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 4418):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 4417):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 4416):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 4415):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 4414):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 4413):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 4412):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 4411):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 4410):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 4409):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 4408):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 4407):
#0 0x00007f4856a15fb9 in ?? ()
#1 0x00007f48414decc0 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019ca0 in ?? ()
#5 0x00007f48414decb0 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 4406):
#0 0x00007f4856a15fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 4405):
#0 0x00007f4856a15fb9 in ?? ()
#1 0x00007f484251e240 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007f484251e230 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 4404):
#0 0x00007f4856a15ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 4403):
#0 0x00007f484f748a47 in ?? ()
#1 0x00007f484356e320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f484356e310 in ?? ()
#4 0x00007f484356e520 in ?? ()
#5 0x00007f484356e360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f484356e3e0 in ?? ()
#8 0x00007f4851dbb25d in ?? ()
#9 0x3fb98b1739bf8000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x0000000084000000 in ?? ()
#14 0x00007f4800000000 in ?? ()
#15 0x41da16179c5dd6b3 in ?? ()
#16 0x00000fe9886a5c7c in ?? ()
#17 0x00007f484356e3c0 in ?? ()
#18 0x00007f4851dbfba3 in ?? ()
#19 0x00007f484356e390 in ?? ()
#20 0x3fb98b1739bf8000 in ?? ()
#21 0x000000004356e3e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb98b1739bf8000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 4402):
#0 0x00007f484f748a47 in ?? ()
#1 0x00007f4843d85320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f4843d85310 in ?? ()
#4 0x00007f4843d85520 in ?? ()
#5 0x00007f4843d85360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f4843d853e0 in ?? ()
#8 0x00007f4851dbb25d in ?? ()
#9 0x3fb98a11a29e4000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x000000001f000000 in ?? ()
#14 0x00007f4800000000 in ?? ()
#15 0x41da16179c5dd6b7 in ?? ()
#16 0x00000fe9887a8a7c in ?? ()
#17 0x00007f4843d853c0 in ?? ()
#18 0x00007f4851dbfba3 in ?? ()
#19 0x00007f4843d85390 in ?? ()
#20 0x3fb98a11a29e4000 in ?? ()
#21 0x0000000043d853e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb98a11a29e4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 4401):
#0 0x00007f484f748a47 in ?? ()
#1 0x00007f484459c320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f484459c310 in ?? ()
#4 0x00007f484459c520 in ?? ()
#5 0x00007f484459c360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f484459c3e0 in ?? ()
#8 0x00007f4851dbb25d in ?? ()
#9 0x3fb97bd0bde2c000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x000000007e000000 in ?? ()
#14 0x00007f4800000000 in ?? ()
#15 0x41da16179c5dd6b5 in ?? ()
#16 0x00000fe9888ab87c in ?? ()
#17 0x00007f484459c3c0 in ?? ()
#18 0x00007f4851dbfba3 in ?? ()
#19 0x00007f484459c390 in ?? ()
#20 0x3fb97bd0bde2c000 in ?? ()
#21 0x000000004459c3e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb97bd0bde2c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 4400):
#0 0x00007f484f748a47 in ?? ()
#1 0x00007f48465ad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f48465ad310 in ?? ()
#4 0x00007f48465ad520 in ?? ()
#5 0x00007f48465ad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007f48465ad3e0 in ?? ()
#8 0x00007f4851dbb25d in ?? ()
#9 0x3fb97fe629e35000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x000000001d000000 in ?? ()
#14 0x00007f4800000000 in ?? ()
#15 0x41da16179c5dd6b5 in ?? ()
#16 0x00000fe988cada7c in ?? ()
#17 0x00007f48465ad3c0 in ?? ()
#18 0x00007f4851dbfba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 4397):
#0 0x00007f484f73bcb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f4847fb77c8 in ?? ()
#3 0x000060200001f510 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 4396):
#0 0x00007f4856a15fb9 in ?? ()
#1 0x00007f48477b60c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007f48477b60b0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 4395):
#0 0x00007f4856a199e2 in ?? ()
#1 0x00007f4846fb3ba0 in ?? ()
#2 0x00007f4846fb3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f4846fb3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007f4846fb3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f4846fb3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007f485c32cedc in ?? ()
#12 0x00007f485c31c8d9 in ?? ()
#13 0x00007f485c320ec6 in ?? ()
#14 0x00007f485c325900 in ?? ()
#15 0x00007f485c325729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007f48535f14fe in ?? ()
#18 0x00007f4856a0f6db in ?? ()
#19 0x00000fe988dee684 in ?? ()
#20 0x00007f4846fb3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 4389):
#0 0x00007f4856a15fb9 in ?? ()
#1 0x00007f4848fb8c80 in ?? ()
#2 0x00000000000002be in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007f4848fb8c70 in ?? ()
#6 0x000000000000057c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 4388):
#0 0x00007f4856a15fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 4387):
#0 0x00007f4856a15fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 4386):
#0 0x00007f4856a15fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f484a7bc450 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 4383):
#0 0x00007f4856a19d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250622 19:58:09.677430 2016 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID 54555a8ecf924b339db8e8306cbc8bb4 and pid 4526
************************ BEGIN STACKS **************************
[New LWP 4530]
[New LWP 4531]
[New LWP 4532]
[New LWP 4533]
[New LWP 4540]
[New LWP 4541]
[New LWP 4542]
[New LWP 4545]
[New LWP 4546]
[New LWP 4547]
[New LWP 4548]
[New LWP 4549]
[New LWP 4550]
[New LWP 4552]
[New LWP 4553]
[New LWP 4554]
[New LWP 4555]
[New LWP 4556]
[New LWP 4557]
[New LWP 4558]
[New LWP 4559]
[New LWP 4560]
[New LWP 4561]
[New LWP 4562]
[New LWP 4563]
[New LWP 4565]
[New LWP 4566]
[New LWP 4567]
[New LWP 4568]
[New LWP 4569]
[New LWP 4570]
[New LWP 4571]
[New LWP 4572]
[New LWP 4573]
[New LWP 4574]
[New LWP 4575]
[New LWP 4576]
[New LWP 4577]
[New LWP 4578]
[New LWP 4579]
[New LWP 4580]
[New LWP 4581]
[New LWP 4582]
[New LWP 4583]
[New LWP 4584]
[New LWP 4585]
[New LWP 4586]
[New LWP 4587]
[New LWP 4588]
[New LWP 4589]
[New LWP 4590]
[New LWP 4591]
[New LWP 4592]
[New LWP 4593]
[New LWP 4594]
[New LWP 4595]
[New LWP 4596]
[New LWP 4597]
[New LWP 4598]
[New LWP 4599]
[New LWP 4600]
[New LWP 4601]
[New LWP 4602]
[New LWP 4603]
[New LWP 4604]
[New LWP 4605]
[New LWP 4606]
[New LWP 4607]
[New LWP 4608]
[New LWP 4609]
[New LWP 4610]
[New LWP 4611]
[New LWP 4612]
[New LWP 4613]
[New LWP 4614]
[New LWP 4615]
[New LWP 4616]
[New LWP 4617]
[New LWP 4618]
[New LWP 4619]
[New LWP 4621]
[New LWP 4622]
[New LWP 4623]
[New LWP 4624]
[New LWP 4625]
[New LWP 4626]
[New LWP 4627]
[New LWP 4628]
[New LWP 4629]
[New LWP 4630]
[New LWP 4631]
[New LWP 4632]
[New LWP 4633]
[New LWP 4634]
[New LWP 4635]
[New LWP 4636]
[New LWP 4637]
[New LWP 4638]
[New LWP 4639]
[New LWP 4640]
[New LWP 4641]
[New LWP 4642]
[New LWP 4643]
[New LWP 4644]
[New LWP 4645]
[New LWP 4646]
[New LWP 4647]
[New LWP 4648]
[New LWP 4649]
[New LWP 4650]
[New LWP 4651]
[New LWP 4652]
[New LWP 4653]
[New LWP 4654]
[New LWP 4655]
[New LWP 4656]
[New LWP 4657]
[New LWP 4658]
[New LWP 4659]
[New LWP 4660]
[New LWP 4663]
[New LWP 6288]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007fd6217d8d50 in ?? ()
Id Target Id Frame
* 1 LWP 4526 "kudu" 0x00007fd6217d8d50 in ?? ()
2 LWP 4530 "kudu" 0x00007fd6217d4fb9 in ?? ()
3 LWP 4531 "kudu" 0x00007fd6217d4fb9 in ?? ()
4 LWP 4532 "kudu" 0x00007fd6217d4fb9 in ?? ()
5 LWP 4533 "kernel-watcher-" 0x00007fd6217d4fb9 in ?? ()
6 LWP 4540 "ntp client-4540" 0x00007fd6217d89e2 in ?? ()
7 LWP 4541 "file cache-evic" 0x00007fd6217d4fb9 in ?? ()
8 LWP 4542 "sq_acceptor" 0x00007fd61a4facb9 in ?? ()
9 LWP 4545 "rpc reactor-454" 0x00007fd61a507a47 in ?? ()
10 LWP 4546 "rpc reactor-454" 0x00007fd61a507a47 in ?? ()
11 LWP 4547 "rpc reactor-454" 0x00007fd61a507a47 in ?? ()
12 LWP 4548 "rpc reactor-454" 0x00007fd61a507a47 in ?? ()
13 LWP 4549 "MaintenanceMgr " 0x00007fd62b232931 in ?? ()
14 LWP 4550 "txn-status-mana" 0x00007fd6217d4fb9 in ?? ()
15 LWP 4552 "collect_and_rem" 0x00007fd6217d4fb9 in ?? ()
16 LWP 4553 "tc-session-exp-" 0x00007fd6217d4fb9 in ?? ()
17 LWP 4554 "rpc worker-4554" 0x00007fd6217d4ad3 in ?? ()
18 LWP 4555 "rpc worker-4555" 0x00007fd6217d4ad3 in ?? ()
19 LWP 4556 "rpc worker-4556" 0x00007fd6217d4ad3 in ?? ()
20 LWP 4557 "rpc worker-4557" 0x00007fd6217d4ad3 in ?? ()
21 LWP 4558 "rpc worker-4558" 0x00007fd6217d4ad3 in ?? ()
22 LWP 4559 "rpc worker-4559" 0x00007fd6217d4ad3 in ?? ()
23 LWP 4560 "rpc worker-4560" 0x00007fd6217d4ad3 in ?? ()
24 LWP 4561 "rpc worker-4561" 0x00007fd6217d4ad3 in ?? ()
25 LWP 4562 "rpc worker-4562" 0x00007fd6217d4ad3 in ?? ()
26 LWP 4563 "rpc worker-4563" 0x00007fd6217d4ad3 in ?? ()
27 LWP 4565 "rpc worker-4565" 0x00007fd6217d4ad3 in ?? ()
28 LWP 4566 "rpc worker-4566" 0x00007fd6217d4ad3 in ?? ()
29 LWP 4567 "rpc worker-4567" 0x00007fd6217d4ad3 in ?? ()
30 LWP 4568 "rpc worker-4568" 0x00007fd6217d4ad3 in ?? ()
31 LWP 4569 "rpc worker-4569" 0x00007fd6217d4ad3 in ?? ()
32 LWP 4570 "rpc worker-4570" 0x00007fd6217d4ad3 in ?? ()
33 LWP 4571 "rpc worker-4571" 0x00007fd6217d4ad3 in ?? ()
34 LWP 4572 "rpc worker-4572" 0x00007fd6217d4ad3 in ?? ()
35 LWP 4573 "rpc worker-4573" 0x00007fd6217d4ad3 in ?? ()
36 LWP 4574 "rpc worker-4574" 0x00007fd6217d4ad3 in ?? ()
37 LWP 4575 "rpc worker-4575" 0x00007fd6217d4ad3 in ?? ()
38 LWP 4576 "rpc worker-4576" 0x00007fd6217d4ad3 in ?? ()
39 LWP 4577 "rpc worker-4577" 0x00007fd6217d4ad3 in ?? ()
40 LWP 4578 "rpc worker-4578" 0x00007fd6217d4ad3 in ?? ()
41 LWP 4579 "rpc worker-4579" 0x00007fd6217d4ad3 in ?? ()
42 LWP 4580 "rpc worker-4580" 0x00007fd6217d4ad3 in ?? ()
43 LWP 4581 "rpc worker-4581" 0x00007fd6217d4ad3 in ?? ()
44 LWP 4582 "rpc worker-4582" 0x00007fd6217d4ad3 in ?? ()
45 LWP 4583 "rpc worker-4583" 0x00007fd6217d4ad3 in ?? ()
46 LWP 4584 "rpc worker-4584" 0x00007fd6217d4ad3 in ?? ()
47 LWP 4585 "rpc worker-4585" 0x00007fd6217d4ad3 in ?? ()
48 LWP 4586 "rpc worker-4586" 0x00007fd6217d4ad3 in ?? ()
49 LWP 4587 "rpc worker-4587" 0x00007fd6217d4ad3 in ?? ()
50 LWP 4588 "rpc worker-4588" 0x00007fd6217d4ad3 in ?? ()
51 LWP 4589 "rpc worker-4589" 0x00007fd6217d4ad3 in ?? ()
52 LWP 4590 "rpc worker-4590" 0x00007fd6217d4ad3 in ?? ()
53 LWP 4591 "rpc worker-4591" 0x00007fd6217d4ad3 in ?? ()
54 LWP 4592 "rpc worker-4592" 0x00007fd6217d4ad3 in ?? ()
55 LWP 4593 "rpc worker-4593" 0x00007fd6217d4ad3 in ?? ()
56 LWP 4594 "rpc worker-4594" 0x00007fd6217d4ad3 in ?? ()
57 LWP 4595 "rpc worker-4595" 0x00007fd6217d4ad3 in ?? ()
58 LWP 4596 "rpc worker-4596" 0x00007fd6217d4ad3 in ?? ()
59 LWP 4597 "rpc worker-4597" 0x00007fd6217d4ad3 in ?? ()
60 LWP 4598 "rpc worker-4598" 0x00007fd6217d4ad3 in ?? ()
61 LWP 4599 "rpc worker-4599" 0x00007fd6217d4ad3 in ?? ()
62 LWP 4600 "rpc worker-4600" 0x00007fd6217d4ad3 in ?? ()
63 LWP 4601 "rpc worker-4601" 0x00007fd6217d4ad3 in ?? ()
64 LWP 4602 "rpc worker-4602" 0x00007fd6217d4ad3 in ?? ()
65 LWP 4603 "rpc worker-4603" 0x00007fd6217d4ad3 in ?? ()
66 LWP 4604 "rpc worker-4604" 0x00007fd6217d4ad3 in ?? ()
67 LWP 4605 "rpc worker-4605" 0x00007fd6217d4ad3 in ?? ()
68 LWP 4606 "rpc worker-4606" 0x00007fd6217d4ad3 in ?? ()
69 LWP 4607 "rpc worker-4607" 0x00007fd6217d4ad3 in ?? ()
70 LWP 4608 "rpc worker-4608" 0x00007fd6217d4ad3 in ?? ()
71 LWP 4609 "rpc worker-4609" 0x00007fd6217d4ad3 in ?? ()
72 LWP 4610 "rpc worker-4610" 0x00007fd6217d4ad3 in ?? ()
73 LWP 4611 "rpc worker-4611" 0x00007fd6217d4ad3 in ?? ()
74 LWP 4612 "rpc worker-4612" 0x00007fd6217d4ad3 in ?? ()
75 LWP 4613 "rpc worker-4613" 0x00007fd6217d4ad3 in ?? ()
76 LWP 4614 "rpc worker-4614" 0x00007fd6217d4ad3 in ?? ()
77 LWP 4615 "rpc worker-4615" 0x00007fd6217d4ad3 in ?? ()
78 LWP 4616 "rpc worker-4616" 0x00007fd6217d4ad3 in ?? ()
79 LWP 4617 "rpc worker-4617" 0x00007fd6217d4ad3 in ?? ()
80 LWP 4618 "rpc worker-4618" 0x00007fd6217d4ad3 in ?? ()
81 LWP 4619 "rpc worker-4619" 0x00007fd6217d4ad3 in ?? ()
82 LWP 4621 "rpc worker-4621" 0x00007fd6217d4ad3 in ?? ()
83 LWP 4622 "rpc worker-4622" 0x00007fd6217d4ad3 in ?? ()
84 LWP 4623 "rpc worker-4623" 0x00007fd6217d4ad3 in ?? ()
85 LWP 4624 "rpc worker-4624" 0x00007fd6217d4ad3 in ?? ()
86 LWP 4625 "rpc worker-4625" 0x00007fd6217d4ad3 in ?? ()
87 LWP 4626 "rpc worker-4626" 0x00007fd6217d4ad3 in ?? ()
88 LWP 4627 "rpc worker-4627" 0x00007fd6217d4ad3 in ?? ()
89 LWP 4628 "rpc worker-4628" 0x00007fd6217d4ad3 in ?? ()
90 LWP 4629 "rpc worker-4629" 0x00007fd6217d4ad3 in ?? ()
91 LWP 4630 "rpc worker-4630" 0x00007fd6217d4ad3 in ?? ()
92 LWP 4631 "rpc worker-4631" 0x00007fd6217d4ad3 in ?? ()
93 LWP 4632 "rpc worker-4632" 0x00007fd6217d4ad3 in ?? ()
94 LWP 4633 "rpc worker-4633" 0x00007fd6217d4ad3 in ?? ()
95 LWP 4634 "rpc worker-4634" 0x00007fd6217d4ad3 in ?? ()
96 LWP 4635 "rpc worker-4635" 0x00007fd6217d4ad3 in ?? ()
97 LWP 4636 "rpc worker-4636" 0x00007fd6217d4ad3 in ?? ()
98 LWP 4637 "rpc worker-4637" 0x00007fd6217d4ad3 in ?? ()
99 LWP 4638 "rpc worker-4638" 0x00007fd6217d4ad3 in ?? ()
100 LWP 4639 "rpc worker-4639" 0x00007fd6217d4ad3 in ?? ()
101 LWP 4640 "rpc worker-4640" 0x00007fd6217d4ad3 in ?? ()
102 LWP 4641 "rpc worker-4641" 0x00007fd6217d4ad3 in ?? ()
103 LWP 4642 "rpc worker-4642" 0x00007fd6217d4ad3 in ?? ()
104 LWP 4643 "rpc worker-4643" 0x00007fd6217d4ad3 in ?? ()
105 LWP 4644 "rpc worker-4644" 0x00007fd6217d4ad3 in ?? ()
106 LWP 4645 "rpc worker-4645" 0x00007fd6217d4ad3 in ?? ()
107 LWP 4646 "rpc worker-4646" 0x00007fd6217d4ad3 in ?? ()
108 LWP 4647 "rpc worker-4647" 0x00007fd6217d4ad3 in ?? ()
109 LWP 4648 "rpc worker-4648" 0x00007fd6217d4ad3 in ?? ()
110 LWP 4649 "rpc worker-4649" 0x00007fd6217d4ad3 in ?? ()
111 LWP 4650 "rpc worker-4650" 0x00007fd6217d4ad3 in ?? ()
112 LWP 4651 "rpc worker-4651" 0x00007fd6217d4ad3 in ?? ()
113 LWP 4652 "rpc worker-4652" 0x00007fd6217d4ad3 in ?? ()
114 LWP 4653 "rpc worker-4653" 0x00007fd6217d4ad3 in ?? ()
115 LWP 4654 "rpc worker-4654" 0x00007fd6217d4ad3 in ?? ()
116 LWP 4655 "rpc worker-4655" 0x00007fd6217d4ad3 in ?? ()
117 LWP 4656 "diag-logger-465" 0x00007fd6217d4fb9 in ?? ()
118 LWP 4657 "result-tracker-" 0x00007fd6217d4fb9 in ?? ()
119 LWP 4658 "excess-log-dele" 0x00007fd6217d4fb9 in ?? ()
120 LWP 4659 "acceptor-4659" 0x00007fd61a5090c7 in ?? ()
121 LWP 4660 "heartbeat-4660" 0x00007fd6217d4fb9 in ?? ()
122 LWP 4663 "maintenance_sch" 0x00007fd6217d4fb9 in ?? ()
123 LWP 6288 "raft [worker]-6" 0x00007fd6217d4fb9 in ?? ()
Thread 123 (LWP 6288):
#0 0x00007fd6217d4fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000009d in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00007fd5cb89cba4 in ?? ()
#5 0x00007fd5cb89c830 in ?? ()
#6 0x000000000000013b in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fd5cb89c850 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 122 (LWP 4663):
#0 0x00007fd6217d4fb9 in ?? ()
#1 0x00007fd5d600c6e0 in ?? ()
#2 0x0000000000000229 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007fd5d600c730 in ?? ()
#6 0x0000000000000452 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 4660):
#0 0x00007fd6217d4fb9 in ?? ()
#1 0x8ea7c1ad18c34923 in ?? ()
#2 0x000000000000008d in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x0000613000019404 in ?? ()
#5 0x00007fd5d703c5f0 in ?? ()
#6 0x000000000000011b in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fd5d703c610 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007fd5d703c690 in ?? ()
#12 0x00000ffabae078d4 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 120 (LWP 4659):
#0 0x00007fd61a5090c7 in ?? ()
#1 0x00006140000210a8 in ?? ()
#2 0x00007fd5d7863d90 in ?? ()
#3 0x00007fd5d7863dc0 in ?? ()
#4 0x00007fd5d7863ec0 in ?? ()
#5 0x00007fd5d7863db0 in ?? ()
#6 0x00007fd5d7863e20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x00007fd61e38f980 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 4658):
#0 0x00007fd6217d4fb9 in ?? ()
#1 0x00007fd5d807cf40 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff3c669230 in ?? ()
#5 0x00007fd5d807cf30 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 4657):
#0 0x00007fd6217d4fb9 in ?? ()
#1 0x00007fd5d8896100 in ?? ()
#2 0x000000000000008a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007fd5d88960f0 in ?? ()
#6 0x0000000000000114 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 4656):
#0 0x00007fd6217d4fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 4655):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b6848 in ?? ()
#4 0x00007fd5d98c5e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd5d98c5eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 4654):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x000000000000001c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007fd5da0dde90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd5da0ddeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 4653):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 4652):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 4651):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 4650):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 4649):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 4648):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 4647):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 4646):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 4645):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 4644):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 4643):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 4642):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 4641):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 4640):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 4639):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 4638):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 4637):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 4636):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 4635):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 4634):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 4633):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 4632):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 4631):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 4630):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 4629):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 4628):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 4627):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 4626):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 4625):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 4624):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 4623):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 4622):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 4621):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 4619):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 4618):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 4617):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 4616):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 4615):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 4614):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000672 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007fd5edc96e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd5edc96eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 4613):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x00000000000003e4 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a6888 in ?? ()
#4 0x00007fd5ee4aee90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd5ee4aeeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 4612):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x000000000000021c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a0088 in ?? ()
#4 0x00007fd5eecc6e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd5eecc6eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 73 (LWP 4611):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 4610):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 4609):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 4608):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 4607):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 4606):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 4605):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 4604):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 4603):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 4602):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 4601):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 4600):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 4599):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 4598):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 4597):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 4596):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 4595):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 4594):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x000000000000000d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007fd5f765fe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd5f765feb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007fd6217d4770 in ?? ()
#10 0x00007fd5f765feb0 in ?? ()
#11 0x00000ffabeecbfd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 4593):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 4592):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 4591):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 4590):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 4589):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 4588):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 4587):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 4586):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 4585):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 4584):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 4583):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 4582):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 4581):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 4580):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 4579):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 4578):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 4577):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 4576):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 4575):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 4574):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000a57 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007fd601840e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd601840eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007fd6217d4770 in ?? ()
#10 0x00007fd601840eb0 in ?? ()
#11 0x00000ffac03081d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 4573):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x000000000000097c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00009ffe8 in ?? ()
#4 0x00007fd602058e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd602058eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 34 (LWP 4572):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x000000000000088d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000967fc in ?? ()
#4 0x00007fd602870e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd602870eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000967b0 in ?? ()
#9 0x00007fd6217d4770 in ?? ()
#10 0x00007fd602870eb0 in ?? ()
#11 0x00000ffac050e1d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 33 (LWP 4571):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 4570):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 4569):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 4568):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 4567):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 4566):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 4565):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 4563):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 4562):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 4561):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 4560):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 4559):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 4558):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 4557):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 4556):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 4555):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 4554):
#0 0x00007fd6217d4ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 4553):
#0 0x00007fd6217d4fb9 in ?? ()
#1 0x00007fd60ba45cc0 in ?? ()
#2 0x000000000000000d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a3a0 in ?? ()
#5 0x00007fd60ba45cb0 in ?? ()
#6 0x000000000000001a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 4552):
#0 0x00007fd6217d4fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 4550):
#0 0x00007fd6217d4fb9 in ?? ()
#1 0x00007fd60d2af240 in ?? ()
#2 0x000000000000000d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007fd60d2af230 in ?? ()
#6 0x000000000000001a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 4549):
#0 0x00007fd62b232931 in ?? ()
#1 0x00007fd60daad1f0 in ?? ()
#2 0x00007fd62b1dda32 in ?? ()
#3 0x0000000041b58ab3 in ?? ()
#4 0x00007fd62ab11f52 in ?? ()
#5 0x00007fd62b1dd8f0 in ?? ()
#6 0x00007fd60daac6c0 in ?? ()
#7 0x00007fd60daac630 in ?? ()
#8 0x00007fd60daac140 in ?? ()
#9 0x00007fd60daac0b0 in ?? ()
#10 0x00007fd60daac750 in ?? ()
#11 0x00007fd60daac540 in ?? ()
#12 0x00007fd60daabfc0 in ?? ()
#13 0x00007fd60daacb70 in ?? ()
#14 0x00007fd60daacae0 in ?? ()
#15 0x00007fd60daacac0 in ?? ()
#16 0x00007fd60daac3e0 in ?? ()
#17 0x00007fd60daac350 in ?? ()
#18 0x00007fd60daac330 in ?? ()
#19 0x00007fd60daad530 in ?? ()
#20 0x00007fd60daabea0 in ?? ()
#21 0x00007fd60daabe80 in ?? ()
#22 0x00007fd60daac478 in ?? ()
#23 0x0000000000000000 in ?? ()
Thread 12 (LWP 4548):
#0 0x00007fd61a507a47 in ?? ()
#1 0x00007fd60e2b2320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007fd60e2b2310 in ?? ()
#4 0x00007fd60e2b2520 in ?? ()
#5 0x00007fd60e2b2360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fd60e2b23e0 in ?? ()
#8 0x00007fd61cb7a25d in ?? ()
#9 0x3fa6eb652d1a4000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x0000000074000000 in ?? ()
#14 0x00007fd600000000 in ?? ()
#15 0x41da16179c5dd6b6 in ?? ()
#16 0x00000ffb41c4e47c in ?? ()
#17 0x00007fd60e2b23c0 in ?? ()
#18 0x00007fd61cb7eba3 in ?? ()
#19 0x00007fd60e2b2390 in ?? ()
#20 0x3fa6eb652d1a4000 in ?? ()
#21 0x000000000e2b23e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fa6eb652d1a4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 4547):
#0 0x00007fd61a507a47 in ?? ()
#1 0x00007fd60eab3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007fd60eab3310 in ?? ()
#4 0x00007fd60eab3520 in ?? ()
#5 0x00007fd60eab3360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007fd60eab33e0 in ?? ()
#8 0x00007fd61cb7a25d in ?? ()
#9 0x3fb965ad771cb000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x000000003c000000 in ?? ()
#14 0x00007fd600000000 in ?? ()
#15 0x41da16179c5dd6b5 in ?? ()
#16 0x00000ffb41d4e67c in ?? ()
#17 0x00007fd60eab33c0 in ?? ()
#18 0x00007fd61cb7eba3 in ?? ()
#19 0x00007fd60eab3390 in ?? ()
#20 0x3fb965ad771cb000 in ?? ()
#21 0x000000000eab33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb965ad771cb000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 4546):
#0 0x00007fd61a507a47 in ?? ()
#1 0x00007fd60f2b4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007fd60f2b4310 in ?? ()
#4 0x00007fd60f2b4520 in ?? ()
#5 0x00007fd60f2b4360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007fd60f2b43e0 in ?? ()
#8 0x00007fd61cb7a25d in ?? ()
#9 0x3fb0d2b3a7ebd000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x00000000b3500000 in ?? ()
#14 0x00007fd600000000 in ?? ()
#15 0x41da16179c5dd6b7 in ?? ()
#16 0x00000ffb41e4e87c in ?? ()
#17 0x00007fd60f2b43c0 in ?? ()
#18 0x00007fd61cb7eba3 in ?? ()
#19 0x00007fd60f2b4390 in ?? ()
#20 0x3fb0d2b3a7ebd000 in ?? ()
#21 0x000000000f2b43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb0d2b3a7ebd000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 4545):
#0 0x00007fd61a507a47 in ?? ()
#1 0x00007fd6112ad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007fd6112ad310 in ?? ()
#4 0x00007fd6112ad520 in ?? ()
#5 0x00007fd6112ad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007fd6112ad3e0 in ?? ()
#8 0x00007fd61cb7a25d in ?? ()
#9 0x3fb96a99e31eb000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x000000006e000000 in ?? ()
#14 0x00007fd600000000 in ?? ()
#15 0x41da16179c5dd6b4 in ?? ()
#16 0x00000ffb4224da7c in ?? ()
#17 0x00007fd6112ad3c0 in ?? ()
#18 0x00007fd61cb7eba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 4542):
#0 0x00007fd61a4facb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fd612cb77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 4541):
#0 0x00007fd6217d4fb9 in ?? ()
#1 0x00007fd6124b60c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007fd6124b60b0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 4540):
#0 0x00007fd6217d89e2 in ?? ()
#1 0x00007fd611cb3ba0 in ?? ()
#2 0x00007fd611cb3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fd611cb3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007fd611cb3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007fd611cb3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007fd6270ebedc in ?? ()
#12 0x00007fd6270db8d9 in ?? ()
#13 0x00007fd6270dfec6 in ?? ()
#14 0x00007fd6270e4900 in ?? ()
#15 0x00007fd6270e4729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007fd61e3b04fe in ?? ()
#18 0x00007fd6217ce6db in ?? ()
#19 0x00000ffb4238e684 in ?? ()
#20 0x00007fd611cb3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 4533):
#0 0x00007fd6217d4fb9 in ?? ()
#1 0x00007fd613cb8c80 in ?? ()
#2 0x00000000000002b9 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007fd613cb8c70 in ?? ()
#6 0x0000000000000572 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 4532):
#0 0x00007fd6217d4fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 4531):
#0 0x00007fd6217d4fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 4530):
#0 0x00007fd6217d4fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007fd6154bc450 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 4526):
#0 0x00007fd6217d8d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250622 19:58:10.871017 2016 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 3 with UUID 144888b25afe4c04b4de5350bddf224e and pid 5012
************************ BEGIN STACKS **************************
[New LWP 5014]
[New LWP 5015]
[New LWP 5016]
[New LWP 5018]
[New LWP 5025]
[New LWP 5026]
[New LWP 5027]
[New LWP 5030]
[New LWP 5031]
[New LWP 5032]
[New LWP 5033]
[New LWP 5034]
[New LWP 5035]
[New LWP 5036]
[New LWP 5037]
[New LWP 5038]
[New LWP 5039]
[New LWP 5040]
[New LWP 5041]
[New LWP 5042]
[New LWP 5043]
[New LWP 5044]
[New LWP 5045]
[New LWP 5046]
[New LWP 5047]
[New LWP 5048]
[New LWP 5049]
[New LWP 5050]
[New LWP 5051]
[New LWP 5052]
[New LWP 5053]
[New LWP 5054]
[New LWP 5055]
[New LWP 5056]
[New LWP 5057]
[New LWP 5058]
[New LWP 5059]
[New LWP 5060]
[New LWP 5061]
[New LWP 5062]
[New LWP 5063]
[New LWP 5064]
[New LWP 5065]
[New LWP 5066]
[New LWP 5067]
[New LWP 5068]
[New LWP 5069]
[New LWP 5070]
[New LWP 5071]
[New LWP 5072]
[New LWP 5073]
[New LWP 5074]
[New LWP 5075]
[New LWP 5076]
[New LWP 5078]
[New LWP 5079]
[New LWP 5080]
[New LWP 5081]
[New LWP 5082]
[New LWP 5083]
[New LWP 5084]
[New LWP 5085]
[New LWP 5086]
[New LWP 5087]
[New LWP 5088]
[New LWP 5089]
[New LWP 5090]
[New LWP 5091]
[New LWP 5092]
[New LWP 5093]
[New LWP 5094]
[New LWP 5095]
[New LWP 5096]
[New LWP 5097]
[New LWP 5098]
[New LWP 5099]
[New LWP 5100]
[New LWP 5101]
[New LWP 5102]
[New LWP 5103]
[New LWP 5104]
[New LWP 5105]
[New LWP 5106]
[New LWP 5107]
[New LWP 5108]
[New LWP 5109]
[New LWP 5110]
[New LWP 5111]
[New LWP 5112]
[New LWP 5113]
[New LWP 5114]
[New LWP 5115]
[New LWP 5116]
[New LWP 5117]
[New LWP 5118]
[New LWP 5119]
[New LWP 5120]
[New LWP 5121]
[New LWP 5122]
[New LWP 5123]
[New LWP 5124]
[New LWP 5125]
[New LWP 5126]
[New LWP 5127]
[New LWP 5128]
[New LWP 5129]
[New LWP 5130]
[New LWP 5131]
[New LWP 5132]
[New LWP 5133]
[New LWP 5134]
[New LWP 5135]
[New LWP 5136]
[New LWP 5137]
[New LWP 5138]
[New LWP 5139]
[New LWP 5140]
[New LWP 5141]
[New LWP 5142]
[New LWP 5143]
[New LWP 5144]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007fbd61234d50 in ?? ()
Id Target Id Frame
* 1 LWP 5012 "kudu" 0x00007fbd61234d50 in ?? ()
2 LWP 5014 "kudu" 0x00007fbd61230fb9 in ?? ()
3 LWP 5015 "kudu" 0x00007fbd61230fb9 in ?? ()
4 LWP 5016 "kudu" 0x00007fbd61230fb9 in ?? ()
5 LWP 5018 "kernel-watcher-" 0x00007fbd61230fb9 in ?? ()
6 LWP 5025 "ntp client-5025" 0x00007fbd612349e2 in ?? ()
7 LWP 5026 "file cache-evic" 0x00007fbd61230fb9 in ?? ()
8 LWP 5027 "sq_acceptor" 0x00007fbd59f56cb9 in ?? ()
9 LWP 5030 "rpc reactor-503" 0x00007fbd59f63a47 in ?? ()
10 LWP 5031 "rpc reactor-503" 0x00007fbd59f63a47 in ?? ()
11 LWP 5032 "rpc reactor-503" 0x00007fbd59f63a47 in ?? ()
12 LWP 5033 "rpc reactor-503" 0x00007fbd59f63a47 in ?? ()
13 LWP 5034 "MaintenanceMgr " 0x00007fbd61230ad3 in ?? ()
14 LWP 5035 "txn-status-mana" 0x00007fbd61230fb9 in ?? ()
15 LWP 5036 "collect_and_rem" 0x00007fbd61230fb9 in ?? ()
16 LWP 5037 "tc-session-exp-" 0x00007fbd61230fb9 in ?? ()
17 LWP 5038 "rpc worker-5038" 0x00007fbd61230ad3 in ?? ()
18 LWP 5039 "rpc worker-5039" 0x00007fbd61230ad3 in ?? ()
19 LWP 5040 "rpc worker-5040" 0x00007fbd61230ad3 in ?? ()
20 LWP 5041 "rpc worker-5041" 0x00007fbd61230ad3 in ?? ()
21 LWP 5042 "rpc worker-5042" 0x00007fbd61230ad3 in ?? ()
22 LWP 5043 "rpc worker-5043" 0x00007fbd61230ad3 in ?? ()
23 LWP 5044 "rpc worker-5044" 0x00007fbd61230ad3 in ?? ()
24 LWP 5045 "rpc worker-5045" 0x00007fbd61230ad3 in ?? ()
25 LWP 5046 "rpc worker-5046" 0x00007fbd61230ad3 in ?? ()
26 LWP 5047 "rpc worker-5047" 0x00007fbd61230ad3 in ?? ()
27 LWP 5048 "rpc worker-5048" 0x00007fbd61230ad3 in ?? ()
28 LWP 5049 "rpc worker-5049" 0x00007fbd61230ad3 in ?? ()
29 LWP 5050 "rpc worker-5050" 0x00007fbd61230ad3 in ?? ()
30 LWP 5051 "rpc worker-5051" 0x00007fbd61230ad3 in ?? ()
31 LWP 5052 "rpc worker-5052" 0x00007fbd61230ad3 in ?? ()
32 LWP 5053 "rpc worker-5053" 0x00007fbd61230ad3 in ?? ()
33 LWP 5054 "rpc worker-5054" 0x00007fbd61230ad3 in ?? ()
34 LWP 5055 "rpc worker-5055" 0x00007fbd61230ad3 in ?? ()
35 LWP 5056 "rpc worker-5056" 0x00007fbd61230ad3 in ?? ()
36 LWP 5057 "rpc worker-5057" 0x00007fbd61230ad3 in ?? ()
37 LWP 5058 "rpc worker-5058" 0x00007fbd61230ad3 in ?? ()
38 LWP 5059 "rpc worker-5059" 0x00007fbd61230ad3 in ?? ()
39 LWP 5060 "rpc worker-5060" 0x00007fbd61230ad3 in ?? ()
40 LWP 5061 "rpc worker-5061" 0x00007fbd61230ad3 in ?? ()
41 LWP 5062 "rpc worker-5062" 0x00007fbd61230ad3 in ?? ()
42 LWP 5063 "rpc worker-5063" 0x00007fbd61230ad3 in ?? ()
43 LWP 5064 "rpc worker-5064" 0x00007fbd61230ad3 in ?? ()
44 LWP 5065 "rpc worker-5065" 0x00007fbd61230ad3 in ?? ()
45 LWP 5066 "rpc worker-5066" 0x00007fbd61230ad3 in ?? ()
46 LWP 5067 "rpc worker-5067" 0x00007fbd61230ad3 in ?? ()
47 LWP 5068 "rpc worker-5068" 0x00007fbd61230ad3 in ?? ()
48 LWP 5069 "rpc worker-5069" 0x00007fbd61230ad3 in ?? ()
49 LWP 5070 "rpc worker-5070" 0x00007fbd61230ad3 in ?? ()
50 LWP 5071 "rpc worker-5071" 0x00007fbd61230ad3 in ?? ()
51 LWP 5072 "rpc worker-5072" 0x00007fbd61230ad3 in ?? ()
52 LWP 5073 "rpc worker-5073" 0x00007fbd61230ad3 in ?? ()
53 LWP 5074 "rpc worker-5074" 0x00007fbd61230ad3 in ?? ()
54 LWP 5075 "rpc worker-5075" 0x00007fbd61230ad3 in ?? ()
55 LWP 5076 "rpc worker-5076" 0x00007fbd61230ad3 in ?? ()
56 LWP 5078 "rpc worker-5078" 0x00007fbd61230ad3 in ?? ()
57 LWP 5079 "rpc worker-5079" 0x00007fbd61230ad3 in ?? ()
58 LWP 5080 "rpc worker-5080" 0x00007fbd61230ad3 in ?? ()
59 LWP 5081 "rpc worker-5081" 0x00007fbd61230ad3 in ?? ()
60 LWP 5082 "rpc worker-5082" 0x00007fbd61230ad3 in ?? ()
61 LWP 5083 "rpc worker-5083" 0x00007fbd61230ad3 in ?? ()
62 LWP 5084 "rpc worker-5084" 0x00007fbd61230ad3 in ?? ()
63 LWP 5085 "rpc worker-5085" 0x00007fbd61230ad3 in ?? ()
64 LWP 5086 "rpc worker-5086" 0x00007fbd61230ad3 in ?? ()
65 LWP 5087 "rpc worker-5087" 0x00007fbd61230ad3 in ?? ()
66 LWP 5088 "rpc worker-5088" 0x00007fbd61230ad3 in ?? ()
67 LWP 5089 "rpc worker-5089" 0x00007fbd61230ad3 in ?? ()
68 LWP 5090 "rpc worker-5090" 0x00007fbd61230ad3 in ?? ()
69 LWP 5091 "rpc worker-5091" 0x00007fbd61230ad3 in ?? ()
70 LWP 5092 "rpc worker-5092" 0x00007fbd61230ad3 in ?? ()
71 LWP 5093 "rpc worker-5093" 0x00007fbd61230ad3 in ?? ()
72 LWP 5094 "rpc worker-5094" 0x00007fbd61230ad3 in ?? ()
73 LWP 5095 "rpc worker-5095" 0x00007fbd61230ad3 in ?? ()
74 LWP 5096 "rpc worker-5096" 0x00007fbd61230ad3 in ?? ()
75 LWP 5097 "rpc worker-5097" 0x00007fbd61230ad3 in ?? ()
76 LWP 5098 "rpc worker-5098" 0x00007fbd61230ad3 in ?? ()
77 LWP 5099 "rpc worker-5099" 0x00007fbd61230ad3 in ?? ()
78 LWP 5100 "rpc worker-5100" 0x00007fbd61230ad3 in ?? ()
79 LWP 5101 "rpc worker-5101" 0x00007fbd61230ad3 in ?? ()
80 LWP 5102 "rpc worker-5102" 0x00007fbd61230ad3 in ?? ()
81 LWP 5103 "rpc worker-5103" 0x00007fbd61230ad3 in ?? ()
82 LWP 5104 "rpc worker-5104" 0x00007fbd61230ad3 in ?? ()
83 LWP 5105 "rpc worker-5105" 0x00007fbd61230ad3 in ?? ()
84 LWP 5106 "rpc worker-5106" 0x00007fbd61230ad3 in ?? ()
85 LWP 5107 "rpc worker-5107" 0x00007fbd61230ad3 in ?? ()
86 LWP 5108 "rpc worker-5108" 0x00007fbd61230ad3 in ?? ()
87 LWP 5109 "rpc worker-5109" 0x00007fbd61230ad3 in ?? ()
88 LWP 5110 "rpc worker-5110" 0x00007fbd61230ad3 in ?? ()
89 LWP 5111 "rpc worker-5111" 0x00007fbd61230ad3 in ?? ()
90 LWP 5112 "rpc worker-5112" 0x00007fbd61230ad3 in ?? ()
91 LWP 5113 "rpc worker-5113" 0x00007fbd61230ad3 in ?? ()
92 LWP 5114 "rpc worker-5114" 0x00007fbd61230ad3 in ?? ()
93 LWP 5115 "rpc worker-5115" 0x00007fbd61230ad3 in ?? ()
94 LWP 5116 "rpc worker-5116" 0x00007fbd61230ad3 in ?? ()
95 LWP 5117 "rpc worker-5117" 0x00007fbd61230ad3 in ?? ()
96 LWP 5118 "rpc worker-5118" 0x00007fbd61230ad3 in ?? ()
97 LWP 5119 "rpc worker-5119" 0x00007fbd61230ad3 in ?? ()
98 LWP 5120 "rpc worker-5120" 0x00007fbd61230ad3 in ?? ()
99 LWP 5121 "rpc worker-5121" 0x00007fbd61230ad3 in ?? ()
100 LWP 5122 "rpc worker-5122" 0x00007fbd61230ad3 in ?? ()
101 LWP 5123 "rpc worker-5123" 0x00007fbd61230ad3 in ?? ()
102 LWP 5124 "rpc worker-5124" 0x00007fbd61230ad3 in ?? ()
103 LWP 5125 "rpc worker-5125" 0x00007fbd61230ad3 in ?? ()
104 LWP 5126 "rpc worker-5126" 0x00007fbd61230ad3 in ?? ()
105 LWP 5127 "rpc worker-5127" 0x00007fbd61230ad3 in ?? ()
106 LWP 5128 "rpc worker-5128" 0x00007fbd61230ad3 in ?? ()
107 LWP 5129 "rpc worker-5129" 0x00007fbd61230ad3 in ?? ()
108 LWP 5130 "rpc worker-5130" 0x00007fbd61230ad3 in ?? ()
109 LWP 5131 "rpc worker-5131" 0x00007fbd61230ad3 in ?? ()
110 LWP 5132 "rpc worker-5132" 0x00007fbd61230ad3 in ?? ()
111 LWP 5133 "rpc worker-5133" 0x00007fbd61230ad3 in ?? ()
112 LWP 5134 "rpc worker-5134" 0x00007fbd61230ad3 in ?? ()
113 LWP 5135 "rpc worker-5135" 0x00007fbd61230ad3 in ?? ()
114 LWP 5136 "rpc worker-5136" 0x00007fbd61230ad3 in ?? ()
115 LWP 5137 "rpc worker-5137" 0x00007fbd61230ad3 in ?? ()
116 LWP 5138 "rpc worker-5138" 0x00007fbd61230ad3 in ?? ()
117 LWP 5139 "diag-logger-513" 0x00007fbd61230fb9 in ?? ()
118 LWP 5140 "result-tracker-" 0x00007fbd61230fb9 in ?? ()
119 LWP 5141 "excess-log-dele" 0x00007fbd61230fb9 in ?? ()
120 LWP 5142 "acceptor-5142" 0x00007fbd59f650c7 in ?? ()
121 LWP 5143 "heartbeat-5143" 0x00007fbd61230fb9 in ?? ()
122 LWP 5144 "maintenance_sch" 0x00007fbd61230fb9 in ?? ()
Thread 122 (LWP 5144):
#0 0x00007fbd61230fb9 in ?? ()
#1 0x00007fbd16ad86e0 in ?? ()
#2 0x00000000000001a6 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007fbd16ad8730 in ?? ()
#6 0x000000000000034c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 5143):
#0 0x00007fbd61230fb9 in ?? ()
#1 0x8ea7c1ad18c34923 in ?? ()
#2 0x0000000000000069 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019400 in ?? ()
#5 0x00007fbd172f05f0 in ?? ()
#6 0x00000000000000d2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 5142):
#0 0x00007fbd59f650c7 in ?? ()
#1 0x00006140000214a8 in ?? ()
#2 0x00007fbd17b17d90 in ?? ()
#3 0x00007fbd17b17dc0 in ?? ()
#4 0x00007fbd17b17ec0 in ?? ()
#5 0x00007fbd17b17db0 in ?? ()
#6 0x00007fbd17b17e20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 5141):
#0 0x00007fbd61230fb9 in ?? ()
#1 0x00007fbd18331f40 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffcb5f81ad0 in ?? ()
#5 0x00007fbd18331f30 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 5140):
#0 0x00007fbd61230fb9 in ?? ()
#1 0x00007fbd18b4a100 in ?? ()
#2 0x0000000000000069 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007fbd18b4a0f0 in ?? ()
#6 0x00000000000000d2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 5139):
#0 0x00007fbd61230fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 5138):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x000000000000000b in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007fbd19b79e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fbd19b79eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007fbd61230770 in ?? ()
#10 0x00007fbd19b79eb0 in ?? ()
#11 0x00000ff7a336f3d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 5137):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000019 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007fbd1a391e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fbd1a391eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007fbd61230770 in ?? ()
#10 0x00007fbd1a391eb0 in ?? ()
#11 0x00000ff7a34723d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 5136):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 5135):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 5134):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 5133):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 5132):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 5131):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 5130):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 5129):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 5128):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 5127):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 5126):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 5125):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 5124):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 5123):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 5122):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 5121):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 5120):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 5119):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 5118):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 5117):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 5116):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 5115):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 5114):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 5113):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 5112):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 5111):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 5110):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 5109):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 5108):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 5107):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 5106):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 5105):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 5104):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 5103):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 5102):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 5101):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 5100):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 5099):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 5098):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000009 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a688c in ?? ()
#4 0x00007fbd2df4ae90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fbd2df4aeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a6840 in ?? ()
#9 0x00007fbd61230770 in ?? ()
#10 0x00007fbd2df4aeb0 in ?? ()
#11 0x00000ff7a5be95d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 5097):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 5096):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 5095):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 5094):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 5093):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 5092):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 5091):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 5090):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 5089):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 5088):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 5087):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 5086):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 5085):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 5084):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 5083):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 5082):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 5081):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 5080):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 5079):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 5078):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000009 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012003c in ?? ()
#4 0x00007fbd3812be90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fbd3812beb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00011fff0 in ?? ()
#9 0x00007fbd61230770 in ?? ()
#10 0x00007fbd3812beb0 in ?? ()
#11 0x00000ff7a70257d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 5076):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 5075):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 5074):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 5073):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 5072):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 5071):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 5070):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 5069):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 5068):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 5067):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 5066):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 5065):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 5064):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 5063):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 5062):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 5061):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 5060):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 5059):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 5058):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 5057):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000009 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007fbd4230ce90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fbd4230ceb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007fbd61230770 in ?? ()
#10 0x00007fbd4230ceb0 in ?? ()
#11 0x00000ff7a84619d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 5056):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 5055):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 5054):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 5053):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 5052):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 5051):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 5050):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 5049):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 5048):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 5047):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 5046):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 5045):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 5044):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 5043):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 5042):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 5041):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 5040):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 5039):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 5038):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 5037):
#0 0x00007fbd61230fb9 in ?? ()
#1 0x00007fbd4bbc2cc0 in ?? ()
#2 0x000000000000000a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019ca0 in ?? ()
#5 0x00007fbd4bbc2cb0 in ?? ()
#6 0x0000000000000014 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 5036):
#0 0x00007fbd61230fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 5035):
#0 0x00007fbd61230fb9 in ?? ()
#1 0x00007fbd4cbc4240 in ?? ()
#2 0x000000000000000a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007fbd4cbc4230 in ?? ()
#6 0x0000000000000014 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 5034):
#0 0x00007fbd61230ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 5033):
#0 0x00007fbd59f63a47 in ?? ()
#1 0x00007fbd4dbc7320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007fbd4dbc7310 in ?? ()
#4 0x00007fbd4dbc7520 in ?? ()
#5 0x00007fbd4dbc7360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fbd4dbc73e0 in ?? ()
#8 0x00007fbd5c5d625d in ?? ()
#9 0x3fb9658e64ccd000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x000000009a000000 in ?? ()
#14 0x00007fbd00000000 in ?? ()
#15 0x41da16179c5dd6b9 in ?? ()
#16 0x00000ff829b70e7c in ?? ()
#17 0x00007fbd4dbc73c0 in ?? ()
#18 0x00007fbd5c5daba3 in ?? ()
#19 0x00007fbd4dbc7390 in ?? ()
#20 0x3fb9658e64ccd000 in ?? ()
#21 0x000000004dbc73e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb9658e64ccd000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 5032):
#0 0x00007fbd59f63a47 in ?? ()
#1 0x00007fbd4e3c8320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007fbd4e3c8310 in ?? ()
#4 0x00007fbd4e3c8520 in ?? ()
#5 0x00007fbd4e3c8360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007fbd4e3c83e0 in ?? ()
#8 0x00007fbd5c5d625d in ?? ()
#9 0x3fb979a319bac000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x0000000038000000 in ?? ()
#14 0x00007fbd00000000 in ?? ()
#15 0x41da16179c5dd6b5 in ?? ()
#16 0x00000ff829c7107c in ?? ()
#17 0x00007fbd4e3c83c0 in ?? ()
#18 0x00007fbd5c5daba3 in ?? ()
#19 0x00007fbd4e3c8390 in ?? ()
#20 0x3fb979a319bac000 in ?? ()
#21 0x000000004e3c83e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb979a319bac000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 5031):
#0 0x00007fbd59f63a47 in ?? ()
#1 0x00007fbd4ebc9320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007fbd4ebc9310 in ?? ()
#4 0x00007fbd4ebc9520 in ?? ()
#5 0x00007fbd4ebc9360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007fbd4ebc93e0 in ?? ()
#8 0x00007fbd5c5d625d in ?? ()
#9 0x3fb977750a3b8000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x000000005e000000 in ?? ()
#14 0x00007fbd00000000 in ?? ()
#15 0x41da16179c5dd6b3 in ?? ()
#16 0x00000ff829d7127c in ?? ()
#17 0x00007fbd4ebc93c0 in ?? ()
#18 0x00007fbd5c5daba3 in ?? ()
#19 0x00007fbd4ebc9390 in ?? ()
#20 0x3fb977750a3b8000 in ?? ()
#21 0x000000004ebc93e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb977750a3b8000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 5030):
#0 0x00007fbd59f63a47 in ?? ()
#1 0x00007fbd507b3320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007fbd507b3310 in ?? ()
#4 0x00007fbd507b3520 in ?? ()
#5 0x00007fbd507b3360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007fbd507b33e0 in ?? ()
#8 0x00007fbd5c5d625d in ?? ()
#9 0x3fb95a30c581d000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x00000000de000000 in ?? ()
#14 0x00007fbd00000000 in ?? ()
#15 0x41da16179c5dd6b6 in ?? ()
#16 0x00000ff82a0ee67c in ?? ()
#17 0x00007fbd507b33c0 in ?? ()
#18 0x00007fbd5c5daba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 5027):
#0 0x00007fbd59f56cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fbd527b77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 5026):
#0 0x00007fbd61230fb9 in ?? ()
#1 0x00007fbd51fb60c0 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007fbd51fb60b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 5025):
#0 0x00007fbd612349e2 in ?? ()
#1 0x00007fbd50fb2ba0 in ?? ()
#2 0x00007fbd50fb2c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fbd50fb2bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007fbd50fb2c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007fbd50fb2c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007fbd66b47edc in ?? ()
#12 0x00007fbd66b378d9 in ?? ()
#13 0x00007fbd66b3bec6 in ?? ()
#14 0x00007fbd66b40900 in ?? ()
#15 0x00007fbd66b40729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007fbd5de0c4fe in ?? ()
#18 0x00007fbd6122a6db in ?? ()
#19 0x00000ff82a1ee484 in ?? ()
#20 0x00007fbd50fb2440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 5018):
#0 0x00007fbd61230fb9 in ?? ()
#1 0x00007fbd537b8c80 in ?? ()
#2 0x0000000000000215 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007fbd537b8c70 in ?? ()
#6 0x000000000000042a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 5016):
#0 0x00007fbd61230fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 5015):
#0 0x00007fbd61230fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 5014):
#0 0x00007fbd61230fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000005 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007fbd54fbc450 in ?? ()
#6 0x000000000000000a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 5012):
#0 0x00007fbd61234d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250622 19:58:12.074867 2016 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 4 with UUID 9ece0d51c71b47a1865ba3b05920d999 and pid 3653
************************ BEGIN STACKS **************************
[New LWP 3655]
[New LWP 3656]
[New LWP 3657]
[New LWP 3658]
[New LWP 3664]
[New LWP 3665]
[New LWP 3666]
[New LWP 3669]
[New LWP 3670]
[New LWP 3671]
[New LWP 3672]
[New LWP 3673]
[New LWP 3674]
[New LWP 3676]
[New LWP 3677]
[New LWP 3678]
[New LWP 3679]
[New LWP 3680]
[New LWP 3681]
[New LWP 3682]
[New LWP 3683]
[New LWP 3684]
[New LWP 3685]
[New LWP 3686]
[New LWP 3687]
[New LWP 3688]
[New LWP 3689]
[New LWP 3690]
[New LWP 3691]
[New LWP 3692]
[New LWP 3693]
[New LWP 3694]
[New LWP 3695]
[New LWP 3696]
[New LWP 3697]
[New LWP 3698]
[New LWP 3699]
[New LWP 3700]
[New LWP 3701]
[New LWP 3702]
[New LWP 3703]
[New LWP 3704]
[New LWP 3705]
[New LWP 3706]
[New LWP 3707]
[New LWP 3708]
[New LWP 3709]
[New LWP 3710]
[New LWP 3711]
[New LWP 3712]
[New LWP 3713]
[New LWP 3714]
[New LWP 3715]
[New LWP 3716]
[New LWP 3717]
[New LWP 3718]
[New LWP 3719]
[New LWP 3720]
[New LWP 3721]
[New LWP 3722]
[New LWP 3723]
[New LWP 3724]
[New LWP 3726]
[New LWP 3727]
[New LWP 3728]
[New LWP 3729]
[New LWP 3730]
[New LWP 3731]
[New LWP 3732]
[New LWP 3733]
[New LWP 3734]
[New LWP 3735]
[New LWP 3736]
[New LWP 3737]
[New LWP 3738]
[New LWP 3739]
[New LWP 3740]
[New LWP 3741]
[New LWP 3742]
[New LWP 3743]
[New LWP 3744]
[New LWP 3745]
[New LWP 3746]
[New LWP 3747]
[New LWP 3748]
[New LWP 3749]
[New LWP 3750]
[New LWP 3752]
[New LWP 3753]
[New LWP 3754]
[New LWP 3755]
[New LWP 3756]
[New LWP 3757]
[New LWP 3758]
[New LWP 3759]
[New LWP 3760]
[New LWP 3761]
[New LWP 3762]
[New LWP 3763]
[New LWP 3764]
[New LWP 3765]
[New LWP 3766]
[New LWP 3767]
[New LWP 3768]
[New LWP 3769]
[New LWP 3770]
[New LWP 3771]
[New LWP 3772]
[New LWP 3773]
[New LWP 3774]
[New LWP 3775]
[New LWP 3776]
[New LWP 3777]
[New LWP 3778]
[New LWP 3779]
[New LWP 3780]
[New LWP 3781]
[New LWP 3782]
[New LWP 3783]
[New LWP 3784]
[New LWP 3787]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007fa0e9b34d50 in ?? ()
Id Target Id Frame
* 1 LWP 3653 "kudu" 0x00007fa0e9b34d50 in ?? ()
2 LWP 3655 "kudu" 0x00007fa0e9b30fb9 in ?? ()
3 LWP 3656 "kudu" 0x00007fa0e9b30fb9 in ?? ()
4 LWP 3657 "kudu" 0x00007fa0e9b30fb9 in ?? ()
5 LWP 3658 "kernel-watcher-" 0x00007fa0e9b30fb9 in ?? ()
6 LWP 3664 "ntp client-3664" 0x00007fa0e9b349e2 in ?? ()
7 LWP 3665 "file cache-evic" 0x00007fa0e9b30fb9 in ?? ()
8 LWP 3666 "sq_acceptor" 0x00007fa0e2856cb9 in ?? ()
9 LWP 3669 "rpc reactor-366" 0x00007fa0e2863a47 in ?? ()
10 LWP 3670 "rpc reactor-367" 0x00007fa0e2863a47 in ?? ()
11 LWP 3671 "rpc reactor-367" 0x00007fa0e2863a47 in ?? ()
12 LWP 3672 "rpc reactor-367" 0x00007fa0e2863a47 in ?? ()
13 LWP 3673 "MaintenanceMgr " 0x00007fa0e9b30ad3 in ?? ()
14 LWP 3674 "txn-status-mana" 0x00007fa0e9b30fb9 in ?? ()
15 LWP 3676 "collect_and_rem" 0x00007fa0e9b30fb9 in ?? ()
16 LWP 3677 "tc-session-exp-" 0x00007fa0e9b30fb9 in ?? ()
17 LWP 3678 "rpc worker-3678" 0x00007fa0e9b30ad3 in ?? ()
18 LWP 3679 "rpc worker-3679" 0x00007fa0e9b30ad3 in ?? ()
19 LWP 3680 "rpc worker-3680" 0x00007fa0e9b30ad3 in ?? ()
20 LWP 3681 "rpc worker-3681" 0x00007fa0e9b30ad3 in ?? ()
21 LWP 3682 "rpc worker-3682" 0x00007fa0e9b30ad3 in ?? ()
22 LWP 3683 "rpc worker-3683" 0x00007fa0e9b30ad3 in ?? ()
23 LWP 3684 "rpc worker-3684" 0x00007fa0e9b30ad3 in ?? ()
24 LWP 3685 "rpc worker-3685" 0x00007fa0e9b30ad3 in ?? ()
25 LWP 3686 "rpc worker-3686" 0x00007fa0e9b30ad3 in ?? ()
26 LWP 3687 "rpc worker-3687" 0x00007fa0e9b30ad3 in ?? ()
27 LWP 3688 "rpc worker-3688" 0x00007fa0e9b30ad3 in ?? ()
28 LWP 3689 "rpc worker-3689" 0x00007fa0e9b30ad3 in ?? ()
29 LWP 3690 "rpc worker-3690" 0x00007fa0e9b30ad3 in ?? ()
30 LWP 3691 "rpc worker-3691" 0x00007fa0e9b30ad3 in ?? ()
31 LWP 3692 "rpc worker-3692" 0x00007fa0e9b30ad3 in ?? ()
32 LWP 3693 "rpc worker-3693" 0x00007fa0e9b30ad3 in ?? ()
33 LWP 3694 "rpc worker-3694" 0x00007fa0e9b30ad3 in ?? ()
34 LWP 3695 "rpc worker-3695" 0x00007fa0e9b30ad3 in ?? ()
35 LWP 3696 "rpc worker-3696" 0x00007fa0e9b30ad3 in ?? ()
36 LWP 3697 "rpc worker-3697" 0x00007fa0e9b30ad3 in ?? ()
37 LWP 3698 "rpc worker-3698" 0x00007fa0e9b30ad3 in ?? ()
38 LWP 3699 "rpc worker-3699" 0x00007fa0e9b30ad3 in ?? ()
39 LWP 3700 "rpc worker-3700" 0x00007fa0e9b30ad3 in ?? ()
40 LWP 3701 "rpc worker-3701" 0x00007fa0e9b30ad3 in ?? ()
41 LWP 3702 "rpc worker-3702" 0x00007fa0e9b30ad3 in ?? ()
42 LWP 3703 "rpc worker-3703" 0x00007fa0e9b30ad3 in ?? ()
43 LWP 3704 "rpc worker-3704" 0x00007fa0e9b30ad3 in ?? ()
44 LWP 3705 "rpc worker-3705" 0x00007fa0e9b30ad3 in ?? ()
45 LWP 3706 "rpc worker-3706" 0x00007fa0e9b30ad3 in ?? ()
46 LWP 3707 "rpc worker-3707" 0x00007fa0e9b30ad3 in ?? ()
47 LWP 3708 "rpc worker-3708" 0x00007fa0e9b30ad3 in ?? ()
48 LWP 3709 "rpc worker-3709" 0x00007fa0e9b30ad3 in ?? ()
49 LWP 3710 "rpc worker-3710" 0x00007fa0e9b30ad3 in ?? ()
50 LWP 3711 "rpc worker-3711" 0x00007fa0e9b30ad3 in ?? ()
51 LWP 3712 "rpc worker-3712" 0x00007fa0e9b30ad3 in ?? ()
52 LWP 3713 "rpc worker-3713" 0x00007fa0e9b30ad3 in ?? ()
53 LWP 3714 "rpc worker-3714" 0x00007fa0e9b30ad3 in ?? ()
54 LWP 3715 "rpc worker-3715" 0x00007fa0e9b30ad3 in ?? ()
55 LWP 3716 "rpc worker-3716" 0x00007fa0e9b30ad3 in ?? ()
56 LWP 3717 "rpc worker-3717" 0x00007fa0e9b30ad3 in ?? ()
57 LWP 3718 "rpc worker-3718" 0x00007fa0e9b30ad3 in ?? ()
58 LWP 3719 "rpc worker-3719" 0x00007fa0e9b30ad3 in ?? ()
59 LWP 3720 "rpc worker-3720" 0x00007fa0e9b30ad3 in ?? ()
60 LWP 3721 "rpc worker-3721" 0x00007fa0e9b30ad3 in ?? ()
61 LWP 3722 "rpc worker-3722" 0x00007fa0e9b30ad3 in ?? ()
62 LWP 3723 "rpc worker-3723" 0x00007fa0e9b30ad3 in ?? ()
63 LWP 3724 "rpc worker-3724" 0x00007fa0e9b30ad3 in ?? ()
64 LWP 3726 "rpc worker-3726" 0x00007fa0e9b30ad3 in ?? ()
65 LWP 3727 "rpc worker-3727" 0x00007fa0e9b30ad3 in ?? ()
66 LWP 3728 "rpc worker-3728" 0x00007fa0e9b30ad3 in ?? ()
67 LWP 3729 "rpc worker-3729" 0x00007fa0e9b30ad3 in ?? ()
68 LWP 3730 "rpc worker-3730" 0x00007fa0e9b30ad3 in ?? ()
69 LWP 3731 "rpc worker-3731" 0x00007fa0e9b30ad3 in ?? ()
70 LWP 3732 "rpc worker-3732" 0x00007fa0e9b30ad3 in ?? ()
71 LWP 3733 "rpc worker-3733" 0x00007fa0e9b30ad3 in ?? ()
72 LWP 3734 "rpc worker-3734" 0x00007fa0e9b30ad3 in ?? ()
73 LWP 3735 "rpc worker-3735" 0x00007fa0e9b30ad3 in ?? ()
74 LWP 3736 "rpc worker-3736" 0x00007fa0e9b30ad3 in ?? ()
75 LWP 3737 "rpc worker-3737" 0x00007fa0e9b30ad3 in ?? ()
76 LWP 3738 "rpc worker-3738" 0x00007fa0e9b30ad3 in ?? ()
77 LWP 3739 "rpc worker-3739" 0x00007fa0e9b30ad3 in ?? ()
78 LWP 3740 "rpc worker-3740" 0x00007fa0e9b30ad3 in ?? ()
79 LWP 3741 "rpc worker-3741" 0x00007fa0e9b30ad3 in ?? ()
80 LWP 3742 "rpc worker-3742" 0x00007fa0e9b30ad3 in ?? ()
81 LWP 3743 "rpc worker-3743" 0x00007fa0e9b30ad3 in ?? ()
82 LWP 3744 "rpc worker-3744" 0x00007fa0e9b30ad3 in ?? ()
83 LWP 3745 "rpc worker-3745" 0x00007fa0e9b30ad3 in ?? ()
84 LWP 3746 "rpc worker-3746" 0x00007fa0e9b30ad3 in ?? ()
85 LWP 3747 "rpc worker-3747" 0x00007fa0e9b30ad3 in ?? ()
86 LWP 3748 "rpc worker-3748" 0x00007fa0e9b30ad3 in ?? ()
87 LWP 3749 "rpc worker-3749" 0x00007fa0e9b30ad3 in ?? ()
88 LWP 3750 "rpc worker-3750" 0x00007fa0e9b30ad3 in ?? ()
89 LWP 3752 "rpc worker-3752" 0x00007fa0e9b30ad3 in ?? ()
90 LWP 3753 "rpc worker-3753" 0x00007fa0e9b30ad3 in ?? ()
91 LWP 3754 "rpc worker-3754" 0x00007fa0e9b30ad3 in ?? ()
92 LWP 3755 "rpc worker-3755" 0x00007fa0e9b30ad3 in ?? ()
93 LWP 3756 "rpc worker-3756" 0x00007fa0e9b30ad3 in ?? ()
94 LWP 3757 "rpc worker-3757" 0x00007fa0e9b30ad3 in ?? ()
95 LWP 3758 "rpc worker-3758" 0x00007fa0e9b30ad3 in ?? ()
96 LWP 3759 "rpc worker-3759" 0x00007fa0e9b30ad3 in ?? ()
97 LWP 3760 "rpc worker-3760" 0x00007fa0e9b30ad3 in ?? ()
98 LWP 3761 "rpc worker-3761" 0x00007fa0e9b30ad3 in ?? ()
99 LWP 3762 "rpc worker-3762" 0x00007fa0e9b30ad3 in ?? ()
100 LWP 3763 "rpc worker-3763" 0x00007fa0e9b30ad3 in ?? ()
101 LWP 3764 "rpc worker-3764" 0x00007fa0e9b30ad3 in ?? ()
102 LWP 3765 "rpc worker-3765" 0x00007fa0e9b30ad3 in ?? ()
103 LWP 3766 "rpc worker-3766" 0x00007fa0e9b30ad3 in ?? ()
104 LWP 3767 "rpc worker-3767" 0x00007fa0e9b30ad3 in ?? ()
105 LWP 3768 "rpc worker-3768" 0x00007fa0e9b30ad3 in ?? ()
106 LWP 3769 "rpc worker-3769" 0x00007fa0e9b30ad3 in ?? ()
107 LWP 3770 "rpc worker-3770" 0x00007fa0e9b30ad3 in ?? ()
108 LWP 3771 "rpc worker-3771" 0x00007fa0e9b30ad3 in ?? ()
109 LWP 3772 "rpc worker-3772" 0x00007fa0e9b30ad3 in ?? ()
110 LWP 3773 "rpc worker-3773" 0x00007fa0e9b30ad3 in ?? ()
111 LWP 3774 "rpc worker-3774" 0x00007fa0e9b30ad3 in ?? ()
112 LWP 3775 "rpc worker-3775" 0x00007fa0e9b30ad3 in ?? ()
113 LWP 3776 "rpc worker-3776" 0x00007fa0e9b30ad3 in ?? ()
114 LWP 3777 "rpc worker-3777" 0x00007fa0e9b30ad3 in ?? ()
115 LWP 3778 "rpc worker-3778" 0x00007fa0e9b30ad3 in ?? ()
116 LWP 3779 "rpc worker-3779" 0x00007fa0e9b30ad3 in ?? ()
117 LWP 3780 "diag-logger-378" 0x00007fa0e9b30fb9 in ?? ()
118 LWP 3781 "result-tracker-" 0x00007fa0e9b30fb9 in ?? ()
119 LWP 3782 "excess-log-dele" 0x00007fa0e9b30fb9 in ?? ()
120 LWP 3783 "acceptor-3783" 0x00007fa0e28650c7 in ?? ()
121 LWP 3784 "heartbeat-3784" 0x00007fa0e9b30fb9 in ?? ()
122 LWP 3787 "maintenance_sch" 0x00007fa0e9b30fb9 in ?? ()
Thread 122 (LWP 3787):
#0 0x00007fa0e9b30fb9 in ?? ()
#1 0x00007fa0bc0016e0 in ?? ()
#2 0x00000000000002a6 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00006160000157a4 in ?? ()
#5 0x00007fa0bc001730 in ?? ()
#6 0x000000000000054d in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fa0bc001750 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007fa0e65c0380 in ?? ()
#12 0x00007fa0bc0020d0 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 121 (LWP 3784):
#0 0x00007fa0e9b30fb9 in ?? ()
#1 0x8ea7c1ad18c34923 in ?? ()
#2 0x00000000000000ac in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019400 in ?? ()
#5 0x00007fa09e99e5f0 in ?? ()
#6 0x0000000000000158 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 3783):
#0 0x00007fa0e28650c7 in ?? ()
#1 0x00006140000210a8 in ?? ()
#2 0x00007fa09f234d90 in ?? ()
#3 0x00007fa09f234dc0 in ?? ()
#4 0x00007fa09f234ec0 in ?? ()
#5 0x00007fa09f234db0 in ?? ()
#6 0x00007fa09f234e20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x00007fa0e66eb980 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 3782):
#0 0x00007fa0e9b30fb9 in ?? ()
#1 0x00007fa09fa4df40 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff86717690 in ?? ()
#5 0x00007fa09fa4df30 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 3781):
#0 0x00007fa0e9b30fb9 in ?? ()
#1 0x00007fa0a0266100 in ?? ()
#2 0x00000000000000a9 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007fa0a02660f0 in ?? ()
#6 0x0000000000000152 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 3780):
#0 0x00007fa0e9b30fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 3779):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000023 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00039683c in ?? ()
#4 0x00007fa0a12a6e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa0a12a6eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0003967f0 in ?? ()
#9 0x00007fa0e9b30770 in ?? ()
#10 0x00007fa0a12a6eb0 in ?? ()
#11 0x00000ff414254dd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 3778):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000019 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00038d04c in ?? ()
#4 0x00007fa0a1abee90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa0a1abeeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00038d000 in ?? ()
#9 0x00007fa0e9b30770 in ?? ()
#10 0x00007fa0a1abeeb0 in ?? ()
#11 0x00000ff414357dd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 3777):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 3776):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 3775):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 3774):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 3773):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 3772):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 3771):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 3770):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 3769):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 3768):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 3767):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 3766):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 3765):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 3764):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 3763):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 3762):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 3761):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 3760):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 3759):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 3758):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 3757):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 3756):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 3755):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 3754):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 3753):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 3752):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 3750):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 3749):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 3748):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 3747):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 3746):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 3745):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 3744):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 3743):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 3742):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 3741):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 3740):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 3739):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 3738):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000001b06 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000206828 in ?? ()
#4 0x00007fa0b56a8e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa0b56a8eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 3737):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000001c40 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001e6848 in ?? ()
#4 0x00007fa0b5ec0e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa0b5ec0eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 3736):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000001279 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001c686c in ?? ()
#4 0x00007fa0b66d8e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa0b66d8eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001c6820 in ?? ()
#9 0x00007fa0e9b30770 in ?? ()
#10 0x00007fa0b66d8eb0 in ?? ()
#11 0x00000ff416cdb1d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 73 (LWP 3735):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 3734):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 3733):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 3732):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 3731):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 3730):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 3729):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 3728):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 3727):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 3726):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 3724):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 3723):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 3722):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 3721):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 3720):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 3719):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 3718):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 3717):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007fa0c01d3e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa0c01d3eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007fa0e9b30770 in ?? ()
#10 0x00007fa0c01d3eb0 in ?? ()
#11 0x00000ff41803a7d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 3716):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 3715):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 3714):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 3713):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 3712):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 3711):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 3710):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 3709):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 3708):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 3707):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 3706):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 3705):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 3704):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 3703):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 3702):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 3701):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 3700):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 3699):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 3698):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 3697):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000001d49 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007fa0c9b9ce90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa0c9b9ceb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007fa0e9b30770 in ?? ()
#10 0x00007fa0c9b9ceb0 in ?? ()
#11 0x00000ff4193739d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 3696):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000001d6d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007fa0ca3b4e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa0ca3b4eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007fa0e9b30770 in ?? ()
#10 0x00007fa0ca3b4eb0 in ?? ()
#11 0x00000ff4194769d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 34 (LWP 3695):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x00000000000014db in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000967fc in ?? ()
#4 0x00007fa0cabcce90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa0cabcceb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000967b0 in ?? ()
#9 0x00007fa0e9b30770 in ?? ()
#10 0x00007fa0cabcceb0 in ?? ()
#11 0x00000ff4195799d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 33 (LWP 3694):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000001d4e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00008fff8 in ?? ()
#4 0x00007fa0cb3e4e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa0cb3e4eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 32 (LWP 3693):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000001ed6 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00008d008 in ?? ()
#4 0x00007fa0cbbfce90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa0cbbfceb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 31 (LWP 3692):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x00000000000010c0 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000086808 in ?? ()
#4 0x00007fa0cc414e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa0cc414eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 30 (LWP 3691):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 3690):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 3689):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 3688):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 3687):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 3686):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 3685):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 3684):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 3683):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 3682):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 3681):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 3680):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 3679):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 3678):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 3677):
#0 0x00007fa0e9b30fb9 in ?? ()
#1 0x00007fa0d3da1cc0 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a3a0 in ?? ()
#5 0x00007fa0d3da1cb0 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 3676):
#0 0x00007fa0e9b30fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 3674):
#0 0x00007fa0e9b30fb9 in ?? ()
#1 0x00007fa0d5637240 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007fa0d5637230 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 3673):
#0 0x00007fa0e9b30ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007fa0d5e5eba4 in ?? ()
#4 0x00007fa0d5e5e8b0 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa0d5e5e8d0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00006160000159c0 in ?? ()
#9 0x00007fa0e9b30770 in ?? ()
#10 0x00007fa0d5e5e8d0 in ?? ()
#11 0x00007fa0d5e5e8d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 12 (LWP 3672):
#0 0x00007fa0e2863a47 in ?? ()
#1 0x00007fa0d6686320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007fa0d6686310 in ?? ()
#4 0x00007fa0d6686520 in ?? ()
#5 0x00007fa0d6686360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fa0d66863e0 in ?? ()
#8 0x00007fa0e4ed625d in ?? ()
#9 0x3fb95602bcd9b000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x000000009e000000 in ?? ()
#14 0x00007fa000000000 in ?? ()
#15 0x41da16179c5dd6b4 in ?? ()
#16 0x00000ff49acc8c7c in ?? ()
#17 0x00007fa0d66863c0 in ?? ()
#18 0x00007fa0e4edaba3 in ?? ()
#19 0x00007fa0d6686390 in ?? ()
#20 0x3fb95602bcd9b000 in ?? ()
#21 0x00000000d66863e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb95602bcd9b000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 3671):
#0 0x00007fa0e2863a47 in ?? ()
#1 0x00007fa0d6e9d320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007fa0d6e9d310 in ?? ()
#4 0x00007fa0d6e9d520 in ?? ()
#5 0x00007fa0d6e9d360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007fa0d6e9d3e0 in ?? ()
#8 0x00007fa0e4ed625d in ?? ()
#9 0x3fb954d0f9124000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x00000000be000000 in ?? ()
#14 0x00007fa000000000 in ?? ()
#15 0x41da16179c5dd6b7 in ?? ()
#16 0x00000ff49adcba7c in ?? ()
#17 0x00007fa0d6e9d3c0 in ?? ()
#18 0x00007fa0e4edaba3 in ?? ()
#19 0x00007fa0d6e9d390 in ?? ()
#20 0x3fb954d0f9124000 in ?? ()
#21 0x00000000d6e9d3e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb954d0f9124000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 3670):
#0 0x00007fa0e2863a47 in ?? ()
#1 0x00007fa0d76b4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007fa0d76b4310 in ?? ()
#4 0x00007fa0d76b4520 in ?? ()
#5 0x00007fa0d76b4360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007fa0d76b43e0 in ?? ()
#8 0x00007fa0e4ed625d in ?? ()
#9 0x3fb4324f99d73000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x00000000b8900000 in ?? ()
#14 0x00007fa000000000 in ?? ()
#15 0x41da16179c5dd6b2 in ?? ()
#16 0x00000ff49aece87c in ?? ()
#17 0x00007fa0d76b43c0 in ?? ()
#18 0x00007fa0e4edaba3 in ?? ()
#19 0x00007fa0d76b4390 in ?? ()
#20 0x3fb4324f99d73000 in ?? ()
#21 0x00000000d76b43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb4324f99d73000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 3669):
#0 0x00007fa0e2863a47 in ?? ()
#1 0x00007fa0d96ad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007fa0d96ad310 in ?? ()
#4 0x00007fa0d96ad520 in ?? ()
#5 0x00007fa0d96ad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007fa0d96ad3e0 in ?? ()
#8 0x00007fa0e4ed625d in ?? ()
#9 0x3fb97f0ead5b7000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x00000000e1800000 in ?? ()
#14 0x00007fa000000000 in ?? ()
#15 0x41da16179c5dd6b3 in ?? ()
#16 0x00000ff49b2cda7c in ?? ()
#17 0x00007fa0d96ad3c0 in ?? ()
#18 0x00007fa0e4edaba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 3666):
#0 0x00007fa0e2856cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fa0db0b77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 3665):
#0 0x00007fa0e9b30fb9 in ?? ()
#1 0x00007fa0da8b60c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007fa0da8b60b0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 3664):
#0 0x00007fa0e9b349e2 in ?? ()
#1 0x00007fa0da0b3ba0 in ?? ()
#2 0x00007fa0da0b3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fa0da0b3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007fa0da0b3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007fa0da0b3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007fa0ef447edc in ?? ()
#12 0x00007fa0ef4378d9 in ?? ()
#13 0x00007fa0ef43bec6 in ?? ()
#14 0x00007fa0ef440900 in ?? ()
#15 0x00007fa0ef440729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007fa0e670c4fe in ?? ()
#18 0x00007fa0e9b2a6db in ?? ()
#19 0x00000ff49b40e684 in ?? ()
#20 0x00007fa0da0b3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 3658):
#0 0x00007fa0e9b30fb9 in ?? ()
#1 0x00007fa0dc0b8c80 in ?? ()
#2 0x0000000000000352 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007fa0dc0b8c70 in ?? ()
#6 0x00000000000006a4 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 3657):
#0 0x00007fa0e9b30fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 3656):
#0 0x00007fa0e9b30fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 3655):
#0 0x00007fa0e9b30fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007fa0dd8bc450 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 3653):
#0 0x00007fa0e9b34d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250622 19:58:12.984102 2016 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 5 with UUID e63bffd596554411a2e0489c19e78fff and pid 3510
************************ BEGIN STACKS **************************
[New LWP 3514]
[New LWP 3515]
[New LWP 3516]
[New LWP 3517]
[New LWP 3523]
[New LWP 3524]
[New LWP 3525]
[New LWP 3528]
[New LWP 3529]
[New LWP 3530]
[New LWP 3531]
[New LWP 3532]
[New LWP 3533]
[New LWP 3534]
[New LWP 3535]
[New LWP 3536]
[New LWP 3537]
[New LWP 3538]
[New LWP 3539]
[New LWP 3540]
[New LWP 3541]
[New LWP 3542]
[New LWP 3543]
[New LWP 3544]
[New LWP 3545]
[New LWP 3546]
[New LWP 3547]
[New LWP 3548]
[New LWP 3549]
[New LWP 3550]
[New LWP 3551]
[New LWP 3552]
[New LWP 3553]
[New LWP 3554]
[New LWP 3555]
[New LWP 3556]
[New LWP 3557]
[New LWP 3558]
[New LWP 3559]
[New LWP 3561]
[New LWP 3562]
[New LWP 3563]
[New LWP 3564]
[New LWP 3565]
[New LWP 3566]
[New LWP 3567]
[New LWP 3568]
[New LWP 3569]
[New LWP 3570]
[New LWP 3571]
[New LWP 3572]
[New LWP 3573]
[New LWP 3574]
[New LWP 3575]
[New LWP 3576]
[New LWP 3577]
[New LWP 3578]
[New LWP 3579]
[New LWP 3580]
[New LWP 3581]
[New LWP 3582]
[New LWP 3583]
[New LWP 3584]
[New LWP 3586]
[New LWP 3587]
[New LWP 3588]
[New LWP 3589]
[New LWP 3590]
[New LWP 3591]
[New LWP 3592]
[New LWP 3593]
[New LWP 3594]
[New LWP 3595]
[New LWP 3596]
[New LWP 3597]
[New LWP 3598]
[New LWP 3599]
[New LWP 3600]
[New LWP 3601]
[New LWP 3602]
[New LWP 3603]
[New LWP 3604]
[New LWP 3605]
[New LWP 3606]
[New LWP 3607]
[New LWP 3608]
[New LWP 3609]
[New LWP 3610]
[New LWP 3611]
[New LWP 3612]
[New LWP 3613]
[New LWP 3614]
[New LWP 3615]
[New LWP 3616]
[New LWP 3617]
[New LWP 3618]
[New LWP 3619]
[New LWP 3620]
[New LWP 3621]
[New LWP 3622]
[New LWP 3623]
[New LWP 3624]
[New LWP 3625]
[New LWP 3626]
[New LWP 3627]
[New LWP 3628]
[New LWP 3629]
[New LWP 3630]
[New LWP 3631]
[New LWP 3632]
[New LWP 3633]
[New LWP 3634]
[New LWP 3635]
[New LWP 3636]
[New LWP 3637]
[New LWP 3638]
[New LWP 3639]
[New LWP 3640]
[New LWP 3641]
[New LWP 3642]
[New LWP 3645]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007f2a48498d50 in ?? ()
Id Target Id Frame
* 1 LWP 3510 "kudu" 0x00007f2a48498d50 in ?? ()
2 LWP 3514 "kudu" 0x00007f2a48494fb9 in ?? ()
3 LWP 3515 "kudu" 0x00007f2a48494fb9 in ?? ()
4 LWP 3516 "kudu" 0x00007f2a48494fb9 in ?? ()
5 LWP 3517 "kernel-watcher-" 0x00007f2a48494fb9 in ?? ()
6 LWP 3523 "ntp client-3523" 0x00007f2a484989e2 in ?? ()
7 LWP 3524 "file cache-evic" 0x00007f2a48494fb9 in ?? ()
8 LWP 3525 "sq_acceptor" 0x00007f2a411bacb9 in ?? ()
9 LWP 3528 "rpc reactor-352" 0x00007f2a411c7a47 in ?? ()
10 LWP 3529 "rpc reactor-352" 0x00007f2a411c7a47 in ?? ()
11 LWP 3530 "rpc reactor-353" 0x00007f2a411c7a47 in ?? ()
12 LWP 3531 "rpc reactor-353" 0x00007f2a411c7a47 in ?? ()
13 LWP 3532 "MaintenanceMgr " 0x00007f2a48494ad3 in ?? ()
14 LWP 3533 "txn-status-mana" 0x00007f2a48494fb9 in ?? ()
15 LWP 3534 "collect_and_rem" 0x00007f2a48494fb9 in ?? ()
16 LWP 3535 "tc-session-exp-" 0x00007f2a48494fb9 in ?? ()
17 LWP 3536 "rpc worker-3536" 0x00007f2a48494ad3 in ?? ()
18 LWP 3537 "rpc worker-3537" 0x00007f2a48494ad3 in ?? ()
19 LWP 3538 "rpc worker-3538" 0x00007f2a48494ad3 in ?? ()
20 LWP 3539 "rpc worker-3539" 0x00007f2a48494ad3 in ?? ()
21 LWP 3540 "rpc worker-3540" 0x00007f2a48494ad3 in ?? ()
22 LWP 3541 "rpc worker-3541" 0x00007f2a48494ad3 in ?? ()
23 LWP 3542 "rpc worker-3542" 0x00007f2a48494ad3 in ?? ()
24 LWP 3543 "rpc worker-3543" 0x00007f2a48494ad3 in ?? ()
25 LWP 3544 "rpc worker-3544" 0x00007f2a48494ad3 in ?? ()
26 LWP 3545 "rpc worker-3545" 0x00007f2a48494ad3 in ?? ()
27 LWP 3546 "rpc worker-3546" 0x00007f2a48494ad3 in ?? ()
28 LWP 3547 "rpc worker-3547" 0x00007f2a48494ad3 in ?? ()
29 LWP 3548 "rpc worker-3548" 0x00007f2a48494ad3 in ?? ()
30 LWP 3549 "rpc worker-3549" 0x00007f2a48494ad3 in ?? ()
31 LWP 3550 "rpc worker-3550" 0x00007f2a48494ad3 in ?? ()
32 LWP 3551 "rpc worker-3551" 0x00007f2a48494ad3 in ?? ()
33 LWP 3552 "rpc worker-3552" 0x00007f2a48494ad3 in ?? ()
34 LWP 3553 "rpc worker-3553" 0x00007f2a48494ad3 in ?? ()
35 LWP 3554 "rpc worker-3554" 0x00007f2a48494ad3 in ?? ()
36 LWP 3555 "rpc worker-3555" 0x00007f2a48494ad3 in ?? ()
37 LWP 3556 "rpc worker-3556" 0x00007f2a48494ad3 in ?? ()
38 LWP 3557 "rpc worker-3557" 0x00007f2a48494ad3 in ?? ()
39 LWP 3558 "rpc worker-3558" 0x00007f2a48494ad3 in ?? ()
40 LWP 3559 "rpc worker-3559" 0x00007f2a48494ad3 in ?? ()
41 LWP 3561 "rpc worker-3561" 0x00007f2a48494ad3 in ?? ()
42 LWP 3562 "rpc worker-3562" 0x00007f2a48494ad3 in ?? ()
43 LWP 3563 "rpc worker-3563" 0x00007f2a48494ad3 in ?? ()
44 LWP 3564 "rpc worker-3564" 0x00007f2a48494ad3 in ?? ()
45 LWP 3565 "rpc worker-3565" 0x00007f2a48494ad3 in ?? ()
46 LWP 3566 "rpc worker-3566" 0x00007f2a48494ad3 in ?? ()
47 LWP 3567 "rpc worker-3567" 0x00007f2a48494ad3 in ?? ()
48 LWP 3568 "rpc worker-3568" 0x00007f2a48494ad3 in ?? ()
49 LWP 3569 "rpc worker-3569" 0x00007f2a48494ad3 in ?? ()
50 LWP 3570 "rpc worker-3570" 0x00007f2a48494ad3 in ?? ()
51 LWP 3571 "rpc worker-3571" 0x00007f2a48494ad3 in ?? ()
52 LWP 3572 "rpc worker-3572" 0x00007f2a48494ad3 in ?? ()
53 LWP 3573 "rpc worker-3573" 0x00007f2a48494ad3 in ?? ()
54 LWP 3574 "rpc worker-3574" 0x00007f2a48494ad3 in ?? ()
55 LWP 3575 "rpc worker-3575" 0x00007f2a48494ad3 in ?? ()
56 LWP 3576 "rpc worker-3576" 0x00007f2a48494ad3 in ?? ()
57 LWP 3577 "rpc worker-3577" 0x00007f2a48494ad3 in ?? ()
58 LWP 3578 "rpc worker-3578" 0x00007f2a48494ad3 in ?? ()
59 LWP 3579 "rpc worker-3579" 0x00007f2a48494ad3 in ?? ()
60 LWP 3580 "rpc worker-3580" 0x00007f2a48494ad3 in ?? ()
61 LWP 3581 "rpc worker-3581" 0x00007f2a48494ad3 in ?? ()
62 LWP 3582 "rpc worker-3582" 0x00007f2a48494ad3 in ?? ()
63 LWP 3583 "rpc worker-3583" 0x00007f2a48494ad3 in ?? ()
64 LWP 3584 "rpc worker-3584" 0x00007f2a48494ad3 in ?? ()
65 LWP 3586 "rpc worker-3586" 0x00007f2a48494ad3 in ?? ()
66 LWP 3587 "rpc worker-3587" 0x00007f2a48494ad3 in ?? ()
67 LWP 3588 "rpc worker-3588" 0x00007f2a48494ad3 in ?? ()
68 LWP 3589 "rpc worker-3589" 0x00007f2a48494ad3 in ?? ()
69 LWP 3590 "rpc worker-3590" 0x00007f2a48494ad3 in ?? ()
70 LWP 3591 "rpc worker-3591" 0x00007f2a48494ad3 in ?? ()
71 LWP 3592 "rpc worker-3592" 0x00007f2a48494ad3 in ?? ()
72 LWP 3593 "rpc worker-3593" 0x00007f2a48494ad3 in ?? ()
73 LWP 3594 "rpc worker-3594" 0x00007f2a48494ad3 in ?? ()
74 LWP 3595 "rpc worker-3595" 0x00007f2a48494ad3 in ?? ()
75 LWP 3596 "rpc worker-3596" 0x00007f2a48494ad3 in ?? ()
76 LWP 3597 "rpc worker-3597" 0x00007f2a48494ad3 in ?? ()
77 LWP 3598 "rpc worker-3598" 0x00007f2a48494ad3 in ?? ()
78 LWP 3599 "rpc worker-3599" 0x00007f2a48494ad3 in ?? ()
79 LWP 3600 "rpc worker-3600" 0x00007f2a48494ad3 in ?? ()
80 LWP 3601 "rpc worker-3601" 0x00007f2a48494ad3 in ?? ()
81 LWP 3602 "rpc worker-3602" 0x00007f2a48494ad3 in ?? ()
82 LWP 3603 "rpc worker-3603" 0x00007f2a48494ad3 in ?? ()
83 LWP 3604 "rpc worker-3604" 0x00007f2a48494ad3 in ?? ()
84 LWP 3605 "rpc worker-3605" 0x00007f2a48494ad3 in ?? ()
85 LWP 3606 "rpc worker-3606" 0x00007f2a48494ad3 in ?? ()
86 LWP 3607 "rpc worker-3607" 0x00007f2a48494ad3 in ?? ()
87 LWP 3608 "rpc worker-3608" 0x00007f2a48494ad3 in ?? ()
88 LWP 3609 "rpc worker-3609" 0x00007f2a48494ad3 in ?? ()
89 LWP 3610 "rpc worker-3610" 0x00007f2a48494ad3 in ?? ()
90 LWP 3611 "rpc worker-3611" 0x00007f2a48494ad3 in ?? ()
91 LWP 3612 "rpc worker-3612" 0x00007f2a48494ad3 in ?? ()
92 LWP 3613 "rpc worker-3613" 0x00007f2a48494ad3 in ?? ()
93 LWP 3614 "rpc worker-3614" 0x00007f2a48494ad3 in ?? ()
94 LWP 3615 "rpc worker-3615" 0x00007f2a48494ad3 in ?? ()
95 LWP 3616 "rpc worker-3616" 0x00007f2a48494ad3 in ?? ()
96 LWP 3617 "rpc worker-3617" 0x00007f2a48494ad3 in ?? ()
97 LWP 3618 "rpc worker-3618" 0x00007f2a48494ad3 in ?? ()
98 LWP 3619 "rpc worker-3619" 0x00007f2a48494ad3 in ?? ()
99 LWP 3620 "rpc worker-3620" 0x00007f2a48494ad3 in ?? ()
100 LWP 3621 "rpc worker-3621" 0x00007f2a48494ad3 in ?? ()
101 LWP 3622 "rpc worker-3622" 0x00007f2a48494ad3 in ?? ()
102 LWP 3623 "rpc worker-3623" 0x00007f2a48494ad3 in ?? ()
103 LWP 3624 "rpc worker-3624" 0x00007f2a48494ad3 in ?? ()
104 LWP 3625 "rpc worker-3625" 0x00007f2a48494ad3 in ?? ()
105 LWP 3626 "rpc worker-3626" 0x00007f2a48494ad3 in ?? ()
106 LWP 3627 "rpc worker-3627" 0x00007f2a48494ad3 in ?? ()
107 LWP 3628 "rpc worker-3628" 0x00007f2a48494ad3 in ?? ()
108 LWP 3629 "rpc worker-3629" 0x00007f2a48494ad3 in ?? ()
109 LWP 3630 "rpc worker-3630" 0x00007f2a48494ad3 in ?? ()
110 LWP 3631 "rpc worker-3631" 0x00007f2a48494ad3 in ?? ()
111 LWP 3632 "rpc worker-3632" 0x00007f2a48494ad3 in ?? ()
112 LWP 3633 "rpc worker-3633" 0x00007f2a48494ad3 in ?? ()
113 LWP 3634 "rpc worker-3634" 0x00007f2a48494ad3 in ?? ()
114 LWP 3635 "rpc worker-3635" 0x00007f2a48494ad3 in ?? ()
115 LWP 3636 "rpc worker-3636" 0x00007f2a48494ad3 in ?? ()
116 LWP 3637 "rpc worker-3637" 0x00007f2a48494ad3 in ?? ()
117 LWP 3638 "diag-logger-363" 0x00007f2a48494fb9 in ?? ()
118 LWP 3639 "result-tracker-" 0x00007f2a48494fb9 in ?? ()
119 LWP 3640 "excess-log-dele" 0x00007f2a48494fb9 in ?? ()
120 LWP 3641 "acceptor-3641" 0x00007f2a411c90c7 in ?? ()
121 LWP 3642 "heartbeat-3642" 0x00007f2a48494fb9 in ?? ()
122 LWP 3645 "maintenance_sch" 0x00007f2a48494fb9 in ?? ()
Thread 122 (LWP 3645):
#0 0x00007f2a48494fb9 in ?? ()
#1 0x00007f29fdd3c6e0 in ?? ()
#2 0x00000000000002b1 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f29fdd3c730 in ?? ()
#6 0x0000000000000562 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 3642):
#0 0x00007f2a48494fb9 in ?? ()
#1 0x8ea7c1ad18c34923 in ?? ()
#2 0x00000000000000ab in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019400 in ?? ()
#5 0x00007f29fe5545f0 in ?? ()
#6 0x0000000000000156 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 3641):
#0 0x00007f2a411c90c7 in ?? ()
#1 0x00006140000212a8 in ?? ()
#2 0x00007f29fed7bd90 in ?? ()
#3 0x00007f29fed7bdc0 in ?? ()
#4 0x00007f29fed7bec0 in ?? ()
#5 0x00007f29fed7bdb0 in ?? ()
#6 0x00007f29fed7be20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 3640):
#0 0x00007f2a48494fb9 in ?? ()
#1 0x00007f29ff595f40 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffcab33acd0 in ?? ()
#5 0x00007f29ff595f30 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 3639):
#0 0x00007f2a48494fb9 in ?? ()
#1 0x00007f29ffdae100 in ?? ()
#2 0x00000000000000ac in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f29ffdae0f0 in ?? ()
#6 0x0000000000000158 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 3638):
#0 0x00007f2a48494fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 3637):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000027 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007f2a00ddde90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2a00dddeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007f2a48494770 in ?? ()
#10 0x00007f2a00dddeb0 in ?? ()
#11 0x00000fe5401bbbd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 3636):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000015 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007f2a015f5e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2a015f5eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007f2a48494770 in ?? ()
#10 0x00007f2a015f5eb0 in ?? ()
#11 0x00000fe5402bebd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 3635):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 3634):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 3633):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 3632):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 3631):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 3630):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 3629):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 3628):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 3627):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 3626):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 3625):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 3624):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 3623):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 3622):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 3621):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 3620):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 3619):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 3618):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 3617):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 3616):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 3615):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 3614):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 3613):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 3612):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 3611):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 3610):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 3609):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 3608):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 3607):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 3606):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 3605):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 3604):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 3603):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 3602):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 3601):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 3600):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 3599):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 3598):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 3597):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a688c in ?? ()
#4 0x00007f2a151aee90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2a151aeeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a6840 in ?? ()
#9 0x00007f2a48494770 in ?? ()
#10 0x00007f2a151aeeb0 in ?? ()
#11 0x00000fe542a35dd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 3596):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 3595):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 3594):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 3593):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 3592):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 3591):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 3590):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 3589):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 3588):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 3587):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 3586):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 3584):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 3583):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 3582):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 3581):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 3580):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 3579):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 3578):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 3577):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 3576):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012003c in ?? ()
#4 0x00007f2a1f38fe90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2a1f38feb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00011fff0 in ?? ()
#9 0x00007f2a48494770 in ?? ()
#10 0x00007f2a1f38feb0 in ?? ()
#11 0x00000fe543e71fd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 3575):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 3574):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 3573):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 3572):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 3571):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 3570):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 3569):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 3568):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 3567):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 3566):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 3565):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 3564):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 3563):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 3562):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 3561):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 3559):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 3558):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 3557):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 3556):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 3555):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007f2a28d58e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2a28d58eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007f2a48494770 in ?? ()
#10 0x00007f2a28d58eb0 in ?? ()
#11 0x00000fe5451ab1d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 3554):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 3553):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 3552):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 3551):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 3550):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 3549):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 3548):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 3547):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 3546):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 3545):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 3544):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 3543):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 3542):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 3541):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 3540):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 3539):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 3538):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 3537):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 3536):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 3535):
#0 0x00007f2a48494fb9 in ?? ()
#1 0x00007f2a32f59cc0 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019ca0 in ?? ()
#5 0x00007f2a32f59cb0 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 3534):
#0 0x00007f2a48494fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 3533):
#0 0x00007f2a48494fb9 in ?? ()
#1 0x00007f2a33f97240 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007f2a33f97230 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 3532):
#0 0x00007f2a48494ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 3531):
#0 0x00007f2a411c7a47 in ?? ()
#1 0x00007f2a34fb2320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f2a34fb2310 in ?? ()
#4 0x00007f2a34fb2520 in ?? ()
#5 0x00007f2a34fb2360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f2a34fb23e0 in ?? ()
#8 0x00007f2a4383a25d in ?? ()
#9 0x3fb9604e8e47c000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x00000000c8000000 in ?? ()
#14 0x00007f2a00000000 in ?? ()
#15 0x41da16179c5dd6b5 in ?? ()
#16 0x00000fe5c69ee47c in ?? ()
#17 0x00007f2a34fb23c0 in ?? ()
#18 0x00007f2a4383eba3 in ?? ()
#19 0x00007f2a34fb2390 in ?? ()
#20 0x3fb9604e8e47c000 in ?? ()
#21 0x0000000034fb23e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb9604e8e47c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 3530):
#0 0x00007f2a411c7a47 in ?? ()
#1 0x00007f2a357b3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f2a357b3310 in ?? ()
#4 0x00007f2a357b3520 in ?? ()
#5 0x00007f2a357b3360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f2a357b33e0 in ?? ()
#8 0x00007f2a4383a25d in ?? ()
#9 0x3fb97a1827c82000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x000000006a000000 in ?? ()
#14 0x00007f2a00000000 in ?? ()
#15 0x41da16179c5dd6b5 in ?? ()
#16 0x00000fe5c6aee67c in ?? ()
#17 0x00007f2a357b33c0 in ?? ()
#18 0x00007f2a4383eba3 in ?? ()
#19 0x00007f2a357b3390 in ?? ()
#20 0x3fb97a1827c82000 in ?? ()
#21 0x00000000357b33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb97a1827c82000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 3529):
#0 0x00007f2a411c7a47 in ?? ()
#1 0x00007f2a35fb4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f2a35fb4310 in ?? ()
#4 0x00007f2a35fb4520 in ?? ()
#5 0x00007f2a35fb4360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f2a35fb43e0 in ?? ()
#8 0x00007f2a4383a25d in ?? ()
#9 0x3fb9711056c36000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x000000006e000000 in ?? ()
#14 0x00007f2a00000000 in ?? ()
#15 0x41da16179c5dd6ba in ?? ()
#16 0x00000fe5c6bee87c in ?? ()
#17 0x00007f2a35fb43c0 in ?? ()
#18 0x00007f2a4383eba3 in ?? ()
#19 0x00007f2a35fb4390 in ?? ()
#20 0x3fb9711056c36000 in ?? ()
#21 0x0000000035fb43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb9711056c36000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 3528):
#0 0x00007f2a411c7a47 in ?? ()
#1 0x00007f2a37fad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f2a37fad310 in ?? ()
#4 0x00007f2a37fad520 in ?? ()
#5 0x00007f2a37fad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007f2a37fad3e0 in ?? ()
#8 0x00007f2a4383a25d in ?? ()
#9 0x3fb95e66b9842000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x00000000c0000000 in ?? ()
#14 0x00007f2a00000000 in ?? ()
#15 0x41da16179c5dd6b5 in ?? ()
#16 0x00000fe5c6feda7c in ?? ()
#17 0x00007f2a37fad3c0 in ?? ()
#18 0x00007f2a4383eba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 3525):
#0 0x00007f2a411bacb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f2a399b77c8 in ?? ()
#3 0x000060200001dc90 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 3524):
#0 0x00007f2a48494fb9 in ?? ()
#1 0x00007f2a391b60c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007f2a391b60b0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 3523):
#0 0x00007f2a484989e2 in ?? ()
#1 0x00007f2a389b3ba0 in ?? ()
#2 0x00007f2a389b3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f2a389b3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007f2a389b3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f2a389b3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007f2a4ddabedc in ?? ()
#12 0x00007f2a4dd9b8d9 in ?? ()
#13 0x00007f2a4dd9fec6 in ?? ()
#14 0x00007f2a4dda4900 in ?? ()
#15 0x00007f2a4dda4729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007f2a450704fe in ?? ()
#18 0x00007f2a4848e6db in ?? ()
#19 0x00000fe5c712e684 in ?? ()
#20 0x00007f2a389b3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 3517):
#0 0x00007f2a48494fb9 in ?? ()
#1 0x00007f2a3a9b8c80 in ?? ()
#2 0x0000000000000362 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007f2a3a9b8c70 in ?? ()
#6 0x00000000000006c4 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 3516):
#0 0x00007f2a48494fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 3515):
#0 0x00007f2a48494fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 3514):
#0 0x00007f2a48494fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f2a3c1bc450 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 3510):
#0 0x00007f2a48498d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250622 19:58:13.774436 2016 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 6 with UUID 07ce178dce43415a852dbcce80f47f3e and pid 3792
************************ BEGIN STACKS **************************
[New LWP 3795]
[New LWP 3796]
[New LWP 3797]
[New LWP 3798]
[New LWP 3804]
[New LWP 3805]
[New LWP 3806]
[New LWP 3811]
[New LWP 3812]
[New LWP 3813]
[New LWP 3814]
[New LWP 3815]
[New LWP 3816]
[New LWP 3817]
[New LWP 3818]
[New LWP 3819]
[New LWP 3820]
[New LWP 3821]
[New LWP 3822]
[New LWP 3823]
[New LWP 3824]
[New LWP 3825]
[New LWP 3826]
[New LWP 3827]
[New LWP 3828]
[New LWP 3829]
[New LWP 3830]
[New LWP 3831]
[New LWP 3832]
[New LWP 3833]
[New LWP 3834]
[New LWP 3835]
[New LWP 3836]
[New LWP 3837]
[New LWP 3838]
[New LWP 3839]
[New LWP 3840]
[New LWP 3841]
[New LWP 3842]
[New LWP 3843]
[New LWP 3844]
[New LWP 3845]
[New LWP 3846]
[New LWP 3847]
[New LWP 3848]
[New LWP 3849]
[New LWP 3850]
[New LWP 3851]
[New LWP 3852]
[New LWP 3853]
[New LWP 3854]
[New LWP 3855]
[New LWP 3856]
[New LWP 3857]
[New LWP 3858]
[New LWP 3859]
[New LWP 3860]
[New LWP 3861]
[New LWP 3862]
[New LWP 3863]
[New LWP 3864]
[New LWP 3865]
[New LWP 3866]
[New LWP 3867]
[New LWP 3868]
[New LWP 3869]
[New LWP 3870]
[New LWP 3871]
[New LWP 3872]
[New LWP 3873]
[New LWP 3874]
[New LWP 3875]
[New LWP 3876]
[New LWP 3877]
[New LWP 3878]
[New LWP 3879]
[New LWP 3880]
[New LWP 3881]
[New LWP 3882]
[New LWP 3883]
[New LWP 3884]
[New LWP 3886]
[New LWP 3887]
[New LWP 3888]
[New LWP 3889]
[New LWP 3890]
[New LWP 3891]
[New LWP 3892]
[New LWP 3893]
[New LWP 3894]
[New LWP 3895]
[New LWP 3896]
[New LWP 3897]
[New LWP 3898]
[New LWP 3899]
[New LWP 3900]
[New LWP 3901]
[New LWP 3902]
[New LWP 3903]
[New LWP 3904]
[New LWP 3905]
[New LWP 3906]
[New LWP 3907]
[New LWP 3908]
[New LWP 3909]
[New LWP 3910]
[New LWP 3911]
[New LWP 3912]
[New LWP 3913]
[New LWP 3914]
[New LWP 3915]
[New LWP 3916]
[New LWP 3917]
[New LWP 3918]
[New LWP 3919]
[New LWP 3920]
[New LWP 3921]
[New LWP 3922]
[New LWP 3923]
[New LWP 3924]
[New LWP 3925]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007f4bac96dd50 in ?? ()
Id Target Id Frame
* 1 LWP 3792 "kudu" 0x00007f4bac96dd50 in ?? ()
2 LWP 3795 "kudu" 0x00007f4bac969fb9 in ?? ()
3 LWP 3796 "kudu" 0x00007f4bac969fb9 in ?? ()
4 LWP 3797 "kudu" 0x00007f4bac969fb9 in ?? ()
5 LWP 3798 "kernel-watcher-" 0x00007f4bac969fb9 in ?? ()
6 LWP 3804 "ntp client-3804" 0x00007f4bac96d9e2 in ?? ()
7 LWP 3805 "file cache-evic" 0x00007f4bac969fb9 in ?? ()
8 LWP 3806 "sq_acceptor" 0x00007f4ba568fcb9 in ?? ()
9 LWP 3811 "rpc reactor-381" 0x00007f4ba569ca47 in ?? ()
10 LWP 3812 "rpc reactor-381" 0x00007f4ba569ca47 in ?? ()
11 LWP 3813 "rpc reactor-381" 0x00007f4ba569ca47 in ?? ()
12 LWP 3814 "rpc reactor-381" 0x00007f4ba569ca47 in ?? ()
13 LWP 3815 "MaintenanceMgr " 0x00007f4bac969ad3 in ?? ()
14 LWP 3816 "txn-status-mana" 0x00007f4bac969fb9 in ?? ()
15 LWP 3817 "collect_and_rem" 0x00007f4bac969fb9 in ?? ()
16 LWP 3818 "tc-session-exp-" 0x00007f4bac969fb9 in ?? ()
17 LWP 3819 "rpc worker-3819" 0x00007f4bac969ad3 in ?? ()
18 LWP 3820 "rpc worker-3820" 0x00007f4bac969ad3 in ?? ()
19 LWP 3821 "rpc worker-3821" 0x00007f4bac969ad3 in ?? ()
20 LWP 3822 "rpc worker-3822" 0x00007f4bac969ad3 in ?? ()
21 LWP 3823 "rpc worker-3823" 0x00007f4bac969ad3 in ?? ()
22 LWP 3824 "rpc worker-3824" 0x00007f4bac969ad3 in ?? ()
23 LWP 3825 "rpc worker-3825" 0x00007f4bac969ad3 in ?? ()
24 LWP 3826 "rpc worker-3826" 0x00007f4bac969ad3 in ?? ()
25 LWP 3827 "rpc worker-3827" 0x00007f4bac969ad3 in ?? ()
26 LWP 3828 "rpc worker-3828" 0x00007f4bac969ad3 in ?? ()
27 LWP 3829 "rpc worker-3829" 0x00007f4bac969ad3 in ?? ()
28 LWP 3830 "rpc worker-3830" 0x00007f4bac969ad3 in ?? ()
29 LWP 3831 "rpc worker-3831" 0x00007f4bac969ad3 in ?? ()
30 LWP 3832 "rpc worker-3832" 0x00007f4bac969ad3 in ?? ()
31 LWP 3833 "rpc worker-3833" 0x00007f4bac969ad3 in ?? ()
32 LWP 3834 "rpc worker-3834" 0x00007f4bac969ad3 in ?? ()
33 LWP 3835 "rpc worker-3835" 0x00007f4bac969ad3 in ?? ()
34 LWP 3836 "rpc worker-3836" 0x00007f4bac969ad3 in ?? ()
35 LWP 3837 "rpc worker-3837" 0x00007f4bac969ad3 in ?? ()
36 LWP 3838 "rpc worker-3838" 0x00007f4bac969ad3 in ?? ()
37 LWP 3839 "rpc worker-3839" 0x00007f4bac969ad3 in ?? ()
38 LWP 3840 "rpc worker-3840" 0x00007f4bac969ad3 in ?? ()
39 LWP 3841 "rpc worker-3841" 0x00007f4bac969ad3 in ?? ()
40 LWP 3842 "rpc worker-3842" 0x00007f4bac969ad3 in ?? ()
41 LWP 3843 "rpc worker-3843" 0x00007f4bac969ad3 in ?? ()
42 LWP 3844 "rpc worker-3844" 0x00007f4bac969ad3 in ?? ()
43 LWP 3845 "rpc worker-3845" 0x00007f4bac969ad3 in ?? ()
44 LWP 3846 "rpc worker-3846" 0x00007f4bac969ad3 in ?? ()
45 LWP 3847 "rpc worker-3847" 0x00007f4bac969ad3 in ?? ()
46 LWP 3848 "rpc worker-3848" 0x00007f4bac969ad3 in ?? ()
47 LWP 3849 "rpc worker-3849" 0x00007f4bac969ad3 in ?? ()
48 LWP 3850 "rpc worker-3850" 0x00007f4bac969ad3 in ?? ()
49 LWP 3851 "rpc worker-3851" 0x00007f4bac969ad3 in ?? ()
50 LWP 3852 "rpc worker-3852" 0x00007f4bac969ad3 in ?? ()
51 LWP 3853 "rpc worker-3853" 0x00007f4bac969ad3 in ?? ()
52 LWP 3854 "rpc worker-3854" 0x00007f4bac969ad3 in ?? ()
53 LWP 3855 "rpc worker-3855" 0x00007f4bac969ad3 in ?? ()
54 LWP 3856 "rpc worker-3856" 0x00007f4bac969ad3 in ?? ()
55 LWP 3857 "rpc worker-3857" 0x00007f4bac969ad3 in ?? ()
56 LWP 3858 "rpc worker-3858" 0x00007f4bac969ad3 in ?? ()
57 LWP 3859 "rpc worker-3859" 0x00007f4bac969ad3 in ?? ()
58 LWP 3860 "rpc worker-3860" 0x00007f4bac969ad3 in ?? ()
59 LWP 3861 "rpc worker-3861" 0x00007f4bac969ad3 in ?? ()
60 LWP 3862 "rpc worker-3862" 0x00007f4bac969ad3 in ?? ()
61 LWP 3863 "rpc worker-3863" 0x00007f4bac969ad3 in ?? ()
62 LWP 3864 "rpc worker-3864" 0x00007f4bac969ad3 in ?? ()
63 LWP 3865 "rpc worker-3865" 0x00007f4bac969ad3 in ?? ()
64 LWP 3866 "rpc worker-3866" 0x00007f4bac969ad3 in ?? ()
65 LWP 3867 "rpc worker-3867" 0x00007f4bac969ad3 in ?? ()
66 LWP 3868 "rpc worker-3868" 0x00007f4bac969ad3 in ?? ()
67 LWP 3869 "rpc worker-3869" 0x00007f4bac969ad3 in ?? ()
68 LWP 3870 "rpc worker-3870" 0x00007f4bac969ad3 in ?? ()
69 LWP 3871 "rpc worker-3871" 0x00007f4bac969ad3 in ?? ()
70 LWP 3872 "rpc worker-3872" 0x00007f4bac969ad3 in ?? ()
71 LWP 3873 "rpc worker-3873" 0x00007f4bac969ad3 in ?? ()
72 LWP 3874 "rpc worker-3874" 0x00007f4bac969ad3 in ?? ()
73 LWP 3875 "rpc worker-3875" 0x00007f4bac969ad3 in ?? ()
74 LWP 3876 "rpc worker-3876" 0x00007f4bac969ad3 in ?? ()
75 LWP 3877 "rpc worker-3877" 0x00007f4bac969ad3 in ?? ()
76 LWP 3878 "rpc worker-3878" 0x00007f4bac969ad3 in ?? ()
77 LWP 3879 "rpc worker-3879" 0x00007f4bac969ad3 in ?? ()
78 LWP 3880 "rpc worker-3880" 0x00007f4bac969ad3 in ?? ()
79 LWP 3881 "rpc worker-3881" 0x00007f4bac969ad3 in ?? ()
80 LWP 3882 "rpc worker-3882" 0x00007f4bac969ad3 in ?? ()
81 LWP 3883 "rpc worker-3883" 0x00007f4bac969ad3 in ?? ()
82 LWP 3884 "rpc worker-3884" 0x00007f4bac969ad3 in ?? ()
83 LWP 3886 "rpc worker-3886" 0x00007f4bac969ad3 in ?? ()
84 LWP 3887 "rpc worker-3887" 0x00007f4bac969ad3 in ?? ()
85 LWP 3888 "rpc worker-3888" 0x00007f4bac969ad3 in ?? ()
86 LWP 3889 "rpc worker-3889" 0x00007f4bac969ad3 in ?? ()
87 LWP 3890 "rpc worker-3890" 0x00007f4bac969ad3 in ?? ()
88 LWP 3891 "rpc worker-3891" 0x00007f4bac969ad3 in ?? ()
89 LWP 3892 "rpc worker-3892" 0x00007f4bac969ad3 in ?? ()
90 LWP 3893 "rpc worker-3893" 0x00007f4bac969ad3 in ?? ()
91 LWP 3894 "rpc worker-3894" 0x00007f4bac969ad3 in ?? ()
92 LWP 3895 "rpc worker-3895" 0x00007f4bac969ad3 in ?? ()
93 LWP 3896 "rpc worker-3896" 0x00007f4bac969ad3 in ?? ()
94 LWP 3897 "rpc worker-3897" 0x00007f4bac969ad3 in ?? ()
95 LWP 3898 "rpc worker-3898" 0x00007f4bac969ad3 in ?? ()
96 LWP 3899 "rpc worker-3899" 0x00007f4bac969ad3 in ?? ()
97 LWP 3900 "rpc worker-3900" 0x00007f4bac969ad3 in ?? ()
98 LWP 3901 "rpc worker-3901" 0x00007f4bac969ad3 in ?? ()
99 LWP 3902 "rpc worker-3902" 0x00007f4bac969ad3 in ?? ()
100 LWP 3903 "rpc worker-3903" 0x00007f4bac969ad3 in ?? ()
101 LWP 3904 "rpc worker-3904" 0x00007f4bac969ad3 in ?? ()
102 LWP 3905 "rpc worker-3905" 0x00007f4bac969ad3 in ?? ()
103 LWP 3906 "rpc worker-3906" 0x00007f4bac969ad3 in ?? ()
104 LWP 3907 "rpc worker-3907" 0x00007f4bac969ad3 in ?? ()
105 LWP 3908 "rpc worker-3908" 0x00007f4bac969ad3 in ?? ()
106 LWP 3909 "rpc worker-3909" 0x00007f4bac969ad3 in ?? ()
107 LWP 3910 "rpc worker-3910" 0x00007f4bac969ad3 in ?? ()
108 LWP 3911 "rpc worker-3911" 0x00007f4bac969ad3 in ?? ()
109 LWP 3912 "rpc worker-3912" 0x00007f4bac969ad3 in ?? ()
110 LWP 3913 "rpc worker-3913" 0x00007f4bac969ad3 in ?? ()
111 LWP 3914 "rpc worker-3914" 0x00007f4bac969ad3 in ?? ()
112 LWP 3915 "rpc worker-3915" 0x00007f4bac969ad3 in ?? ()
113 LWP 3916 "rpc worker-3916" 0x00007f4bac969ad3 in ?? ()
114 LWP 3917 "rpc worker-3917" 0x00007f4bac969ad3 in ?? ()
115 LWP 3918 "rpc worker-3918" 0x00007f4bac969ad3 in ?? ()
116 LWP 3919 "rpc worker-3919" 0x00007f4bac969ad3 in ?? ()
117 LWP 3920 "diag-logger-392" 0x00007f4bac969fb9 in ?? ()
118 LWP 3921 "result-tracker-" 0x00007f4bac969fb9 in ?? ()
119 LWP 3922 "excess-log-dele" 0x00007f4bac969fb9 in ?? ()
120 LWP 3923 "acceptor-3923" 0x00007f4ba569e0c7 in ?? ()
121 LWP 3924 "heartbeat-3924" 0x00007f4bac969fb9 in ?? ()
122 LWP 3925 "maintenance_sch" 0x00007f4bac969fb9 in ?? ()
Thread 122 (LWP 3925):
#0 0x00007f4bac969fb9 in ?? ()
#1 0x00007f4b622116e0 in ?? ()
#2 0x00000000000002a2 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f4b62211730 in ?? ()
#6 0x0000000000000544 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 3924):
#0 0x00007f4bac969fb9 in ?? ()
#1 0x8ea7c1ad18c34923 in ?? ()
#2 0x00000000000000a8 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019400 in ?? ()
#5 0x00007f4b62a295f0 in ?? ()
#6 0x0000000000000150 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 3923):
#0 0x00007f4ba569e0c7 in ?? ()
#1 0x00006140000216a8 in ?? ()
#2 0x00007f4b63250d90 in ?? ()
#3 0x00007f4b63250dc0 in ?? ()
#4 0x00007f4b63250ec0 in ?? ()
#5 0x00007f4b63250db0 in ?? ()
#6 0x00007f4b63250e20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 3922):
#0 0x00007f4bac969fb9 in ?? ()
#1 0x00007f4b63a6af40 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffd58b8c930 in ?? ()
#5 0x00007f4b63a6af30 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 3921):
#0 0x00007f4bac969fb9 in ?? ()
#1 0x00007f4b64283100 in ?? ()
#2 0x00000000000000a8 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f4b642830f0 in ?? ()
#6 0x0000000000000150 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 3920):
#0 0x00007f4bac969fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 3919):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000020 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002a6858 in ?? ()
#4 0x00007f4b652b2e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4b652b2eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 3918):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 3917):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x000000000000001c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007f4b662e2e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4b662e2eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 113 (LWP 3916):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 3915):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 3914):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 3913):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 3912):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 3911):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 3910):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 3909):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 3908):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 3907):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 3906):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 3905):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 3904):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 3903):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 3902):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 3901):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 3900):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 3899):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 3898):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 3897):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 3896):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 3895):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 3894):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 3893):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 3892):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 3891):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 3890):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 3889):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 3888):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 3887):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 3886):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 3884):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 3883):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 3882):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 3881):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 3880):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 3879):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 3878):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a688c in ?? ()
#4 0x00007f4b79683e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4b79683eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a6840 in ?? ()
#9 0x00007f4bac969770 in ?? ()
#10 0x00007f4b79683eb0 in ?? ()
#11 0x00000fe96f2d07d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 3877):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 3876):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 3875):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 3874):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 3873):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 3872):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 3871):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 3870):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 3869):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 3868):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 3867):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 3866):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 3865):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 3864):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 3863):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 3862):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 3861):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 3860):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 3859):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 3858):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012003c in ?? ()
#4 0x00007f4b8304ce90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4b8304ceb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00011fff0 in ?? ()
#9 0x00007f4bac969770 in ?? ()
#10 0x00007f4b8304ceb0 in ?? ()
#11 0x00000fe9706099d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 3857):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 3856):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 3855):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 3854):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 3853):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 3852):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 3851):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 3850):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 3849):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 3848):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 3847):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 3846):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 3845):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 3844):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 3843):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 3842):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 3841):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 3840):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 3839):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 3838):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x000000000000000f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007f4b8d22de90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4b8d22deb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007f4bac969770 in ?? ()
#10 0x00007f4b8d22deb0 in ?? ()
#11 0x00000fe971a45bd8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 3837):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 3836):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 3835):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 3834):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 3833):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 3832):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 3831):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 3830):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 3829):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 3828):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 3827):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 3826):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 3825):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 3824):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 3823):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 3822):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 3821):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 3820):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 3819):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 3818):
#0 0x00007f4bac969fb9 in ?? ()
#1 0x00007f4b97432cc0 in ?? ()
#2 0x0000000000000010 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019ca0 in ?? ()
#5 0x00007f4b97432cb0 in ?? ()
#6 0x0000000000000020 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 3817):
#0 0x00007f4bac969fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 3816):
#0 0x00007f4bac969fb9 in ?? ()
#1 0x00007f4b98471240 in ?? ()
#2 0x0000000000000010 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007f4b98471230 in ?? ()
#6 0x0000000000000020 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 3815):
#0 0x00007f4bac969ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 3814):
#0 0x00007f4ba569ca47 in ?? ()
#1 0x00007f4b994b2320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f4b994b2310 in ?? ()
#4 0x00007f4b994b2520 in ?? ()
#5 0x00007f4b994b2360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f4b994b23e0 in ?? ()
#8 0x00007f4ba7d0f25d in ?? ()
#9 0x3fb9771b32be7000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x0000000072800000 in ?? ()
#14 0x00007f4b00000000 in ?? ()
#15 0x41da16179c5dd6b5 in ?? ()
#16 0x00000fe9f328e47c in ?? ()
#17 0x00007f4b994b23c0 in ?? ()
#18 0x00007f4ba7d13ba3 in ?? ()
#19 0x00007f4b994b2390 in ?? ()
#20 0x3fb9771b32be7000 in ?? ()
#21 0x00000000994b23e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb9771b32be7000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 3813):
#0 0x00007f4ba569ca47 in ?? ()
#1 0x00007f4b99cb3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f4b99cb3310 in ?? ()
#4 0x00007f4b99cb3520 in ?? ()
#5 0x00007f4b99cb3360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f4b99cb33e0 in ?? ()
#8 0x00007f4ba7d0f25d in ?? ()
#9 0x3fb963a117f04000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x00000000ae000000 in ?? ()
#14 0x00007f4b00000000 in ?? ()
#15 0x41da16179c5dd6b5 in ?? ()
#16 0x00000fe9f338e67c in ?? ()
#17 0x00007f4b99cb33c0 in ?? ()
#18 0x00007f4ba7d13ba3 in ?? ()
#19 0x00007f4b99cb3390 in ?? ()
#20 0x3fb963a117f04000 in ?? ()
#21 0x0000000099cb33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb963a117f04000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 3812):
#0 0x00007f4ba569ca47 in ?? ()
#1 0x00007f4b9a4b4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f4b9a4b4310 in ?? ()
#4 0x00007f4b9a4b4520 in ?? ()
#5 0x00007f4b9a4b4360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f4b9a4b43e0 in ?? ()
#8 0x00007f4ba7d0f25d in ?? ()
#9 0x3fb95174ec298000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x0000000046000000 in ?? ()
#14 0x00007f4b00000000 in ?? ()
#15 0x41da16179c5dd6b6 in ?? ()
#16 0x00000fe9f348e87c in ?? ()
#17 0x00007f4b9a4b43c0 in ?? ()
#18 0x00007f4ba7d13ba3 in ?? ()
#19 0x00007f4b9a4b4390 in ?? ()
#20 0x3fb95174ec298000 in ?? ()
#21 0x000000009a4b43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb95174ec298000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 3811):
#0 0x00007f4ba569ca47 in ?? ()
#1 0x00007f4b9c4ad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f4b9c4ad310 in ?? ()
#4 0x00007f4b9c4ad520 in ?? ()
#5 0x00007f4b9c4ad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007f4b9c4ad3e0 in ?? ()
#8 0x00007f4ba7d0f25d in ?? ()
#9 0x3fb9520e3b92c000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x00000000fc800000 in ?? ()
#14 0x00007f4b00000000 in ?? ()
#15 0x41da16179c5dd6b5 in ?? ()
#16 0x00000fe9f388da7c in ?? ()
#17 0x00007f4b9c4ad3c0 in ?? ()
#18 0x00007f4ba7d13ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 3806):
#0 0x00007f4ba568fcb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f4b9deb77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 3805):
#0 0x00007f4bac969fb9 in ?? ()
#1 0x00007f4b9d6b60c0 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007f4b9d6b60b0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 3804):
#0 0x00007f4bac96d9e2 in ?? ()
#1 0x00007f4b9ceb3ba0 in ?? ()
#2 0x00007f4b9ceb3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f4b9ceb3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007f4b9ceb3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f4b9ceb3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007f4bb2280edc in ?? ()
#12 0x00007f4bb22708d9 in ?? ()
#13 0x00007f4bb2274ec6 in ?? ()
#14 0x00007f4bb2279900 in ?? ()
#15 0x00007f4bb2279729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007f4ba95454fe in ?? ()
#18 0x00007f4bac9636db in ?? ()
#19 0x00000fe9f39ce684 in ?? ()
#20 0x00007f4b9ceb3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 3798):
#0 0x00007f4bac969fb9 in ?? ()
#1 0x00007f4b9eeb8c80 in ?? ()
#2 0x000000000000034f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007f4b9eeb8c70 in ?? ()
#6 0x000000000000069e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 3797):
#0 0x00007f4bac969fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 3796):
#0 0x00007f4bac969fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 3795):
#0 0x00007f4bac969fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f4ba06bc450 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 3792):
#0 0x00007f4bac96dd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250622 19:58:14.560902 2016 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 7 with UUID 7e22adb60d65484c9844ee7d2108a396 and pid 5613
************************ BEGIN STACKS **************************
[New LWP 5615]
[New LWP 5616]
[New LWP 5617]
[New LWP 5618]
[New LWP 5624]
[New LWP 5625]
[New LWP 5626]
[New LWP 5629]
[New LWP 5630]
[New LWP 5631]
[New LWP 5632]
[New LWP 5633]
[New LWP 5634]
[New LWP 5636]
[New LWP 5637]
[New LWP 5638]
[New LWP 5639]
[New LWP 5640]
[New LWP 5641]
[New LWP 5642]
[New LWP 5643]
[New LWP 5644]
[New LWP 5645]
[New LWP 5646]
[New LWP 5647]
[New LWP 5648]
[New LWP 5649]
[New LWP 5650]
[New LWP 5651]
[New LWP 5652]
[New LWP 5654]
[New LWP 5655]
[New LWP 5656]
[New LWP 5657]
[New LWP 5658]
[New LWP 5659]
[New LWP 5660]
[New LWP 5661]
[New LWP 5662]
[New LWP 5663]
[New LWP 5664]
[New LWP 5665]
[New LWP 5666]
[New LWP 5667]
[New LWP 5668]
[New LWP 5669]
[New LWP 5670]
[New LWP 5671]
[New LWP 5672]
[New LWP 5673]
[New LWP 5674]
[New LWP 5675]
[New LWP 5676]
[New LWP 5677]
[New LWP 5678]
[New LWP 5679]
[New LWP 5680]
[New LWP 5681]
[New LWP 5682]
[New LWP 5685]
[New LWP 5686]
[New LWP 5687]
[New LWP 5688]
[New LWP 5689]
[New LWP 5690]
[New LWP 5691]
[New LWP 5692]
[New LWP 5693]
[New LWP 5694]
[New LWP 5695]
[New LWP 5696]
[New LWP 5697]
[New LWP 5698]
[New LWP 5699]
[New LWP 5700]
[New LWP 5701]
[New LWP 5702]
[New LWP 5703]
[New LWP 5704]
[New LWP 5705]
[New LWP 5706]
[New LWP 5707]
[New LWP 5708]
[New LWP 5709]
[New LWP 5710]
[New LWP 5711]
[New LWP 5712]
[New LWP 5713]
[New LWP 5714]
[New LWP 5715]
[New LWP 5716]
[New LWP 5717]
[New LWP 5718]
[New LWP 5719]
[New LWP 5720]
[New LWP 5721]
[New LWP 5722]
[New LWP 5723]
[New LWP 5725]
[New LWP 5726]
[New LWP 5727]
[New LWP 5728]
[New LWP 5729]
[New LWP 5730]
[New LWP 5731]
[New LWP 5732]
[New LWP 5733]
[New LWP 5734]
[New LWP 5735]
[New LWP 5736]
[New LWP 5738]
[New LWP 5739]
[New LWP 5741]
[New LWP 5742]
[New LWP 5743]
[New LWP 5744]
[New LWP 5745]
[New LWP 5746]
[New LWP 5747]
[New LWP 5748]
[New LWP 5751]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007fc5ba572d50 in ?? ()
Id Target Id Frame
* 1 LWP 5613 "kudu" 0x00007fc5ba572d50 in ?? ()
2 LWP 5615 "kudu" 0x00007fc5ba56efb9 in ?? ()
3 LWP 5616 "kudu" 0x00007fc5ba56efb9 in ?? ()
4 LWP 5617 "kudu" 0x00007fc5ba56efb9 in ?? ()
5 LWP 5618 "kernel-watcher-" 0x00007fc5ba56efb9 in ?? ()
6 LWP 5624 "ntp client-5624" 0x00007fc5ba5729e2 in ?? ()
7 LWP 5625 "file cache-evic" 0x00007fc5ba56efb9 in ?? ()
8 LWP 5626 "sq_acceptor" 0x00007fc5b3294cb9 in ?? ()
9 LWP 5629 "rpc reactor-562" 0x00007fc5b32a1a47 in ?? ()
10 LWP 5630 "rpc reactor-563" 0x00007fc5b32a1a47 in ?? ()
11 LWP 5631 "rpc reactor-563" 0x00007fc5b32a1a47 in ?? ()
12 LWP 5632 "rpc reactor-563" 0x00007fc5b32a1a47 in ?? ()
13 LWP 5633 "MaintenanceMgr " 0x00007fc5ba56ead3 in ?? ()
14 LWP 5634 "txn-status-mana" 0x00007fc5ba56efb9 in ?? ()
15 LWP 5636 "collect_and_rem" 0x00007fc5ba56efb9 in ?? ()
16 LWP 5637 "tc-session-exp-" 0x00007fc5ba56efb9 in ?? ()
17 LWP 5638 "rpc worker-5638" 0x00007fc5ba56ead3 in ?? ()
18 LWP 5639 "rpc worker-5639" 0x00007fc5ba56ead3 in ?? ()
19 LWP 5640 "rpc worker-5640" 0x00007fc5ba56ead3 in ?? ()
20 LWP 5641 "rpc worker-5641" 0x00007fc5ba56ead3 in ?? ()
21 LWP 5642 "rpc worker-5642" 0x00007fc5ba56ead3 in ?? ()
22 LWP 5643 "rpc worker-5643" 0x00007fc5ba56ead3 in ?? ()
23 LWP 5644 "rpc worker-5644" 0x00007fc5ba56ead3 in ?? ()
24 LWP 5645 "rpc worker-5645" 0x00007fc5ba56ead3 in ?? ()
25 LWP 5646 "rpc worker-5646" 0x00007fc5ba56ead3 in ?? ()
26 LWP 5647 "rpc worker-5647" 0x00007fc5ba56ead3 in ?? ()
27 LWP 5648 "rpc worker-5648" 0x00007fc5ba56ead3 in ?? ()
28 LWP 5649 "rpc worker-5649" 0x00007fc5ba56ead3 in ?? ()
29 LWP 5650 "rpc worker-5650" 0x00007fc5ba56ead3 in ?? ()
30 LWP 5651 "rpc worker-5651" 0x00007fc5ba56ead3 in ?? ()
31 LWP 5652 "rpc worker-5652" 0x00007fc5ba56ead3 in ?? ()
32 LWP 5654 "rpc worker-5654" 0x00007fc5ba56ead3 in ?? ()
33 LWP 5655 "rpc worker-5655" 0x00007fc5ba56ead3 in ?? ()
34 LWP 5656 "rpc worker-5656" 0x00007fc5ba56ead3 in ?? ()
35 LWP 5657 "rpc worker-5657" 0x00007fc5ba56ead3 in ?? ()
36 LWP 5658 "rpc worker-5658" 0x00007fc5ba56ead3 in ?? ()
37 LWP 5659 "rpc worker-5659" 0x00007fc5ba56ead3 in ?? ()
38 LWP 5660 "rpc worker-5660" 0x00007fc5ba56ead3 in ?? ()
39 LWP 5661 "rpc worker-5661" 0x00007fc5ba56ead3 in ?? ()
40 LWP 5662 "rpc worker-5662" 0x00007fc5ba56ead3 in ?? ()
41 LWP 5663 "rpc worker-5663" 0x00007fc5ba56ead3 in ?? ()
42 LWP 5664 "rpc worker-5664" 0x00007fc5ba56ead3 in ?? ()
43 LWP 5665 "rpc worker-5665" 0x00007fc5ba56ead3 in ?? ()
44 LWP 5666 "rpc worker-5666" 0x00007fc5ba56ead3 in ?? ()
45 LWP 5667 "rpc worker-5667" 0x00007fc5ba56ead3 in ?? ()
46 LWP 5668 "rpc worker-5668" 0x00007fc5ba56ead3 in ?? ()
47 LWP 5669 "rpc worker-5669" 0x00007fc5ba56ead3 in ?? ()
48 LWP 5670 "rpc worker-5670" 0x00007fc5ba56ead3 in ?? ()
49 LWP 5671 "rpc worker-5671" 0x00007fc5ba56ead3 in ?? ()
50 LWP 5672 "rpc worker-5672" 0x00007fc5ba56ead3 in ?? ()
51 LWP 5673 "rpc worker-5673" 0x00007fc5ba56ead3 in ?? ()
52 LWP 5674 "rpc worker-5674" 0x00007fc5ba56ead3 in ?? ()
53 LWP 5675 "rpc worker-5675" 0x00007fc5ba56ead3 in ?? ()
54 LWP 5676 "rpc worker-5676" 0x00007fc5ba56ead3 in ?? ()
55 LWP 5677 "rpc worker-5677" 0x00007fc5ba56ead3 in ?? ()
56 LWP 5678 "rpc worker-5678" 0x00007fc5ba56ead3 in ?? ()
57 LWP 5679 "rpc worker-5679" 0x00007fc5ba56ead3 in ?? ()
58 LWP 5680 "rpc worker-5680" 0x00007fc5ba56ead3 in ?? ()
59 LWP 5681 "rpc worker-5681" 0x00007fc5ba56ead3 in ?? ()
60 LWP 5682 "rpc worker-5682" 0x00007fc5ba56ead3 in ?? ()
61 LWP 5685 "rpc worker-5685" 0x00007fc5ba56ead3 in ?? ()
62 LWP 5686 "rpc worker-5686" 0x00007fc5ba56ead3 in ?? ()
63 LWP 5687 "rpc worker-5687" 0x00007fc5ba56ead3 in ?? ()
64 LWP 5688 "rpc worker-5688" 0x00007fc5ba56ead3 in ?? ()
65 LWP 5689 "rpc worker-5689" 0x00007fc5ba56ead3 in ?? ()
66 LWP 5690 "rpc worker-5690" 0x00007fc5ba56ead3 in ?? ()
67 LWP 5691 "rpc worker-5691" 0x00007fc5ba56ead3 in ?? ()
68 LWP 5692 "rpc worker-5692" 0x00007fc5ba56ead3 in ?? ()
69 LWP 5693 "rpc worker-5693" 0x00007fc5ba56ead3 in ?? ()
70 LWP 5694 "rpc worker-5694" 0x00007fc5ba56ead3 in ?? ()
71 LWP 5695 "rpc worker-5695" 0x00007fc5ba56ead3 in ?? ()
72 LWP 5696 "rpc worker-5696" 0x00007fc5ba56ead3 in ?? ()
73 LWP 5697 "rpc worker-5697" 0x00007fc5ba56ead3 in ?? ()
74 LWP 5698 "rpc worker-5698" 0x00007fc5ba56ead3 in ?? ()
75 LWP 5699 "rpc worker-5699" 0x00007fc5ba56ead3 in ?? ()
76 LWP 5700 "rpc worker-5700" 0x00007fc5ba56ead3 in ?? ()
77 LWP 5701 "rpc worker-5701" 0x00007fc5ba56ead3 in ?? ()
78 LWP 5702 "rpc worker-5702" 0x00007fc5ba56ead3 in ?? ()
79 LWP 5703 "rpc worker-5703" 0x00007fc5ba56ead3 in ?? ()
80 LWP 5704 "rpc worker-5704" 0x00007fc5ba56ead3 in ?? ()
81 LWP 5705 "rpc worker-5705" 0x00007fc5ba56ead3 in ?? ()
82 LWP 5706 "rpc worker-5706" 0x00007fc5ba56ead3 in ?? ()
83 LWP 5707 "rpc worker-5707" 0x00007fc5ba56ead3 in ?? ()
84 LWP 5708 "rpc worker-5708" 0x00007fc5ba56ead3 in ?? ()
85 LWP 5709 "rpc worker-5709" 0x00007fc5ba56ead3 in ?? ()
86 LWP 5710 "rpc worker-5710" 0x00007fc5ba56ead3 in ?? ()
87 LWP 5711 "rpc worker-5711" 0x00007fc5ba56ead3 in ?? ()
88 LWP 5712 "rpc worker-5712" 0x00007fc5ba56ead3 in ?? ()
89 LWP 5713 "rpc worker-5713" 0x00007fc5ba56ead3 in ?? ()
90 LWP 5714 "rpc worker-5714" 0x00007fc5ba56ead3 in ?? ()
91 LWP 5715 "rpc worker-5715" 0x00007fc5ba56ead3 in ?? ()
92 LWP 5716 "rpc worker-5716" 0x00007fc5ba56ead3 in ?? ()
93 LWP 5717 "rpc worker-5717" 0x00007fc5ba56ead3 in ?? ()
94 LWP 5718 "rpc worker-5718" 0x00007fc5ba56ead3 in ?? ()
95 LWP 5719 "rpc worker-5719" 0x00007fc5ba56ead3 in ?? ()
96 LWP 5720 "rpc worker-5720" 0x00007fc5ba56ead3 in ?? ()
97 LWP 5721 "rpc worker-5721" 0x00007fc5ba56ead3 in ?? ()
98 LWP 5722 "rpc worker-5722" 0x00007fc5ba56ead3 in ?? ()
99 LWP 5723 "rpc worker-5723" 0x00007fc5ba56ead3 in ?? ()
100 LWP 5725 "rpc worker-5725" 0x00007fc5ba56ead3 in ?? ()
101 LWP 5726 "rpc worker-5726" 0x00007fc5ba56ead3 in ?? ()
102 LWP 5727 "rpc worker-5727" 0x00007fc5ba56ead3 in ?? ()
103 LWP 5728 "rpc worker-5728" 0x00007fc5ba56ead3 in ?? ()
104 LWP 5729 "rpc worker-5729" 0x00007fc5ba56ead3 in ?? ()
105 LWP 5730 "rpc worker-5730" 0x00007fc5ba56ead3 in ?? ()
106 LWP 5731 "rpc worker-5731" 0x00007fc5ba56ead3 in ?? ()
107 LWP 5732 "rpc worker-5732" 0x00007fc5ba56ead3 in ?? ()
108 LWP 5733 "rpc worker-5733" 0x00007fc5ba56ead3 in ?? ()
109 LWP 5734 "rpc worker-5734" 0x00007fc5ba56ead3 in ?? ()
110 LWP 5735 "rpc worker-5735" 0x00007fc5ba56ead3 in ?? ()
111 LWP 5736 "rpc worker-5736" 0x00007fc5ba56ead3 in ?? ()
112 LWP 5738 "rpc worker-5738" 0x00007fc5ba56ead3 in ?? ()
113 LWP 5739 "rpc worker-5739" 0x00007fc5ba56ead3 in ?? ()
114 LWP 5741 "rpc worker-5741" 0x00007fc5ba56ead3 in ?? ()
115 LWP 5742 "rpc worker-5742" 0x00007fc5ba56ead3 in ?? ()
116 LWP 5743 "rpc worker-5743" 0x00007fc5ba56ead3 in ?? ()
117 LWP 5744 "diag-logger-574" 0x00007fc5ba56efb9 in ?? ()
118 LWP 5745 "result-tracker-" 0x00007fc5ba56efb9 in ?? ()
119 LWP 5746 "excess-log-dele" 0x00007fc5ba56efb9 in ?? ()
120 LWP 5747 "acceptor-5747" 0x00007fc5b32a30c7 in ?? ()
121 LWP 5748 "heartbeat-5748" 0x00007fc5ba56efb9 in ?? ()
122 LWP 5751 "maintenance_sch" 0x00007fc5ba56efb9 in ?? ()
Thread 122 (LWP 5751):
#0 0x00007fc5ba56efb9 in ?? ()
#1 0x00007fc56eac26e0 in ?? ()
#2 0x0000000000000113 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007fc56eac2730 in ?? ()
#6 0x0000000000000226 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 5748):
#0 0x00007fc5ba56efb9 in ?? ()
#1 0x8ea7c1ad18c34923 in ?? ()
#2 0x0000000000000045 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019400 in ?? ()
#5 0x00007fc56faf95f0 in ?? ()
#6 0x000000000000008a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 5747):
#0 0x00007fc5b32a30c7 in ?? ()
#1 0x00006140000212a8 in ?? ()
#2 0x00007fc570310d90 in ?? ()
#3 0x00007fc570310dc0 in ?? ()
#4 0x00007fc570310ec0 in ?? ()
#5 0x00007fc570310db0 in ?? ()
#6 0x00007fc570310e20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 5746):
#0 0x00007fc5ba56efb9 in ?? ()
#1 0x00007fc570b29f40 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffcd63b6fd0 in ?? ()
#5 0x00007fc570b29f30 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 5745):
#0 0x00007fc5ba56efb9 in ?? ()
#1 0x00007fc571342100 in ?? ()
#2 0x0000000000000045 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007fc5713420f0 in ?? ()
#6 0x000000000000008a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 5744):
#0 0x00007fc5ba56efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 5743):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x000000000000000b in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0004e688c in ?? ()
#4 0x00007fc572382e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc572382eb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0004e6840 in ?? ()
#9 0x00007fc5ba56e770 in ?? ()
#10 0x00007fc572382eb0 in ?? ()
#11 0x00000ff8ae4705d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 5742):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000015 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0004a67fc in ?? ()
#4 0x00007fc572b9ae90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc572b9aeb0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0004a67b0 in ?? ()
#9 0x00007fc5ba56e770 in ?? ()
#10 0x00007fc572b9aeb0 in ?? ()
#11 0x00000ff8ae5735d8 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 5741):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 5739):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 5738):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 5736):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 5735):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 5734):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 5733):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 5732):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 5731):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 5730):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 5729):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 5728):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 5727):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 5726):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 5725):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 5723):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 5722):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 5721):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 5720):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 5719):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 5718):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 5717):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 5716):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 5715):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 5714):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 5713):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 5712):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 5711):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 5710):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 5709):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 5708):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 5707):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 5706):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 5705):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 5704):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 5703):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 5702):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 5701):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 5700):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000258 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00029d068 in ?? ()
#4 0x00007fc5868bde90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc5868bdeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 5699):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000296868 in ?? ()
#4 0x00007fc5870d5e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc5870d5eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 5698):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 5697):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 5696):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 5695):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 5694):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 5693):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 5692):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 5691):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 5690):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 5689):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 5688):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 5687):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 5686):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 5685):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 5682):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 5681):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 5680):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 5679):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 5678):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00016ffe8 in ?? ()
#4 0x00007fc5902d7e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc5902d7eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 5677):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 5676):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 5675):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 5674):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 5673):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 5672):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 5671):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 5670):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 5669):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 5668):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 5667):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 5666):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 5665):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 5664):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 5663):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 5662):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 5661):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 5660):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 5659):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 5658):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x000000000000000e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000affd8 in ?? ()
#4 0x00007fc599ca0e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fc599ca0eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 5657):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 5656):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 5655):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 5654):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 5652):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 5651):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 5650):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 5649):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 5648):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 5647):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 5646):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 5645):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 5644):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 5643):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 5642):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 5641):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 5640):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 5639):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 5638):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 5637):
#0 0x00007fc5ba56efb9 in ?? ()
#1 0x00007fc5a47dfcc0 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001a3a0 in ?? ()
#5 0x00007fc5a47dfcb0 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 5636):
#0 0x00007fc5ba56efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 5634):
#0 0x00007fc5ba56efb9 in ?? ()
#1 0x00007fc5a6077240 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007fc5a6077230 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 5633):
#0 0x00007fc5ba56ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 5632):
#0 0x00007fc5b32a1a47 in ?? ()
#1 0x00007fc5a70b2320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007fc5a70b2310 in ?? ()
#4 0x00007fc5a70b2520 in ?? ()
#5 0x00007fc5a70b2360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fc5a70b23e0 in ?? ()
#8 0x00007fc5b591425d in ?? ()
#9 0x3fb98bbf14d3f000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x0000000072800000 in ?? ()
#14 0x00007fc500000000 in ?? ()
#15 0x41da16179c5dd6b8 in ?? ()
#16 0x00000ff934e0e47c in ?? ()
#17 0x00007fc5a70b23c0 in ?? ()
#18 0x00007fc5b5918ba3 in ?? ()
#19 0x00007fc5a70b2390 in ?? ()
#20 0x3fb98bbf14d3f000 in ?? ()
#21 0x00000000a70b23e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb98bbf14d3f000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 5631):
#0 0x00007fc5b32a1a47 in ?? ()
#1 0x00007fc5a78b3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007fc5a78b3310 in ?? ()
#4 0x00007fc5a78b3520 in ?? ()
#5 0x00007fc5a78b3360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007fc5a78b33e0 in ?? ()
#8 0x00007fc5b591425d in ?? ()
#9 0x3fb96b50fe538000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x0000000076000000 in ?? ()
#14 0x00007fc500000000 in ?? ()
#15 0x41da16179c5dd6b8 in ?? ()
#16 0x00000ff934f0e67c in ?? ()
#17 0x00007fc5a78b33c0 in ?? ()
#18 0x00007fc5b5918ba3 in ?? ()
#19 0x00007fc5a78b3390 in ?? ()
#20 0x3fb96b50fe538000 in ?? ()
#21 0x00000000a78b33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb96b50fe538000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 5630):
#0 0x00007fc5b32a1a47 in ?? ()
#1 0x00007fc5a80b4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007fc5a80b4310 in ?? ()
#4 0x00007fc5a80b4520 in ?? ()
#5 0x00007fc5a80b4360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007fc5a80b43e0 in ?? ()
#8 0x00007fc5b591425d in ?? ()
#9 0x3fb95935c8c2f000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x000000007c000000 in ?? ()
#14 0x00007fc500000000 in ?? ()
#15 0x41da16179c5dd6b7 in ?? ()
#16 0x00000ff93500e87c in ?? ()
#17 0x00007fc5a80b43c0 in ?? ()
#18 0x00007fc5b5918ba3 in ?? ()
#19 0x00007fc5a80b4390 in ?? ()
#20 0x3fb95935c8c2f000 in ?? ()
#21 0x00000000a80b43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb95935c8c2f000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 5629):
#0 0x00007fc5b32a1a47 in ?? ()
#1 0x00007fc5aa0ad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007fc5aa0ad310 in ?? ()
#4 0x00007fc5aa0ad520 in ?? ()
#5 0x00007fc5aa0ad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007fc5aa0ad3e0 in ?? ()
#8 0x00007fc5b591425d in ?? ()
#9 0x3fb7c2cceec27000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x00000000e7200000 in ?? ()
#14 0x00007fc500000000 in ?? ()
#15 0x41da16179c5dd6b4 in ?? ()
#16 0x00000ff93540da7c in ?? ()
#17 0x00007fc5aa0ad3c0 in ?? ()
#18 0x00007fc5b5918ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 5626):
#0 0x00007fc5b3294cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fc5abab77c8 in ?? ()
#3 0x000060200001dc90 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 5625):
#0 0x00007fc5ba56efb9 in ?? ()
#1 0x00007fc5ab2b60c0 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007fc5ab2b60b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 5624):
#0 0x00007fc5ba5729e2 in ?? ()
#1 0x00007fc5aaab3ba0 in ?? ()
#2 0x00007fc5aaab3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fc5aaab3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007fc5aaab3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007fc5aaab3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007fc5bfe85edc in ?? ()
#12 0x00007fc5bfe758d9 in ?? ()
#13 0x00007fc5bfe79ec6 in ?? ()
#14 0x00007fc5bfe7e900 in ?? ()
#15 0x00007fc5bfe7e729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007fc5b714a4fe in ?? ()
#18 0x00007fc5ba5686db in ?? ()
#19 0x00000ff93554e684 in ?? ()
#20 0x00007fc5aaab3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 5618):
#0 0x00007fc5ba56efb9 in ?? ()
#1 0x00007fc5acab8c80 in ?? ()
#2 0x0000000000000163 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007fc5acab8c70 in ?? ()
#6 0x00000000000002c6 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 5617):
#0 0x00007fc5ba56efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 5616):
#0 0x00007fc5ba56efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 5615):
#0 0x00007fc5ba56efb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000004 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007fc5ae2bc450 in ?? ()
#6 0x0000000000000008 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 5613):
#0 0x00007fc5ba572d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250622 19:58:15.335323 2016 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 8 with UUID 7520035c9f604041a00535b523c45c1b and pid 5465
************************ BEGIN STACKS **************************
[New LWP 5466]
[New LWP 5467]
[New LWP 5468]
[New LWP 5469]
[New LWP 5475]
[New LWP 5477]
[New LWP 5478]
[New LWP 5482]
[New LWP 5483]
[New LWP 5484]
[New LWP 5485]
[New LWP 5487]
[New LWP 5488]
[New LWP 5489]
[New LWP 5490]
[New LWP 5491]
[New LWP 5492]
[New LWP 5493]
[New LWP 5494]
[New LWP 5495]
[New LWP 5496]
[New LWP 5497]
[New LWP 5498]
[New LWP 5499]
[New LWP 5500]
[New LWP 5501]
[New LWP 5502]
[New LWP 5503]
[New LWP 5504]
[New LWP 5505]
[New LWP 5506]
[New LWP 5507]
[New LWP 5508]
[New LWP 5509]
[New LWP 5510]
[New LWP 5511]
[New LWP 5512]
[New LWP 5513]
[New LWP 5514]
[New LWP 5515]
[New LWP 5517]
[New LWP 5518]
[New LWP 5519]
[New LWP 5521]
[New LWP 5522]
[New LWP 5523]
[New LWP 5524]
[New LWP 5525]
[New LWP 5526]
[New LWP 5527]
[New LWP 5529]
[New LWP 5530]
[New LWP 5531]
[New LWP 5532]
[New LWP 5533]
[New LWP 5534]
[New LWP 5535]
[New LWP 5536]
[New LWP 5537]
[New LWP 5538]
[New LWP 5539]
[New LWP 5540]
[New LWP 5541]
[New LWP 5542]
[New LWP 5543]
[New LWP 5544]
[New LWP 5545]
[New LWP 5546]
[New LWP 5547]
[New LWP 5548]
[New LWP 5549]
[New LWP 5550]
[New LWP 5551]
[New LWP 5552]
[New LWP 5553]
[New LWP 5554]
[New LWP 5555]
[New LWP 5556]
[New LWP 5557]
[New LWP 5558]
[New LWP 5560]
[New LWP 5561]
[New LWP 5563]
[New LWP 5564]
[New LWP 5565]
[New LWP 5566]
[New LWP 5567]
[New LWP 5568]
[New LWP 5569]
[New LWP 5570]
[New LWP 5571]
[New LWP 5572]
[New LWP 5573]
[New LWP 5574]
[New LWP 5575]
[New LWP 5576]
[New LWP 5577]
[New LWP 5578]
[New LWP 5579]
[New LWP 5580]
[New LWP 5581]
[New LWP 5582]
[New LWP 5583]
[New LWP 5584]
[New LWP 5585]
[New LWP 5586]
[New LWP 5588]
[New LWP 5589]
[New LWP 5590]
[New LWP 5591]
[New LWP 5592]
[New LWP 5593]
[New LWP 5594]
[New LWP 5595]
[New LWP 5596]
[New LWP 5597]
[New LWP 5598]
[New LWP 5599]
[New LWP 5600]
[New LWP 5601]
[New LWP 5604]
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e71
Cannot access memory at address 0x6563656950676e69
0x00007f25bcec3d50 in ?? ()
Id Target Id Frame
* 1 LWP 5465 "kudu" 0x00007f25bcec3d50 in ?? ()
2 LWP 5466 "kudu" 0x00007f25bcebffb9 in ?? ()
3 LWP 5467 "kudu" 0x00007f25bcebffb9 in ?? ()
4 LWP 5468 "kudu" 0x00007f25bcebffb9 in ?? ()
5 LWP 5469 "kernel-watcher-" 0x00007f25bcebffb9 in ?? ()
6 LWP 5475 "ntp client-5475" 0x00007f25bcec39e2 in ?? ()
7 LWP 5477 "file cache-evic" 0x00007f25bcebffb9 in ?? ()
8 LWP 5478 "sq_acceptor" 0x00007f25b5be5cb9 in ?? ()
9 LWP 5482 "rpc reactor-548" 0x00007f25b5bf2a47 in ?? ()
10 LWP 5483 "rpc reactor-548" 0x00007f25b5bf2a47 in ?? ()
11 LWP 5484 "rpc reactor-548" 0x00007f25b5bf2a47 in ?? ()
12 LWP 5485 "rpc reactor-548" 0x00007f25b5bf2a47 in ?? ()
13 LWP 5487 "MaintenanceMgr " 0x00007f25bcebfad3 in ?? ()
14 LWP 5488 "txn-status-mana" 0x00007f25bcebffb9 in ?? ()
15 LWP 5489 "collect_and_rem" 0x00007f25bcebffb9 in ?? ()
16 LWP 5490 "tc-session-exp-" 0x00007f25bcebffb9 in ?? ()
17 LWP 5491 "rpc worker-5491" 0x00007f25bcebfad3 in ?? ()
18 LWP 5492 "rpc worker-5492" 0x00007f25bcebfad3 in ?? ()
19 LWP 5493 "rpc worker-5493" 0x00007f25bcebfad3 in ?? ()
20 LWP 5494 "rpc worker-5494" 0x00007f25bcebfad3 in ?? ()
21 LWP 5495 "rpc worker-5495" 0x00007f25bcebfad3 in ?? ()
22 LWP 5496 "rpc worker-5496" 0x00007f25bcebfad3 in ?? ()
23 LWP 5497 "rpc worker-5497" 0x00007f25bcebfad3 in ?? ()
24 LWP 5498 "rpc worker-5498" 0x00007f25bcebfad3 in ?? ()
25 LWP 5499 "rpc worker-5499" 0x00007f25bcebfad3 in ?? ()
26 LWP 5500 "rpc worker-5500" 0x00007f25bcebfad3 in ?? ()
27 LWP 5501 "rpc worker-5501" 0x00007f25bcebfad3 in ?? ()
28 LWP 5502 "rpc worker-5502" 0x00007f25bcebfad3 in ?? ()
29 LWP 5503 "rpc worker-5503" 0x00007f25bcebfad3 in ?? ()
30 LWP 5504 "rpc worker-5504" 0x00007f25bcebfad3 in ?? ()
31 LWP 5505 "rpc worker-5505" 0x00007f25bcebfad3 in ?? ()
32 LWP 5506 "rpc worker-5506" 0x00007f25bcebfad3 in ?? ()
33 LWP 5507 "rpc worker-5507" 0x00007f25bcebfad3 in ?? ()
34 LWP 5508 "rpc worker-5508" 0x00007f25bcebfad3 in ?? ()
35 LWP 5509 "rpc worker-5509" 0x00007f25bcebfad3 in ?? ()
36 LWP 5510 "rpc worker-5510" 0x00007f25bcebfad3 in ?? ()
37 LWP 5511 "rpc worker-5511" 0x00007f25bcebfad3 in ?? ()
38 LWP 5512 "rpc worker-5512" 0x00007f25bcebfad3 in ?? ()
39 LWP 5513 "rpc worker-5513" 0x00007f25bcebfad3 in ?? ()
40 LWP 5514 "rpc worker-5514" 0x00007f25bcebfad3 in ?? ()
41 LWP 5515 "rpc worker-5515" 0x00007f25bcebfad3 in ?? ()
42 LWP 5517 "rpc worker-5517" 0x00007f25bcebfad3 in ?? ()
43 LWP 5518 "rpc worker-5518" 0x00007f25bcebfad3 in ?? ()
44 LWP 5519 "rpc worker-5519" 0x00007f25bcebfad3 in ?? ()
45 LWP 5521 "rpc worker-5521" 0x00007f25bcebfad3 in ?? ()
46 LWP 5522 "rpc worker-5522" 0x00007f25bcebfad3 in ?? ()
47 LWP 5523 "rpc worker-5523" 0x00007f25bcebfad3 in ?? ()
48 LWP 5524 "rpc worker-5524" 0x00007f25bcebfad3 in ?? ()
49 LWP 5525 "rpc worker-5525" 0x00007f25bcebfad3 in ?? ()
50 LWP 5526 "rpc worker-5526" 0x00007f25bcebfad3 in ?? ()
51 LWP 5527 "rpc worker-5527" 0x00007f25bcebfad3 in ?? ()
52 LWP 5529 "rpc worker-5529" 0x00007f25bcebfad3 in ?? ()
53 LWP 5530 "rpc worker-5530" 0x00007f25bcebfad3 in ?? ()
54 LWP 5531 "rpc worker-5531" 0x00007f25bcebfad3 in ?? ()
55 LWP 5532 "rpc worker-5532" 0x00007f25bcebfad3 in ?? ()
56 LWP 5533 "rpc worker-5533" 0x00007f25bcebfad3 in ?? ()
57 LWP 5534 "rpc worker-5534" 0x00007f25bcebfad3 in ?? ()
58 LWP 5535 "rpc worker-5535" 0x00007f25bcebfad3 in ?? ()
59 LWP 5536 "rpc worker-5536" 0x00007f25bcebfad3 in ?? ()
60 LWP 5537 "rpc worker-5537" 0x00007f25bcebfad3 in ?? ()
61 LWP 5538 "rpc worker-5538" 0x00007f25bcebfad3 in ?? ()
62 LWP 5539 "rpc worker-5539" 0x00007f25bcebfad3 in ?? ()
63 LWP 5540 "rpc worker-5540" 0x00007f25bcebfad3 in ?? ()
64 LWP 5541 "rpc worker-5541" 0x00007f25bcebfad3 in ?? ()
65 LWP 5542 "rpc worker-5542" 0x00007f25bcebfad3 in ?? ()
66 LWP 5543 "rpc worker-5543" 0x00007f25bcebfad3 in ?? ()
67 LWP 5544 "rpc worker-5544" 0x00007f25bcebfad3 in ?? ()
68 LWP 5545 "rpc worker-5545" 0x00007f25bcebfad3 in ?? ()
69 LWP 5546 "rpc worker-5546" 0x00007f25bcebfad3 in ?? ()
70 LWP 5547 "rpc worker-5547" 0x00007f25bcebfad3 in ?? ()
71 LWP 5548 "rpc worker-5548" 0x00007f25bcebfad3 in ?? ()
72 LWP 5549 "rpc worker-5549" 0x00007f25bcebfad3 in ?? ()
73 LWP 5550 "rpc worker-5550" 0x00007f25bcebfad3 in ?? ()
74 LWP 5551 "rpc worker-5551" 0x00007f25bcebfad3 in ?? ()
75 LWP 5552 "rpc worker-5552" 0x00007f25bcebfad3 in ?? ()
76 LWP 5553 "rpc worker-5553" 0x00007f25bcebfad3 in ?? ()
77 LWP 5554 "rpc worker-5554" 0x00007f25bcebfad3 in ?? ()
78 LWP 5555 "rpc worker-5555" 0x00007f25bcebfad3 in ?? ()
79 LWP 5556 "rpc worker-5556" 0x00007f25bcebfad3 in ?? ()
80 LWP 5557 "rpc worker-5557" 0x00007f25bcebfad3 in ?? ()
81 LWP 5558 "rpc worker-5558" 0x00007f25bcebfad3 in ?? ()
82 LWP 5560 "rpc worker-5560" 0x00007f25bcebfad3 in ?? ()
83 LWP 5561 "rpc worker-5561" 0x00007f25bcebfad3 in ?? ()
84 LWP 5563 "rpc worker-5563" 0x00007f25bcebfad3 in ?? ()
85 LWP 5564 "rpc worker-5564" 0x00007f25bcebfad3 in ?? ()
86 LWP 5565 "rpc worker-5565" 0x00007f25bcebfad3 in ?? ()
87 LWP 5566 "rpc worker-5566" 0x00007f25bcebfad3 in ?? ()
88 LWP 5567 "rpc worker-5567" 0x00007f25bcebfad3 in ?? ()
89 LWP 5568 "rpc worker-5568" 0x00007f25bcebfad3 in ?? ()
90 LWP 5569 "rpc worker-5569" 0x00007f25bcebfad3 in ?? ()
91 LWP 5570 "rpc worker-5570" 0x00007f25bcebfad3 in ?? ()
92 LWP 5571 "rpc worker-5571" 0x00007f25bcebfad3 in ?? ()
93 LWP 5572 "rpc worker-5572" 0x00007f25bcebfad3 in ?? ()
94 LWP 5573 "rpc worker-5573" 0x00007f25bcebfad3 in ?? ()
95 LWP 5574 "rpc worker-5574" 0x00007f25bcebfad3 in ?? ()
96 LWP 5575 "rpc worker-5575" 0x00007f25bcebfad3 in ?? ()
97 LWP 5576 "rpc worker-5576" 0x00007f25bcebfad3 in ?? ()
98 LWP 5577 "rpc worker-5577" 0x00007f25bcebfad3 in ?? ()
99 LWP 5578 "rpc worker-5578" 0x00007f25bcebfad3 in ?? ()
100 LWP 5579 "rpc worker-5579" 0x00007f25bcebfad3 in ?? ()
101 LWP 5580 "rpc worker-5580" 0x00007f25bcebfad3 in ?? ()
102 LWP 5581 "rpc worker-5581" 0x00007f25bcebfad3 in ?? ()
103 LWP 5582 "rpc worker-5582" 0x00007f25bcebfad3 in ?? ()
104 LWP 5583 "rpc worker-5583" 0x00007f25bcebfad3 in ?? ()
105 LWP 5584 "rpc worker-5584" 0x00007f25bcebfad3 in ?? ()
106 LWP 5585 "rpc worker-5585" 0x00007f25bcebfad3 in ?? ()
107 LWP 5586 "rpc worker-5586" 0x00007f25bcebfad3 in ?? ()
108 LWP 5588 "rpc worker-5588" 0x00007f25bcebfad3 in ?? ()
109 LWP 5589 "rpc worker-5589" 0x00007f25bcebfad3 in ?? ()
110 LWP 5590 "rpc worker-5590" 0x00007f25bcebfad3 in ?? ()
111 LWP 5591 "rpc worker-5591" 0x00007f25bcebfad3 in ?? ()
112 LWP 5592 "rpc worker-5592" 0x00007f25bcebfad3 in ?? ()
113 LWP 5593 "rpc worker-5593" 0x00007f25bcebfad3 in ?? ()
114 LWP 5594 "rpc worker-5594" 0x00007f25bcebfad3 in ?? ()
115 LWP 5595 "rpc worker-5595" 0x00007f25bcebfad3 in ?? ()
116 LWP 5596 "rpc worker-5596" 0x00007f25bcebfad3 in ?? ()
117 LWP 5597 "diag-logger-559" 0x00007f25bcebffb9 in ?? ()
118 LWP 5598 "result-tracker-" 0x00007f25bcebffb9 in ?? ()
119 LWP 5599 "excess-log-dele" 0x00007f25bcebffb9 in ?? ()
120 LWP 5600 "acceptor-5600" 0x00007f25b5bf40c7 in ?? ()
121 LWP 5601 "heartbeat-5601" 0x00007f25bcebffb9 in ?? ()
122 LWP 5604 "maintenance_sch" 0x00007f25bcebffb9 in ?? ()
Thread 122 (LWP 5604):
#0 0x00007f25bcebffb9 in ?? ()
#1 0x00007f25727676e0 in ?? ()
#2 0x0000000000000126 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006160000157a0 in ?? ()
#5 0x00007f2572767730 in ?? ()
#6 0x000000000000024c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 5601):
#0 0x00007f25bcebffb9 in ?? ()
#1 0x8ea7c1ad18c34923 in ?? ()
#2 0x0000000000000049 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019400 in ?? ()
#5 0x00007f2572f7f5f0 in ?? ()
#6 0x0000000000000092 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 5600):
#0 0x00007f25b5bf40c7 in ?? ()
#1 0x00006140000214a8 in ?? ()
#2 0x00007f25737a6d90 in ?? ()
#3 0x00007f25737a6dc0 in ?? ()
#4 0x00007f25737a6ec0 in ?? ()
#5 0x00007f25737a6db0 in ?? ()
#6 0x00007f25737a6e20 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008b0dab in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 5599):
#0 0x00007f25bcebffb9 in ?? ()
#1 0x00007f2573fc0f40 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fffd029a670 in ?? ()
#5 0x00007f2573fc0f30 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 5598):
#0 0x00007f25bcebffb9 in ?? ()
#1 0x00007f25747d9100 in ?? ()
#2 0x0000000000000049 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000898f0 in ?? ()
#5 0x00007f25747d90f0 in ?? ()
#6 0x0000000000000092 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 5597):
#0 0x00007f25bcebffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 5596):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000014 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007f2575808e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2575808eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 5595):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x000000000000000c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002a6858 in ?? ()
#4 0x00007f2576020e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2576020eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 5594):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 5593):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 5592):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 5591):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 5590):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 5589):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 5588):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 5586):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 5585):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 5584):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 5583):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 5582):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 5581):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 5580):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 5579):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 5578):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 5577):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 5576):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 5575):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 5574):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 5573):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 5572):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 5571):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 5570):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 5569):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 5568):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 5567):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 5566):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 5565):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 5564):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 5563):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 5561):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 5560):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 5558):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 5557):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 5556):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 5555):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 5554):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 5553):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a6888 in ?? ()
#4 0x00007f2589bd9e90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2589bd9eb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 5552):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 5551):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 5550):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 5549):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 5548):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 5547):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 5546):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 5545):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 5544):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 5543):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 5542):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 5541):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 5540):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 5539):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 5538):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 5537):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 5536):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 5535):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 5534):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 5533):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000120038 in ?? ()
#4 0x00007f2593dbae90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f2593dbaeb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 5532):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 5531):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 5530):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 5529):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 5527):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 5526):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 5525):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 5524):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 5523):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 5522):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 5521):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 5519):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 5518):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 5517):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 5515):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 5514):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 5513):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 5512):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 5511):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 5510):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00009ffe8 in ?? ()
#4 0x00007f259df9be90 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f259df9beb0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 5509):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 5508):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 5507):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 5506):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 5505):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 5504):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 5503):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 5502):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 5501):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 5500):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 5499):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 5498):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 5497):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 5496):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 5495):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 5494):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 5493):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 5492):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 5491):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 5490):
#0 0x00007f25bcebffb9 in ?? ()
#1 0x00007f25a797fcc0 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019ca0 in ?? ()
#5 0x00007f25a797fcb0 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 5489):
#0 0x00007f25bcebffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 5488):
#0 0x00007f25bcebffb9 in ?? ()
#1 0x00007f25a89af240 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000d5a8 in ?? ()
#5 0x00007f25a89af230 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 5487):
#0 0x00007f25bcebfad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 5485):
#0 0x00007f25b5bf2a47 in ?? ()
#1 0x00007f25a99b2320 in ?? ()
#2 0x000061a00000cc80 in ?? ()
#3 0x00007f25a99b2310 in ?? ()
#4 0x00007f25a99b2520 in ?? ()
#5 0x00007f25a99b2360 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f25a99b23e0 in ?? ()
#8 0x00007f25b826525d in ?? ()
#9 0x3fb98515a4d0f000 in ?? ()
#10 0x000061a00000cc80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000cc80 in ?? ()
#13 0x00000000ec000000 in ?? ()
#14 0x00007f2500000000 in ?? ()
#15 0x41da16179c5dd6b6 in ?? ()
#16 0x00000fe53532e47c in ?? ()
#17 0x00007f25a99b23c0 in ?? ()
#18 0x00007f25b8269ba3 in ?? ()
#19 0x00007f25a99b2390 in ?? ()
#20 0x3fb98515a4d0f000 in ?? ()
#21 0x00000000a99b23e0 in ?? ()
#22 0x000061a00000cc80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb98515a4d0f000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 5484):
#0 0x00007f25b5bf2a47 in ?? ()
#1 0x00007f25aa1b3320 in ?? ()
#2 0x000061a00000c680 in ?? ()
#3 0x00007f25aa1b3310 in ?? ()
#4 0x00007f25aa1b3520 in ?? ()
#5 0x00007f25aa1b3360 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f25aa1b33e0 in ?? ()
#8 0x00007f25b826525d in ?? ()
#9 0x3fb9813c9eb24000 in ?? ()
#10 0x000061a00000c680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c680 in ?? ()
#13 0x00000000c0000000 in ?? ()
#14 0x00007f2500000000 in ?? ()
#15 0x41da16179c5dd6b8 in ?? ()
#16 0x00000fe53542e67c in ?? ()
#17 0x00007f25aa1b33c0 in ?? ()
#18 0x00007f25b8269ba3 in ?? ()
#19 0x00007f25aa1b3390 in ?? ()
#20 0x3fb9813c9eb24000 in ?? ()
#21 0x00000000aa1b33e0 in ?? ()
#22 0x000061a00000c680 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb9813c9eb24000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 5483):
#0 0x00007f25b5bf2a47 in ?? ()
#1 0x00007f25aa9b4320 in ?? ()
#2 0x000061a00000c080 in ?? ()
#3 0x00007f25aa9b4310 in ?? ()
#4 0x00007f25aa9b4520 in ?? ()
#5 0x00007f25aa9b4360 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f25aa9b43e0 in ?? ()
#8 0x00007f25b826525d in ?? ()
#9 0x3fb98055bb5b4000 in ?? ()
#10 0x000061a00000c080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000c080 in ?? ()
#13 0x0000000023000000 in ?? ()
#14 0x00007f2500000000 in ?? ()
#15 0x41da16179c5dd6b5 in ?? ()
#16 0x00000fe53552e87c in ?? ()
#17 0x00007f25aa9b43c0 in ?? ()
#18 0x00007f25b8269ba3 in ?? ()
#19 0x00007f25aa9b4390 in ?? ()
#20 0x3fb98055bb5b4000 in ?? ()
#21 0x00000000aa9b43e0 in ?? ()
#22 0x000061a00000c080 in ?? ()
#23 0x0000614000021298 in ?? ()
#24 0x3fb98055bb5b4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 5482):
#0 0x00007f25b5bf2a47 in ?? ()
#1 0x00007f25ac9ad320 in ?? ()
#2 0x000061a00000ba80 in ?? ()
#3 0x00007f25ac9ad310 in ?? ()
#4 0x00007f25ac9ad520 in ?? ()
#5 0x00007f25ac9ad360 in ?? ()
#6 0x0000614000021098 in ?? ()
#7 0x00007f25ac9ad3e0 in ?? ()
#8 0x00007f25b826525d in ?? ()
#9 0x3fb94bfb40bb6000 in ?? ()
#10 0x000061a00000ba80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ba80 in ?? ()
#13 0x000000007c000000 in ?? ()
#14 0x00007f2500000000 in ?? ()
#15 0x41da16179c5dd6b6 in ?? ()
#16 0x00000fe53592da7c in ?? ()
#17 0x00007f25ac9ad3c0 in ?? ()
#18 0x00007f25b8269ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 5478):
#0 0x00007f25b5be5cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f25ae3b77c8 in ?? ()
#3 0x000060200001ecd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a89f1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 5477):
#0 0x00007f25bcebffb9 in ?? ()
#1 0x00007f25adbb60c0 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001d840 in ?? ()
#5 0x00007f25adbb60b0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 5475):
#0 0x00007f25bcec39e2 in ?? ()
#1 0x00007f25ad3b3ba0 in ?? ()
#2 0x00007f25ad3b3c40 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f25ad3b3bf0 in ?? ()
#6 0x000000000177ece0 in ?? ()
#7 0x00007f25ad3b3c50 in ?? ()
#8 0x000061100008a890 in ?? ()
#9 0x00007f25ad3b3c40 in ?? ()
#10 0x00000000008a2ee7 in __sanitizer::theDepot ()
#11 0x00007f25c27d6edc in ?? ()
#12 0x00007f25c27c68d9 in ?? ()
#13 0x00007f25c27caec6 in ?? ()
#14 0x00007f25c27cf900 in ?? ()
#15 0x00007f25c27cf729 in ?? ()
#16 0x0000000000a7573d in __sanitizer::theDepot ()
#17 0x00007f25b9a9b4fe in ?? ()
#18 0x00007f25bceb96db in ?? ()
#19 0x00000fe535a6e684 in ?? ()
#20 0x00007f25ad3b3440 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 5469):
#0 0x00007f25bcebffb9 in ?? ()
#1 0x00007f25af3b8c80 in ?? ()
#2 0x0000000000000179 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001eea8 in ?? ()
#5 0x00007f25af3b8c70 in ?? ()
#6 0x00000000000002f2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 5468):
#0 0x00007f25bcebffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 5467):
#0 0x00007f25bcebffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 5466):
#0 0x00007f25bcebffb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000004 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000126b8 in ?? ()
#5 0x00007f25b0bbc450 in ?? ()
#6 0x0000000000000008 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 5465):
#0 0x00007f25bcec3d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250622 19:58:16.112442 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 4241
I20250622 19:58:16.132687 4370 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:58:16.253882 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 4241
I20250622 19:58:16.281307 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 4383
I20250622 19:58:16.288795 4511 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:58:16.412349 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 4383
I20250622 19:58:16.436897 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 4526
I20250622 19:58:16.444737 4655 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:58:16.692589 6370 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 54555a8ecf924b339db8e8306cbc8bb4)
I20250622 19:58:16.692842 6370 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:58:16.693789 6370 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693), 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117)
I20250622 19:58:16.700690 5700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "9ece0d51c71b47a1865ba3b05920d999" candidate_term: 9 candidate_status { last_received { term: 8 index: 14321 } } ignore_live_leader: false dest_uuid: "7e22adb60d65484c9844ee7d2108a396" is_pre_election: true
I20250622 19:58:16.739853 6374 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 54555a8ecf924b339db8e8306cbc8bb4)
I20250622 19:58:16.740051 6374 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:58:16.740687 6374 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117), 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
I20250622 19:58:16.741268 3738 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "7e22adb60d65484c9844ee7d2108a396" candidate_term: 9 candidate_status { last_received { term: 8 index: 14321 } } ignore_live_leader: false dest_uuid: "9ece0d51c71b47a1865ba3b05920d999" is_pre_election: true
I20250622 19:58:16.741509 3738 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e22adb60d65484c9844ee7d2108a396 in term 8.
I20250622 19:58:16.741983 5629 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e22adb60d65484c9844ee7d2108a396, 9ece0d51c71b47a1865ba3b05920d999; no voters:
I20250622 19:58:16.742283 6374 raft_consensus.cc:2802] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250622 19:58:16.742362 6374 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 54555a8ecf924b339db8e8306cbc8bb4)
I20250622 19:58:16.742475 6374 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 8 FOLLOWER]: Advancing to term 9
I20250622 19:58:16.745827 6374 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:58:16.746486 6374 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 9 election: Requested vote from peers 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117), 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933)
I20250622 19:58:16.747080 3738 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "7e22adb60d65484c9844ee7d2108a396" candidate_term: 9 candidate_status { last_received { term: 8 index: 14321 } } ignore_live_leader: false dest_uuid: "9ece0d51c71b47a1865ba3b05920d999"
I20250622 19:58:16.747311 3738 raft_consensus.cc:3058] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 8 FOLLOWER]: Advancing to term 9
I20250622 19:58:16.750485 3738 raft_consensus.cc:2466] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e22adb60d65484c9844ee7d2108a396 in term 9.
I20250622 19:58:16.751019 5629 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e22adb60d65484c9844ee7d2108a396, 9ece0d51c71b47a1865ba3b05920d999; no voters:
I20250622 19:58:16.751310 6374 raft_consensus.cc:2802] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 9 FOLLOWER]: Leader election won for term 9
I20250622 19:58:16.751746 6374 raft_consensus.cc:695] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 9 LEADER]: Becoming Leader. State: Replica: 7e22adb60d65484c9844ee7d2108a396, State: Running, Role: LEADER
I20250622 19:58:16.752236 6374 consensus_queue.cc:237] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14321, Committed index: 14321, Last appended: 8.14321, Last appended by leader: 14321, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:58:16.755856 2054 catalog_manager.cc:5582] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 reported cstate change: term changed from 8 to 9, leader changed from 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3) to 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8). New cstate: current_term: 9 leader_uuid: "7e22adb60d65484c9844ee7d2108a396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } health_report { overall_health: UNKNOWN } } }
I20250622 19:58:16.870653 3738 raft_consensus.cc:1273] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 9 FOLLOWER]: Refusing update from remote peer 7e22adb60d65484c9844ee7d2108a396: Log matching property violated. Preceding OpId in replica: term: 8 index: 14321. Preceding OpId from leader: term: 9 index: 14322. (index mismatch)
I20250622 19:58:16.871276 6374 consensus_queue.cc:1035] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14322, Last known committed idx: 14321, Time since last communication: 0.000s
I20250622 19:58:16.906982 4612 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "7e22adb60d65484c9844ee7d2108a396" candidate_term: 9 candidate_status { last_received { term: 8 index: 14321 } } ignore_live_leader: false dest_uuid: "54555a8ecf924b339db8e8306cbc8bb4"
I20250622 19:58:16.907490 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 4526
I20250622 19:58:16.909875 5700 raft_consensus.cc:1238] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [term 9 LEADER]: Rejecting Update request from peer 54555a8ecf924b339db8e8306cbc8bb4 for earlier term 8. Current term is 9. Ops: []
I20250622 19:58:16.909912 3738 raft_consensus.cc:1238] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 9 FOLLOWER]: Rejecting Update request from peer 54555a8ecf924b339db8e8306cbc8bb4 for earlier term 8. Current term is 9. Ops: []
W20250622 19:58:16.973734 5630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.248.3:41117 (error 108)
W20250622 19:58:16.973856 2041 connection.cc:537] server connection from 127.1.248.3:52319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250622 19:58:16.973888 3670 connection.cc:537] server connection from 127.1.248.3:56659 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250622 19:58:16.974095 5629 connection.cc:537] server connection from 127.1.248.3:50685 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250622 19:58:16.974344 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 5012
W20250622 19:58:16.974640 6373 negotiation.cc:337] Failed RPC negotiation. Trace:
0622 19:58:16.694894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.248.3:41117 (local address 127.1.248.5:56257)
0622 19:58:16.695416 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0622 19:58:16.695432 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0622 19:58:16.695556 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0622 19:58:16.973612 (+278056us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.248.3:41117: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":308,"threads_started":1}
W20250622 19:58:16.975091 3670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.248.3:41117: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250622 19:58:16.976668 5630 leader_election.cc:336] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Network error: Client connection negotiation failed: client connection to 127.1.248.3:41117: connect: Connection refused (error 111)
W20250622 19:58:16.977010 5630 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.3:41117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250622 19:58:16.978135 3670 leader_election.cc:336] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Network error: Client connection negotiation failed: client connection to 127.1.248.3:41117: connect: Connection refused (error 111)
I20250622 19:58:16.978319 3670 leader_election.cc:304] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ece0d51c71b47a1865ba3b05920d999; no voters: 54555a8ecf924b339db8e8306cbc8bb4, 7e22adb60d65484c9844ee7d2108a396
I20250622 19:58:16.978667 6370 raft_consensus.cc:2747] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250622 19:58:16.982825 5137 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:58:17.099937 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 5012
I20250622 19:58:17.119552 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 3653
I20250622 19:58:17.125757 3778 generic_service.cc:196] Checking for leaks (request via RPC)
W20250622 19:58:17.441071 5630 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117): Couldn't send request to peer 54555a8ecf924b339db8e8306cbc8bb4. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.3:41117: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250622 19:58:17.488623 6370 raft_consensus.cc:491] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 7e22adb60d65484c9844ee7d2108a396)
I20250622 19:58:17.488829 6370 raft_consensus.cc:513] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e22adb60d65484c9844ee7d2108a396" member_type: VOTER last_known_addr { host: "127.1.248.8" port: 39693 } } peers { permanent_uuid: "54555a8ecf924b339db8e8306cbc8bb4" member_type: VOTER last_known_addr { host: "127.1.248.3" port: 41117 } } peers { permanent_uuid: "9ece0d51c71b47a1865ba3b05920d999" member_type: VOTER last_known_addr { host: "127.1.248.5" port: 43933 } }
I20250622 19:58:17.489501 6370 leader_election.cc:290] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 9ece0d51c71b47a1865ba3b05920d999 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 7e22adb60d65484c9844ee7d2108a396 (127.1.248.8:39693), 54555a8ecf924b339db8e8306cbc8bb4 (127.1.248.3:41117)
I20250622 19:58:17.490003 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 3653
I20250622 19:58:17.490903 5700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03ebbbde6ff2448a9a4c52e4cf8d160d" candidate_uuid: "9ece0d51c71b47a1865ba3b05920d999" candidate_term: 10 candidate_status { last_received { term: 9 index: 14322 } } ignore_live_leader: false dest_uuid: "7e22adb60d65484c9844ee7d2108a396" is_pre_election: true
W20250622 19:58:17.555666 5629 connection.cc:537] client connection to 127.1.248.5:43933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250622 19:58:17.556083 5629 connection.cc:537] server connection from 127.1.248.5:34333 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250622 19:58:17.556130 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 3510
W20250622 19:58:17.557005 5629 consensus_peers.cc:487] T 03ebbbde6ff2448a9a4c52e4cf8d160d P 7e22adb60d65484c9844ee7d2108a396 -> Peer 9ece0d51c71b47a1865ba3b05920d999 (127.1.248.5:43933): Couldn't send request to peer 9ece0d51c71b47a1865ba3b05920d999. Status: Network error: Client connection negotiation failed: client connection to 127.1.248.5:43933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250622 19:58:17.562783 3636 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:58:17.686066 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 3510
I20250622 19:58:17.711054 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 3792
I20250622 19:58:17.717375 3917 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:58:17.843587 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 3792
I20250622 19:58:17.868248 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 5613
I20250622 19:58:17.874444 5742 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:58:18.260360 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 5613
I20250622 19:58:18.320247 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 5465
I20250622 19:58:18.329152 5596 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:58:18.430502 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 5465
I20250622 19:58:18.455039 2016 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu pid 2024
I20250622 19:58:18.461385 2085 generic_service.cc:196] Checking for leaks (request via RPC)
I20250622 19:58:18.648703 2016 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskmkqhcg/build/asan/bin/kudu with pid 2024
2025-06-22T19:58:18Z chronyd exiting
I20250622 19:58:18.692430 2016 test_util.cc:183] -----------------------------------------------
I20250622 19:58:18.692551 2016 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskmkqhcg/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1750622104579871-2016-0
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3, where GetParam() = 800-byte object <01-00 00-00 01-61 00-00 09-00 00-00 BE-BE BE-BE A0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-1F 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 03-00 00-00 03-00 00-00 00-BE BE-BE BE-BE BE-BE> (194047 ms)
[----------] 1 test from RollingRestartArgs/RollingRestartITest (194047 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (194048 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3, where GetParam() = 800-byte object <01-00 00-00 01-61 00-00 09-00 00-00 BE-BE BE-BE A0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-1F 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 03-00 00-00 03-00 00-00 00-BE BE-BE BE-BE BE-BE>
1 FAILED TEST