Note: This is test shard 8 of 8.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RollingRestartArgs/RollingRestartITest
[ RUN ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3
2025-01-11T02:54:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-11T02:54:28Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250111 02:54:28.709088 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.190:40247
--webserver_interface=127.1.19.190
--webserver_port=0
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.19.190:40247
--location_mapping_cmd=/tmp/dist-test-taskaChIIQ/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/location-assignment.state --map /L0:3 --map /L1:4 --map /L2:4
--master_client_location_assignment_enabled=false with env {}
W20250111 02:54:29.040506 1110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:54:29.040848 1110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:54:29.040944 1110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:54:29.049216 1110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250111 02:54:29.049350 1110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:54:29.049396 1110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250111 02:54:29.049454 1110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250111 02:54:29.059227 1110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/wal
--location_mapping_cmd=/tmp/dist-test-taskaChIIQ/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/location-assignment.state --map /L0:3 --map /L1:4 --map /L2:4
--ipki_ca_key_size=768
--master_addresses=127.1.19.190:40247
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.19.190:40247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.19.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:54:29.060974 1110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:54:29.062798 1110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:29.074510 1115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:29.075431 1116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:29.076387 1118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:29.077400 1110 server_base.cc:1034] running on GCE node
I20250111 02:54:29.078305 1110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:54:29.079880 1110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:54:29.081115 1110 hybrid_clock.cc:648] HybridClock initialized: now 1736564069081021 us; error 82 us; skew 500 ppm
I20250111 02:54:29.081462 1110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:29.089399 1110 webserver.cc:458] Webserver started at http://127.1.19.190:37593/ using document root <none> and password file <none>
I20250111 02:54:29.090220 1110 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:29.090368 1110 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:29.090690 1110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:29.093576 1110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/data/instance:
uuid: "d713edd59d9544dc99edd0753427cb82"
format_stamp: "Formatted at 2025-01-11 02:54:29 on dist-test-slave-th66"
I20250111 02:54:29.094379 1110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/wal/instance:
uuid: "d713edd59d9544dc99edd0753427cb82"
format_stamp: "Formatted at 2025-01-11 02:54:29 on dist-test-slave-th66"
I20250111 02:54:29.099989 1110 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250111 02:54:29.103591 1124 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:29.105144 1110 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250111 02:54:29.105334 1110 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/wal
uuid: "d713edd59d9544dc99edd0753427cb82"
format_stamp: "Formatted at 2025-01-11 02:54:29 on dist-test-slave-th66"
I20250111 02:54:29.105520 1110 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:29.131072 1110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:29.131987 1110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:54:29.132305 1110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:29.150400 1110 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.190:40247
I20250111 02:54:29.150440 1175 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.190:40247 every 8 connection(s)
I20250111 02:54:29.151813 1110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/data/info.pb
I20250111 02:54:29.155625 1176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:29.158258 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 1110
I20250111 02:54:29.158563 1102 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/wal/instance
I20250111 02:54:29.168047 1176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82: Bootstrap starting.
I20250111 02:54:29.171308 1176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:29.172297 1176 log.cc:826] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82: Log is configured to *not* fsync() on all Append() calls
I20250111 02:54:29.174952 1176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82: No bootstrap required, opened a new log
I20250111 02:54:29.183802 1176 raft_consensus.cc:357] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d713edd59d9544dc99edd0753427cb82" member_type: VOTER last_known_addr { host: "127.1.19.190" port: 40247 } }
I20250111 02:54:29.184388 1176 raft_consensus.cc:383] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:29.184533 1176 raft_consensus.cc:738] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d713edd59d9544dc99edd0753427cb82, State: Initialized, Role: FOLLOWER
I20250111 02:54:29.185508 1176 consensus_queue.cc:260] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d713edd59d9544dc99edd0753427cb82" member_type: VOTER last_known_addr { host: "127.1.19.190" port: 40247 } }
I20250111 02:54:29.185770 1176 raft_consensus.cc:397] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:54:29.185954 1176 raft_consensus.cc:491] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:54:29.186193 1176 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:29.189018 1176 raft_consensus.cc:513] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d713edd59d9544dc99edd0753427cb82" member_type: VOTER last_known_addr { host: "127.1.19.190" port: 40247 } }
I20250111 02:54:29.189567 1176 leader_election.cc:304] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d713edd59d9544dc99edd0753427cb82; no voters:
I20250111 02:54:29.190201 1176 leader_election.cc:290] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:54:29.190332 1181 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:29.192324 1176 sys_catalog.cc:564] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:54:29.192832 1181 raft_consensus.cc:695] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [term 1 LEADER]: Becoming Leader. State: Replica: d713edd59d9544dc99edd0753427cb82, State: Running, Role: LEADER
I20250111 02:54:29.193409 1181 consensus_queue.cc:237] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d713edd59d9544dc99edd0753427cb82" member_type: VOTER last_known_addr { host: "127.1.19.190" port: 40247 } }
I20250111 02:54:29.196612 1182 sys_catalog.cc:455] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d713edd59d9544dc99edd0753427cb82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d713edd59d9544dc99edd0753427cb82" member_type: VOTER last_known_addr { host: "127.1.19.190" port: 40247 } } }
I20250111 02:54:29.196651 1183 sys_catalog.cc:455] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d713edd59d9544dc99edd0753427cb82. Latest consensus state: current_term: 1 leader_uuid: "d713edd59d9544dc99edd0753427cb82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d713edd59d9544dc99edd0753427cb82" member_type: VOTER last_known_addr { host: "127.1.19.190" port: 40247 } } }
I20250111 02:54:29.196955 1182 sys_catalog.cc:458] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [sys.catalog]: This master's current role is: LEADER
I20250111 02:54:29.196983 1183 sys_catalog.cc:458] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82 [sys.catalog]: This master's current role is: LEADER
I20250111 02:54:29.198486 1191 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:54:29.204027 1191 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:54:29.209060 1198 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250111 02:54:29.213871 1191 catalog_manager.cc:1348] Generated new cluster ID: d2b1e5739388420bb42d42ab0eaefc31
I20250111 02:54:29.213989 1191 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:54:29.223906 1191 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:54:29.225612 1191 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:54:29.238545 1191 catalog_manager.cc:5899] T 00000000000000000000000000000000 P d713edd59d9544dc99edd0753427cb82: Generated new TSK 0
I20250111 02:54:29.239508 1191 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:54:29.245883 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.129:0
--local_ip_for_outbound_sockets=127.1.19.129
--webserver_interface=127.1.19.129
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250111 02:54:29.584642 1200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:54:29.584947 1200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:54:29.585104 1200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:54:29.592921 1200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:54:29.593174 1200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.129
I20250111 02:54:29.602840 1200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.19.129
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.129
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:54:29.604585 1200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:54:29.606361 1200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:29.618888 1208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:29.619820 1206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:29.620776 1205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:29.621817 1200 server_base.cc:1034] running on GCE node
I20250111 02:54:29.622530 1200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:54:29.623553 1200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:54:29.624787 1200 hybrid_clock.cc:648] HybridClock initialized: now 1736564069624702 us; error 53 us; skew 500 ppm
I20250111 02:54:29.625154 1200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:29.627599 1200 webserver.cc:458] Webserver started at http://127.1.19.129:32843/ using document root <none> and password file <none>
I20250111 02:54:29.628392 1200 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:29.628521 1200 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:29.628865 1200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:29.631613 1200 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data/instance:
uuid: "b530fdca53b74ee8b97452b96b4e9d9a"
format_stamp: "Formatted at 2025-01-11 02:54:29 on dist-test-slave-th66"
I20250111 02:54:29.632381 1200 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/wal/instance:
uuid: "b530fdca53b74ee8b97452b96b4e9d9a"
format_stamp: "Formatted at 2025-01-11 02:54:29 on dist-test-slave-th66"
I20250111 02:54:29.638473 1200 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20250111 02:54:29.642172 1214 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:29.643721 1200 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250111 02:54:29.643903 1200 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/wal
uuid: "b530fdca53b74ee8b97452b96b4e9d9a"
format_stamp: "Formatted at 2025-01-11 02:54:29 on dist-test-slave-th66"
I20250111 02:54:29.644121 1200 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:29.665419 1200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:29.666347 1200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:54:29.666632 1200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:29.667598 1200 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:29.669238 1200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:29.669337 1200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:29.669445 1200 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:29.669509 1200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:29.704295 1200 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.129:43009
I20250111 02:54:29.704337 1326 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.129:43009 every 8 connection(s)
I20250111 02:54:29.705670 1200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data/info.pb
I20250111 02:54:29.706050 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 1200
I20250111 02:54:29.706267 1102 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/wal/instance
I20250111 02:54:29.710620 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.130:0
--local_ip_for_outbound_sockets=127.1.19.130
--webserver_interface=127.1.19.130
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250111 02:54:29.719760 1327 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:54:29.720165 1327 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:29.721349 1327 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:29.768956 1141 ts_manager.cc:194] Registered new tserver with Master: b530fdca53b74ee8b97452b96b4e9d9a (127.1.19.129:43009)
I20250111 02:54:29.770725 1141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.129:49585
W20250111 02:54:30.052357 1331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:54:30.052696 1331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:54:30.052835 1331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:54:30.060736 1331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:54:30.061010 1331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.130
I20250111 02:54:30.070461 1331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.19.130
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.130
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:54:30.072172 1331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:54:30.074023 1331 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:30.087245 1340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:30.088073 1337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:30.088914 1338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:30.089913 1331 server_base.cc:1034] running on GCE node
I20250111 02:54:30.090564 1331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:54:30.091573 1331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:54:30.092793 1331 hybrid_clock.cc:648] HybridClock initialized: now 1736564070092730 us; error 43 us; skew 500 ppm
I20250111 02:54:30.093103 1331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:30.095449 1331 webserver.cc:458] Webserver started at http://127.1.19.130:33123/ using document root <none> and password file <none>
I20250111 02:54:30.096204 1331 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:30.096335 1331 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:30.096726 1331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:30.099373 1331 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data/instance:
uuid: "ceb745fac9b04980b45a6b9fa9416bc2"
format_stamp: "Formatted at 2025-01-11 02:54:30 on dist-test-slave-th66"
I20250111 02:54:30.100152 1331 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/wal/instance:
uuid: "ceb745fac9b04980b45a6b9fa9416bc2"
format_stamp: "Formatted at 2025-01-11 02:54:30 on dist-test-slave-th66"
I20250111 02:54:30.105487 1331 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20250111 02:54:30.108816 1346 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:30.110165 1331 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250111 02:54:30.110309 1331 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/wal
uuid: "ceb745fac9b04980b45a6b9fa9416bc2"
format_stamp: "Formatted at 2025-01-11 02:54:30 on dist-test-slave-th66"
I20250111 02:54:30.110510 1331 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:30.132433 1331 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:30.133353 1331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:54:30.133621 1331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:30.134570 1331 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:30.136106 1331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:30.136188 1331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:30.136323 1331 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:30.136389 1331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:30.168599 1331 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.130:33299
I20250111 02:54:30.168608 1458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.130:33299 every 8 connection(s)
I20250111 02:54:30.171186 1331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data/info.pb
I20250111 02:54:30.180042 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 1331
I20250111 02:54:30.181605 1102 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/wal/instance
I20250111 02:54:30.187705 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.131:0
--local_ip_for_outbound_sockets=127.1.19.131
--webserver_interface=127.1.19.131
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250111 02:54:30.189567 1459 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:54:30.190028 1459 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:30.191134 1459 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:30.237121 1141 ts_manager.cc:194] Registered new tserver with Master: ceb745fac9b04980b45a6b9fa9416bc2 (127.1.19.130:33299)
I20250111 02:54:30.238658 1141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.130:40811
W20250111 02:54:30.528525 1462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:54:30.528882 1462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:54:30.529019 1462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:54:30.537288 1462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:54:30.537546 1462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.131
I20250111 02:54:30.547199 1462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.19.131
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.131
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:54:30.548964 1462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:54:30.550751 1462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:30.563292 1469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:30.564177 1471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:30.565001 1468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:30.565865 1462 server_base.cc:1034] running on GCE node
I20250111 02:54:30.566515 1462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:54:30.567490 1462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:54:30.568703 1462 hybrid_clock.cc:648] HybridClock initialized: now 1736564070568631 us; error 59 us; skew 500 ppm
I20250111 02:54:30.569018 1462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:30.571303 1462 webserver.cc:458] Webserver started at http://127.1.19.131:35523/ using document root <none> and password file <none>
I20250111 02:54:30.572072 1462 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:30.572201 1462 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:30.572505 1462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:30.575157 1462 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data/instance:
uuid: "486f7fe875934cebb51c6717b5e4312d"
format_stamp: "Formatted at 2025-01-11 02:54:30 on dist-test-slave-th66"
I20250111 02:54:30.575886 1462 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/wal/instance:
uuid: "486f7fe875934cebb51c6717b5e4312d"
format_stamp: "Formatted at 2025-01-11 02:54:30 on dist-test-slave-th66"
I20250111 02:54:30.581104 1462 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250111 02:54:30.584389 1477 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:30.585799 1462 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250111 02:54:30.585956 1462 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/wal
uuid: "486f7fe875934cebb51c6717b5e4312d"
format_stamp: "Formatted at 2025-01-11 02:54:30 on dist-test-slave-th66"
I20250111 02:54:30.586158 1462 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:30.606284 1462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:30.607098 1462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:54:30.607344 1462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:30.608271 1462 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:30.609789 1462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:30.609861 1462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:30.609954 1462 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:30.609997 1462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:30.639312 1462 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.131:37075
I20250111 02:54:30.639351 1589 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.131:37075 every 8 connection(s)
I20250111 02:54:30.640712 1462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data/info.pb
I20250111 02:54:30.650028 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 1462
I20250111 02:54:30.650219 1102 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/wal/instance
I20250111 02:54:30.653390 1590 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:54:30.653635 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.132:0
--local_ip_for_outbound_sockets=127.1.19.132
--webserver_interface=127.1.19.132
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250111 02:54:30.653690 1590 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:30.654510 1590 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:30.697257 1141 ts_manager.cc:194] Registered new tserver with Master: 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075)
I20250111 02:54:30.698192 1141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.131:57381
I20250111 02:54:30.773948 1327 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
W20250111 02:54:31.002274 1593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:54:31.002650 1593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:54:31.002811 1593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:54:31.011406 1593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:54:31.011670 1593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.132
I20250111 02:54:31.021368 1593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.1.19.132
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.132
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:54:31.023020 1593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:54:31.024816 1593 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:31.037770 1599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:31.038779 1600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:31.039710 1602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:31.040735 1593 server_base.cc:1034] running on GCE node
I20250111 02:54:31.041414 1593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:54:31.042423 1593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:54:31.043653 1593 hybrid_clock.cc:648] HybridClock initialized: now 1736564071043593 us; error 42 us; skew 500 ppm
I20250111 02:54:31.043967 1593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:31.046330 1593 webserver.cc:458] Webserver started at http://127.1.19.132:34781/ using document root <none> and password file <none>
I20250111 02:54:31.047106 1593 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:31.047238 1593 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:31.047536 1593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:31.050391 1593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data/instance:
uuid: "185ad035c2754007b1b3d08bb65546c2"
format_stamp: "Formatted at 2025-01-11 02:54:31 on dist-test-slave-th66"
I20250111 02:54:31.051141 1593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/wal/instance:
uuid: "185ad035c2754007b1b3d08bb65546c2"
format_stamp: "Formatted at 2025-01-11 02:54:31 on dist-test-slave-th66"
I20250111 02:54:31.057056 1593 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250111 02:54:31.060649 1608 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:31.062027 1593 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250111 02:54:31.062199 1593 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/wal
uuid: "185ad035c2754007b1b3d08bb65546c2"
format_stamp: "Formatted at 2025-01-11 02:54:31 on dist-test-slave-th66"
I20250111 02:54:31.062386 1593 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:31.084757 1593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:31.085633 1593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:54:31.085912 1593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:31.086855 1593 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:31.088382 1593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:31.088461 1593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:31.088614 1593 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:31.088670 1593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:31.122974 1593 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.132:41691
I20250111 02:54:31.122994 1720 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.132:41691 every 8 connection(s)
I20250111 02:54:31.124285 1593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data/info.pb
I20250111 02:54:31.133435 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 1593
I20250111 02:54:31.133602 1102 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/wal/instance
I20250111 02:54:31.136986 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.133:0
--local_ip_for_outbound_sockets=127.1.19.133
--webserver_interface=127.1.19.133
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250111 02:54:31.138396 1721 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:54:31.138687 1721 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:31.139495 1721 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:31.199378 1141 ts_manager.cc:194] Registered new tserver with Master: 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691)
I20250111 02:54:31.200364 1141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.132:52785
I20250111 02:54:31.241891 1459 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
W20250111 02:54:31.512450 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:54:31.512814 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:54:31.512967 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:54:31.521407 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:54:31.521668 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.133
I20250111 02:54:31.531077 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data/info.pb
--webserver_interface=127.1.19.133
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.133
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:54:31.532835 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:54:31.534667 1724 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:31.547835 1730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:31.548803 1731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:31.549647 1733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:31.550603 1724 server_base.cc:1034] running on GCE node
I20250111 02:54:31.551268 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:54:31.552277 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:54:31.553511 1724 hybrid_clock.cc:648] HybridClock initialized: now 1736564071553413 us; error 73 us; skew 500 ppm
I20250111 02:54:31.553831 1724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:31.556217 1724 webserver.cc:458] Webserver started at http://127.1.19.133:45769/ using document root <none> and password file <none>
I20250111 02:54:31.557073 1724 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:31.557204 1724 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:31.557510 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:31.560343 1724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data/instance:
uuid: "0eb58b1fa7c746f8abc51f33c199e832"
format_stamp: "Formatted at 2025-01-11 02:54:31 on dist-test-slave-th66"
I20250111 02:54:31.561161 1724 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/wal/instance:
uuid: "0eb58b1fa7c746f8abc51f33c199e832"
format_stamp: "Formatted at 2025-01-11 02:54:31 on dist-test-slave-th66"
I20250111 02:54:31.566395 1724 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250111 02:54:31.569861 1739 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:31.571285 1724 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250111 02:54:31.571915 1724 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/wal
uuid: "0eb58b1fa7c746f8abc51f33c199e832"
format_stamp: "Formatted at 2025-01-11 02:54:31 on dist-test-slave-th66"
I20250111 02:54:31.572117 1724 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:31.598870 1724 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:31.599771 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:54:31.600111 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:31.601562 1724 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:31.603130 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:31.603211 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:31.603340 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:31.603395 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:31.638218 1724 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.133:43355
I20250111 02:54:31.638273 1851 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.133:43355 every 8 connection(s)
I20250111 02:54:31.639659 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data/info.pb
I20250111 02:54:31.646328 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 1724
I20250111 02:54:31.646509 1102 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/wal/instance
I20250111 02:54:31.649649 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.134:0
--local_ip_for_outbound_sockets=127.1.19.134
--webserver_interface=127.1.19.134
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250111 02:54:31.654487 1852 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:54:31.654803 1852 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:31.655622 1852 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:31.701004 1590 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
I20250111 02:54:31.716813 1141 ts_manager.cc:194] Registered new tserver with Master: 0eb58b1fa7c746f8abc51f33c199e832 (127.1.19.133:43355)
I20250111 02:54:31.717777 1141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.133:47295
W20250111 02:54:32.021309 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:54:32.021622 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:54:32.021776 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:54:32.030206 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:54:32.030458 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.134
I20250111 02:54:32.040524 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data/info.pb
--webserver_interface=127.1.19.134
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.134
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:54:32.042272 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:54:32.044059 1856 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:32.057379 1862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:32.058338 1863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:32.059274 1865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:32.060248 1856 server_base.cc:1034] running on GCE node
I20250111 02:54:32.061022 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:54:32.062043 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:54:32.063289 1856 hybrid_clock.cc:648] HybridClock initialized: now 1736564072063217 us; error 47 us; skew 500 ppm
I20250111 02:54:32.063616 1856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:32.070248 1856 webserver.cc:458] Webserver started at http://127.1.19.134:39927/ using document root <none> and password file <none>
I20250111 02:54:32.071080 1856 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:32.071210 1856 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:32.071512 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:32.074294 1856 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data/instance:
uuid: "51a314e6744a4a11ab63fa6e76d4f780"
format_stamp: "Formatted at 2025-01-11 02:54:32 on dist-test-slave-th66"
I20250111 02:54:32.075084 1856 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/wal/instance:
uuid: "51a314e6744a4a11ab63fa6e76d4f780"
format_stamp: "Formatted at 2025-01-11 02:54:32 on dist-test-slave-th66"
I20250111 02:54:32.080688 1856 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20250111 02:54:32.084235 1871 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:32.085729 1856 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250111 02:54:32.085908 1856 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/wal
uuid: "51a314e6744a4a11ab63fa6e76d4f780"
format_stamp: "Formatted at 2025-01-11 02:54:32 on dist-test-slave-th66"
I20250111 02:54:32.086097 1856 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:32.111270 1856 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:32.112169 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:54:32.112478 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:32.113471 1856 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:32.115504 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:32.115592 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:32.115705 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:32.115752 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:32.150692 1856 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.134:44209
I20250111 02:54:32.150780 1983 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.134:44209 every 8 connection(s)
I20250111 02:54:32.152056 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data/info.pb
I20250111 02:54:32.159708 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 1856
I20250111 02:54:32.159896 1102 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/wal/instance
I20250111 02:54:32.163422 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.135:0
--local_ip_for_outbound_sockets=127.1.19.135
--webserver_interface=127.1.19.135
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250111 02:54:32.166065 1984 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:54:32.166405 1984 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:32.167392 1984 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:32.203706 1721 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
I20250111 02:54:32.213996 1141 ts_manager.cc:194] Registered new tserver with Master: 51a314e6744a4a11ab63fa6e76d4f780 (127.1.19.134:44209)
I20250111 02:54:32.214978 1141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.134:49873
W20250111 02:54:32.528734 1988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:54:32.529062 1988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:54:32.529225 1988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:54:32.537703 1988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:54:32.537977 1988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.135
I20250111 02:54:32.547477 1988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data/info.pb
--webserver_interface=127.1.19.135
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.135
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:54:32.549180 1988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:54:32.550954 1988 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:32.563795 1995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:32.564743 1994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:32.565598 1997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:32.566537 1988 server_base.cc:1034] running on GCE node
I20250111 02:54:32.567123 1988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:54:32.568122 1988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:54:32.569329 1988 hybrid_clock.cc:648] HybridClock initialized: now 1736564072569252 us; error 57 us; skew 500 ppm
I20250111 02:54:32.569638 1988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:32.571974 1988 webserver.cc:458] Webserver started at http://127.1.19.135:42067/ using document root <none> and password file <none>
I20250111 02:54:32.572773 1988 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:32.572896 1988 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:32.573201 1988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:32.575810 1988 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data/instance:
uuid: "34d6de1ad631433e9b596bf231a8c9d5"
format_stamp: "Formatted at 2025-01-11 02:54:32 on dist-test-slave-th66"
I20250111 02:54:32.576536 1988 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/wal/instance:
uuid: "34d6de1ad631433e9b596bf231a8c9d5"
format_stamp: "Formatted at 2025-01-11 02:54:32 on dist-test-slave-th66"
I20250111 02:54:32.581960 1988 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250111 02:54:32.585297 2003 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:32.587075 1988 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250111 02:54:32.587230 1988 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/wal
uuid: "34d6de1ad631433e9b596bf231a8c9d5"
format_stamp: "Formatted at 2025-01-11 02:54:32 on dist-test-slave-th66"
I20250111 02:54:32.587420 1988 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:32.599911 1988 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:32.600795 1988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:54:32.601075 1988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:32.601986 1988 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:32.603469 1988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:32.603541 1988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:32.603633 1988 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:32.603675 1988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:32.638998 1988 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.135:41329
I20250111 02:54:32.639084 2115 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.135:41329 every 8 connection(s)
I20250111 02:54:32.640494 1988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data/info.pb
I20250111 02:54:32.643136 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 1988
I20250111 02:54:32.643362 1102 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/wal/instance
I20250111 02:54:32.646821 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.136:0
--local_ip_for_outbound_sockets=127.1.19.136
--webserver_interface=127.1.19.136
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250111 02:54:32.655565 2116 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:54:32.655881 2116 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:32.656773 2116 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:32.720417 1852 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
I20250111 02:54:32.720605 1141 ts_manager.cc:194] Registered new tserver with Master: 34d6de1ad631433e9b596bf231a8c9d5 (127.1.19.135:41329)
I20250111 02:54:32.721959 1141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.135:44023
W20250111 02:54:33.006052 2119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:54:33.006366 2119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:54:33.006464 2119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:54:33.014936 2119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:54:33.015173 2119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.136
I20250111 02:54:33.024906 2119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data/info.pb
--webserver_interface=127.1.19.136
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.136
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:54:33.026521 2119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:54:33.028326 2119 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:33.042034 2125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:33.043190 2126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:33.044286 2128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:33.045588 2119 server_base.cc:1034] running on GCE node
I20250111 02:54:33.046278 2119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:54:33.047330 2119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:54:33.048645 2119 hybrid_clock.cc:648] HybridClock initialized: now 1736564073048497 us; error 60 us; skew 500 ppm
I20250111 02:54:33.048997 2119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:33.051515 2119 webserver.cc:458] Webserver started at http://127.1.19.136:42199/ using document root <none> and password file <none>
I20250111 02:54:33.052337 2119 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:33.052453 2119 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:33.052806 2119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:33.055595 2119 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data/instance:
uuid: "5d154c4e3a614e269ad20079b76f38d8"
format_stamp: "Formatted at 2025-01-11 02:54:33 on dist-test-slave-th66"
I20250111 02:54:33.056392 2119 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/wal/instance:
uuid: "5d154c4e3a614e269ad20079b76f38d8"
format_stamp: "Formatted at 2025-01-11 02:54:33 on dist-test-slave-th66"
I20250111 02:54:33.062423 2119 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20250111 02:54:33.066210 2134 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:33.067941 2119 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250111 02:54:33.068135 2119 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/wal
uuid: "5d154c4e3a614e269ad20079b76f38d8"
format_stamp: "Formatted at 2025-01-11 02:54:33 on dist-test-slave-th66"
I20250111 02:54:33.068337 2119 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:33.083179 2119 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:33.084059 2119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:54:33.084353 2119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:33.085374 2119 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:33.086902 2119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:33.086992 2119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:33.087133 2119 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:33.087209 2119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:33.124691 2119 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.136:45497
I20250111 02:54:33.124725 2246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.136:45497 every 8 connection(s)
I20250111 02:54:33.126114 2119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data/info.pb
I20250111 02:54:33.126681 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 2119
I20250111 02:54:33.126873 1102 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/wal/instance
I20250111 02:54:33.130643 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.137:0
--local_ip_for_outbound_sockets=127.1.19.137
--webserver_interface=127.1.19.137
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250111 02:54:33.143564 2247 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:54:33.143927 2247 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:33.144872 2247 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:33.213282 1141 ts_manager.cc:194] Registered new tserver with Master: 5d154c4e3a614e269ad20079b76f38d8 (127.1.19.136:45497)
I20250111 02:54:33.214200 1141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.136:44605
I20250111 02:54:33.218081 1984 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
W20250111 02:54:33.518051 2250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:54:33.518374 2250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:54:33.518512 2250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:54:33.526515 2250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:54:33.526785 2250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.137
I20250111 02:54:33.536769 2250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.137:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data/info.pb
--webserver_interface=127.1.19.137
--webserver_port=0
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.137
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:54:33.538489 2250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:54:33.540329 2250 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:33.553860 2259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:33.555073 2256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:33.556130 2257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:33.557328 2250 server_base.cc:1034] running on GCE node
I20250111 02:54:33.558082 2250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:54:33.559182 2250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:54:33.560469 2250 hybrid_clock.cc:648] HybridClock initialized: now 1736564073560382 us; error 64 us; skew 500 ppm
I20250111 02:54:33.560942 2250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:33.563714 2250 webserver.cc:458] Webserver started at http://127.1.19.137:35313/ using document root <none> and password file <none>
I20250111 02:54:33.564532 2250 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:33.564714 2250 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:33.565054 2250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:33.567860 2250 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data/instance:
uuid: "560d9bddebd54e0da0bf8d2da0a7a593"
format_stamp: "Formatted at 2025-01-11 02:54:33 on dist-test-slave-th66"
I20250111 02:54:33.568712 2250 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/wal/instance:
uuid: "560d9bddebd54e0da0bf8d2da0a7a593"
format_stamp: "Formatted at 2025-01-11 02:54:33 on dist-test-slave-th66"
I20250111 02:54:33.574270 2250 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250111 02:54:33.577661 2265 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:33.578984 2250 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250111 02:54:33.579138 2250 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/wal
uuid: "560d9bddebd54e0da0bf8d2da0a7a593"
format_stamp: "Formatted at 2025-01-11 02:54:33 on dist-test-slave-th66"
I20250111 02:54:33.579325 2250 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:33.605242 2250 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:33.606165 2250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:54:33.606483 2250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:33.607498 2250 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:33.609154 2250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:33.609228 2250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:33.609810 2250 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:33.609870 2250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20250111 02:54:33.646736 2250 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.137:39205
I20250111 02:54:33.646786 2377 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.137:39205 every 8 connection(s)
I20250111 02:54:33.648042 2250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data/info.pb
I20250111 02:54:33.653883 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 2250
I20250111 02:54:33.654067 1102 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/wal/instance
I20250111 02:54:33.662335 2378 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:54:33.662650 2378 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:33.663475 2378 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:33.710619 1141 ts_manager.cc:194] Registered new tserver with Master: 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205)
I20250111 02:54:33.711642 1102 external_mini_cluster.cc:934] 9 TS(s) registered with all masters
I20250111 02:54:33.711709 1141 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.137:55749
I20250111 02:54:33.724946 2116 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
I20250111 02:54:33.734282 1102 test_util.cc:274] Using random seed: 1421779039
I20250111 02:54:33.757251 1141 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60148:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:54:33.800803 2313 tablet_service.cc:1467] Processing CreateTablet for tablet a851ac2cebaa49d5b317fd02bc2766fa (DEFAULT_TABLE table=test-workload [id=550fe10398614a5c97652e92e9f4b488]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:54:33.800989 1525 tablet_service.cc:1467] Processing CreateTablet for tablet a851ac2cebaa49d5b317fd02bc2766fa (DEFAULT_TABLE table=test-workload [id=550fe10398614a5c97652e92e9f4b488]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:54:33.801208 1656 tablet_service.cc:1467] Processing CreateTablet for tablet a851ac2cebaa49d5b317fd02bc2766fa (DEFAULT_TABLE table=test-workload [id=550fe10398614a5c97652e92e9f4b488]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:54:33.802385 1525 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a851ac2cebaa49d5b317fd02bc2766fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:33.802419 2313 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a851ac2cebaa49d5b317fd02bc2766fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:33.802521 1656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a851ac2cebaa49d5b317fd02bc2766fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:33.813876 2403 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap starting.
I20250111 02:54:33.815737 2404 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Bootstrap starting.
I20250111 02:54:33.816926 2405 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Bootstrap starting.
I20250111 02:54:33.817883 2403 tablet_bootstrap.cc:654] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:33.819628 2404 tablet_bootstrap.cc:654] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:33.819597 2403 log.cc:826] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Log is configured to *not* fsync() on all Append() calls
I20250111 02:54:33.820479 2405 tablet_bootstrap.cc:654] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:33.820936 2404 log.cc:826] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Log is configured to *not* fsync() on all Append() calls
I20250111 02:54:33.821779 2405 log.cc:826] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Log is configured to *not* fsync() on all Append() calls
I20250111 02:54:33.823372 2403 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: No bootstrap required, opened a new log
I20250111 02:54:33.823668 2403 ts_tablet_manager.cc:1397] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20250111 02:54:33.824048 2404 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: No bootstrap required, opened a new log
I20250111 02:54:33.824327 2404 ts_tablet_manager.cc:1397] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20250111 02:54:33.825187 2405 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: No bootstrap required, opened a new log
I20250111 02:54:33.825419 2405 ts_tablet_manager.cc:1397] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20250111 02:54:33.831027 2405 raft_consensus.cc:357] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:54:33.831398 2405 raft_consensus.cc:383] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:33.831518 2405 raft_consensus.cc:738] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486f7fe875934cebb51c6717b5e4312d, State: Initialized, Role: FOLLOWER
I20250111 02:54:33.832156 2405 consensus_queue.cc:260] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:54:33.832823 2403 raft_consensus.cc:357] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:54:33.833323 2403 raft_consensus.cc:383] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:33.833465 2403 raft_consensus.cc:738] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185ad035c2754007b1b3d08bb65546c2, State: Initialized, Role: FOLLOWER
I20250111 02:54:33.833560 2404 raft_consensus.cc:357] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:54:33.833981 2404 raft_consensus.cc:383] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:33.834101 2404 raft_consensus.cc:738] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560d9bddebd54e0da0bf8d2da0a7a593, State: Initialized, Role: FOLLOWER
I20250111 02:54:33.834695 2405 ts_tablet_manager.cc:1428] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20250111 02:54:33.834900 2404 consensus_queue.cc:260] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:54:33.836001 2378 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
I20250111 02:54:33.836655 2404 ts_tablet_manager.cc:1428] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20250111 02:54:33.836675 2409 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:33.836894 2409 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:54:33.838478 2403 consensus_queue.cc:260] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:54:33.838824 2409 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691), 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205)
I20250111 02:54:33.847430 2403 ts_tablet_manager.cc:1428] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Time spent starting tablet: real 0.024s user 0.012s sys 0.006s
I20250111 02:54:33.851949 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "486f7fe875934cebb51c6717b5e4312d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2" is_pre_election: true
I20250111 02:54:33.852578 1676 raft_consensus.cc:2463] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 486f7fe875934cebb51c6717b5e4312d in term 0.
I20250111 02:54:33.853353 1480 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 185ad035c2754007b1b3d08bb65546c2, 486f7fe875934cebb51c6717b5e4312d; no voters:
I20250111 02:54:33.853731 2409 raft_consensus.cc:2798] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:54:33.853868 2409 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:33.853978 2409 raft_consensus.cc:3054] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:33.855490 2333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "486f7fe875934cebb51c6717b5e4312d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" is_pre_election: true
I20250111 02:54:33.855892 2333 raft_consensus.cc:2463] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 486f7fe875934cebb51c6717b5e4312d in term 0.
I20250111 02:54:33.856465 2409 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:54:33.857618 2409 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [CANDIDATE]: Term 1 election: Requested vote from peers 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691), 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205)
I20250111 02:54:33.858022 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "486f7fe875934cebb51c6717b5e4312d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2"
I20250111 02:54:33.858075 2333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "486f7fe875934cebb51c6717b5e4312d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "560d9bddebd54e0da0bf8d2da0a7a593"
I20250111 02:54:33.858199 1676 raft_consensus.cc:3054] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:33.858246 2333 raft_consensus.cc:3054] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:33.860252 2333 raft_consensus.cc:2463] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 486f7fe875934cebb51c6717b5e4312d in term 1.
I20250111 02:54:33.860301 1676 raft_consensus.cc:2463] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 486f7fe875934cebb51c6717b5e4312d in term 1.
I20250111 02:54:33.860659 1479 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 486f7fe875934cebb51c6717b5e4312d, 560d9bddebd54e0da0bf8d2da0a7a593; no voters:
I20250111 02:54:33.860944 2409 raft_consensus.cc:2798] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:33.861344 2409 raft_consensus.cc:695] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 1 LEADER]: Becoming Leader. State: Replica: 486f7fe875934cebb51c6717b5e4312d, State: Running, Role: LEADER
I20250111 02:54:33.861858 2409 consensus_queue.cc:237] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:54:33.866408 1141 catalog_manager.cc:5526] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d reported cstate change: term changed from 0 to 1, leader changed from <none> to 486f7fe875934cebb51c6717b5e4312d (127.1.19.131). New cstate: current_term: 1 leader_uuid: "486f7fe875934cebb51c6717b5e4312d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:33.874078 1102 maintenance_mode-itest.cc:745] Restarting batch of 3 tservers: 51a314e6744a4a11ab63fa6e76d4f780,34d6de1ad631433e9b596bf231a8c9d5,0eb58b1fa7c746f8abc51f33c199e832
W20250111 02:54:33.878451 1722 tablet.cc:2367] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250111 02:54:33.894179 1591 tablet.cc:2367] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250111 02:54:33.900985 2379 tablet.cc:2367] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:54:33.943854 2409 consensus_queue.cc:1035] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [LEADER]: Connected to new peer: Peer: permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:33.960623 2414 consensus_queue.cc:1035] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [LEADER]: Connected to new peer: Peer: permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:34.217056 2247 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
I20250111 02:54:35.269394 1139 ts_manager.cc:292] Set tserver state for 51a314e6744a4a11ab63fa6e76d4f780 to MAINTENANCE_MODE
I20250111 02:54:35.333541 1134 ts_manager.cc:292] Set tserver state for 34d6de1ad631433e9b596bf231a8c9d5 to MAINTENANCE_MODE
I20250111 02:54:35.426580 1134 ts_manager.cc:292] Set tserver state for 0eb58b1fa7c746f8abc51f33c199e832 to MAINTENANCE_MODE
I20250111 02:54:38.298911 2051 tablet_service.cc:1422] Tablet server 34d6de1ad631433e9b596bf231a8c9d5 set to quiescing
I20250111 02:54:38.299233 2051 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:54:38.574819 1787 tablet_service.cc:1422] Tablet server 0eb58b1fa7c746f8abc51f33c199e832 set to quiescing
I20250111 02:54:38.575129 1787 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:54:39.648679 1919 tablet_service.cc:1422] Tablet server 51a314e6744a4a11ab63fa6e76d4f780 set to quiescing
I20250111 02:54:39.648981 1919 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:54:41.439846 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 1856
I20250111 02:54:41.494575 1979 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:54:41.651918 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 1856
I20250111 02:54:41.730201 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.134:44209
--local_ip_for_outbound_sockets=127.1.19.134
--tserver_master_addrs=127.1.19.190:40247
--webserver_port=39927
--webserver_interface=127.1.19.134
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250111 02:54:42.359825 2547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:54:42.360249 2547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:54:42.360405 2547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:54:42.377305 2547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:54:42.378031 2547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.134
I20250111 02:54:42.395200 2547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.134:44209
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data/info.pb
--webserver_interface=127.1.19.134
--webserver_port=39927
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.134
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:54:42.397629 2547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:54:42.400127 2547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:54:42.452700 2547 server_base.cc:1034] running on GCE node
W20250111 02:54:42.455211 2561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:42.472782 2555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:42.473254 2556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:42.475291 2547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:54:42.493461 2547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:54:42.497504 2547 hybrid_clock.cc:648] HybridClock initialized: now 1736564082497381 us; error 83 us; skew 500 ppm
I20250111 02:54:42.499882 2547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:42.561903 2547 webserver.cc:458] Webserver started at http://127.1.19.134:39927/ using document root <none> and password file <none>
I20250111 02:54:42.562984 2547 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:42.565173 2547 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:42.577646 2547 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.006s sys 0.000s
I20250111 02:54:42.583343 2566 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:42.585510 2547 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20250111 02:54:42.585754 2547 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/wal
uuid: "51a314e6744a4a11ab63fa6e76d4f780"
format_stamp: "Formatted at 2025-01-11 02:54:32 on dist-test-slave-th66"
I20250111 02:54:42.586927 2547 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:42.623569 2547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:42.624869 2547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:54:42.625329 2547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:42.629020 2547 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:42.630738 2547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:42.630847 2547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:42.631006 2547 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:42.631081 2547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:42.983187 2678 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.134:44209 every 8 connection(s)
I20250111 02:54:42.983551 2547 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.134:44209
I20250111 02:54:42.985729 2547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data/info.pb
I20250111 02:54:42.997354 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 2547
I20250111 02:54:42.997630 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 1988
I20250111 02:54:43.021198 2111 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:54:43.055967 2679 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:54:43.056334 2679 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:43.057402 2679 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:43.061975 1133 ts_manager.cc:194] Re-registered known tserver with Master: 51a314e6744a4a11ab63fa6e76d4f780 (127.1.19.134:44209)
I20250111 02:54:43.063541 1133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.134:60563
I20250111 02:54:43.274252 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 1988
I20250111 02:54:43.306195 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.135:41329
--local_ip_for_outbound_sockets=127.1.19.135
--tserver_master_addrs=127.1.19.190:40247
--webserver_port=42067
--webserver_interface=127.1.19.135
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250111 02:54:44.068742 2679 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
W20250111 02:54:44.107754 2687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:54:44.108204 2687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:54:44.108366 2687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:54:44.140981 2687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:54:44.142256 2687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.135
I20250111 02:54:44.171476 2687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.135:41329
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data/info.pb
--webserver_interface=127.1.19.135
--webserver_port=42067
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.135
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:54:44.174100 2687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:54:44.176503 2687 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:44.228356 2695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:44.244423 2697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:44.245080 2694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:44.245489 2687 server_base.cc:1034] running on GCE node
I20250111 02:54:44.246702 2687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:54:44.248004 2687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:54:44.249459 2687 hybrid_clock.cc:648] HybridClock initialized: now 1736564084249327 us; error 101 us; skew 500 ppm
I20250111 02:54:44.249836 2687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:44.286587 2687 webserver.cc:458] Webserver started at http://127.1.19.135:42067/ using document root <none> and password file <none>
I20250111 02:54:44.287894 2687 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:44.288151 2687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:44.301915 2687 fs_manager.cc:714] Time spent opening directory manager: real 0.011s user 0.006s sys 0.001s
I20250111 02:54:44.310074 2703 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:44.312245 2687 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.001s sys 0.003s
I20250111 02:54:44.312434 2687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/wal
uuid: "34d6de1ad631433e9b596bf231a8c9d5"
format_stamp: "Formatted at 2025-01-11 02:54:32 on dist-test-slave-th66"
I20250111 02:54:44.313684 2687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:44.349090 2687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:44.350376 2687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:54:44.351351 2687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:44.358804 2687 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:44.367349 2687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:44.367534 2687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:44.367671 2687 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:44.367754 2687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:45.368012 2817 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.135:41329 every 8 connection(s)
I20250111 02:54:45.368389 2687 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.135:41329
I20250111 02:54:45.371042 2687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data/info.pb
I20250111 02:54:45.375272 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 2687
I20250111 02:54:45.375483 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 1724
I20250111 02:54:45.455332 1847 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:54:45.464298 2818 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:54:45.464695 2818 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:45.465734 2818 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:45.467785 1133 ts_manager.cc:194] Re-registered known tserver with Master: 34d6de1ad631433e9b596bf231a8c9d5 (127.1.19.135:41329)
I20250111 02:54:45.469040 1133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.135:38177
I20250111 02:54:45.654589 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 1724
I20250111 02:54:45.744778 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.133:43355
--local_ip_for_outbound_sockets=127.1.19.133
--tserver_master_addrs=127.1.19.190:40247
--webserver_port=45769
--webserver_interface=127.1.19.133
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250111 02:54:46.468878 2825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:54:46.469435 2825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:54:46.470034 2825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250111 02:54:46.475358 2818 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
W20250111 02:54:46.506800 2825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:54:46.507200 2825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.133
I20250111 02:54:46.528429 2825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.133:43355
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data/info.pb
--webserver_interface=127.1.19.133
--webserver_port=45769
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.133
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:54:46.530793 2825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:54:46.533301 2825 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:46.591889 2837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:46.594237 2839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:46.601016 2836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:46.607466 2825 server_base.cc:1034] running on GCE node
I20250111 02:54:46.608418 2825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:54:46.633602 2825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:54:46.639037 2825 hybrid_clock.cc:648] HybridClock initialized: now 1736564086638268 us; error 747 us; skew 500 ppm
I20250111 02:54:46.639570 2825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:46.683883 2825 webserver.cc:458] Webserver started at http://127.1.19.133:45769/ using document root <none> and password file <none>
I20250111 02:54:46.685201 2825 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:46.685456 2825 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:46.718917 2825 fs_manager.cc:714] Time spent opening directory manager: real 0.025s user 0.022s sys 0.001s
I20250111 02:54:46.726292 2845 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:46.729117 2825 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.002s sys 0.003s
I20250111 02:54:46.729389 2825 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/wal
uuid: "0eb58b1fa7c746f8abc51f33c199e832"
format_stamp: "Formatted at 2025-01-11 02:54:31 on dist-test-slave-th66"
I20250111 02:54:46.730674 2825 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:46.836129 2825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:46.838624 2825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:54:46.838959 2825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:46.840101 2825 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:46.844870 2825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:46.845068 2825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:46.845249 2825 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:46.845381 2825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:47.387534 2825 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.133:43355
I20250111 02:54:47.388670 2957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.133:43355 every 8 connection(s)
I20250111 02:54:47.389448 2825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data/info.pb
I20250111 02:54:47.396696 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 2825
I20250111 02:54:47.448038 2958 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:54:47.448357 2958 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:47.449641 2958 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:47.461629 1133 ts_manager.cc:194] Re-registered known tserver with Master: 0eb58b1fa7c746f8abc51f33c199e832 (127.1.19.133:43355)
I20250111 02:54:47.462461 1133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.133:44565
I20250111 02:54:48.468487 2958 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
I20250111 02:54:48.699632 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:54:48.700716 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:54:48.737128 1656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:54:48.765030 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:54:48.781433 2182 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:54:48.788322 2313 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:54:48.817735 1394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:54:48.820423 1525 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250111 02:54:48.865538 1262 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:54:50.446170 1135 ts_manager.cc:281] Unset tserver state for 0eb58b1fa7c746f8abc51f33c199e832 from MAINTENANCE_MODE
I20250111 02:54:50.477880 2958 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:50.486855 2818 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:50.813539 1135 ts_manager.cc:281] Unset tserver state for 51a314e6744a4a11ab63fa6e76d4f780 from MAINTENANCE_MODE
I20250111 02:54:50.834939 1327 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:50.837587 1135 ts_manager.cc:281] Unset tserver state for 34d6de1ad631433e9b596bf231a8c9d5 from MAINTENANCE_MODE
I20250111 02:54:51.062528 2378 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:51.076181 1721 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:51.085399 2679 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:51.125934 1590 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:51.315510 1459 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:51.325135 2247 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:51.488850 2958 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:54:51.492869 2818 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
W20250111 02:54:52.121188 1479 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.19.137:39205, user_credentials={real_user=slave}} blocked reactor thread for 47647.3us
I20250111 02:54:54.206283 1102 maintenance_mode-itest.cc:745] Restarting batch of 3 tservers: b530fdca53b74ee8b97452b96b4e9d9a,ceb745fac9b04980b45a6b9fa9416bc2,486f7fe875934cebb51c6717b5e4312d
I20250111 02:54:55.235538 1135 ts_manager.cc:292] Set tserver state for 486f7fe875934cebb51c6717b5e4312d to MAINTENANCE_MODE
I20250111 02:54:55.363094 1138 ts_manager.cc:292] Set tserver state for ceb745fac9b04980b45a6b9fa9416bc2 to MAINTENANCE_MODE
I20250111 02:54:56.138818 1138 ts_manager.cc:292] Set tserver state for b530fdca53b74ee8b97452b96b4e9d9a to MAINTENANCE_MODE
I20250111 02:54:59.150610 1525 tablet_service.cc:1422] Tablet server 486f7fe875934cebb51c6717b5e4312d set to quiescing
I20250111 02:54:59.150866 1525 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250111 02:54:59.164009 3087 raft_consensus.cc:988] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: : Instructing follower 185ad035c2754007b1b3d08bb65546c2 to start an election
I20250111 02:54:59.164147 3087 raft_consensus.cc:1076] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 1 LEADER]: Signalling peer 185ad035c2754007b1b3d08bb65546c2 to start an election
I20250111 02:54:59.165601 1675 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa"
dest_uuid: "185ad035c2754007b1b3d08bb65546c2"
from {username='slave'} at 127.1.19.131:41283
I20250111 02:54:59.165947 1675 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:54:59.166111 1675 raft_consensus.cc:3054] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:59.169132 1675 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:54:59.172039 1674 raft_consensus.cc:1235] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 2 FOLLOWER]: Rejecting Update request from peer 486f7fe875934cebb51c6717b5e4312d for earlier term 1. Current term is 2. Ops: []
I20250111 02:54:59.172787 3069 consensus_queue.cc:1046] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 }, Status: INVALID_TERM, Last received: 1.1648, Next index: 1649, Last known committed idx: 1647, Time since last communication: 0.000s
I20250111 02:54:59.173084 3069 raft_consensus.cc:3049] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:54:59.173147 3069 raft_consensus.cc:738] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 486f7fe875934cebb51c6717b5e4312d, State: Running, Role: LEADER
I20250111 02:54:59.173418 3069 consensus_queue.cc:260] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1648, Committed index: 1648, Last appended: 1.1648, Last appended by leader: 1648, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:54:59.173652 3069 raft_consensus.cc:3054] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:59.183912 1675 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [CANDIDATE]: Term 2 election: Requested vote from peers 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075), 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205)
W20250111 02:54:59.190587 1501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48030: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:54:59.216441 1503 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48030: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:54:59.228739 1502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48030: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:54:59.239424 1636 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
W20250111 02:54:59.239444 1634 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
W20250111 02:54:59.240993 1635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
I20250111 02:54:59.243579 1545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "185ad035c2754007b1b3d08bb65546c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1648 } } ignore_live_leader: true dest_uuid: "486f7fe875934cebb51c6717b5e4312d"
I20250111 02:54:59.247470 1545 raft_consensus.cc:2463] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185ad035c2754007b1b3d08bb65546c2 in term 2.
I20250111 02:54:59.248003 2333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "185ad035c2754007b1b3d08bb65546c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1648 } } ignore_live_leader: true dest_uuid: "560d9bddebd54e0da0bf8d2da0a7a593"
I20250111 02:54:59.248107 1611 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 185ad035c2754007b1b3d08bb65546c2, 486f7fe875934cebb51c6717b5e4312d; no voters:
I20250111 02:54:59.248284 2333 raft_consensus.cc:3054] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:59.251097 2333 raft_consensus.cc:2463] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 185ad035c2754007b1b3d08bb65546c2 in term 2.
I20250111 02:54:59.255714 3160 raft_consensus.cc:2798] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:54:59.268661 3160 raft_consensus.cc:695] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 2 LEADER]: Becoming Leader. State: Replica: 185ad035c2754007b1b3d08bb65546c2, State: Running, Role: LEADER
I20250111 02:54:59.269341 3160 consensus_queue.cc:237] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1647, Committed index: 1647, Last appended: 1.1648, Last appended by leader: 1648, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
W20250111 02:54:59.273221 2291 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58022: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
W20250111 02:54:59.274776 2293 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58022: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
W20250111 02:54:59.275949 2292 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58022: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
I20250111 02:54:59.280828 1138 catalog_manager.cc:5526] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 reported cstate change: term changed from 1 to 2, leader changed from 486f7fe875934cebb51c6717b5e4312d (127.1.19.131) to 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132). New cstate: current_term: 2 leader_uuid: "185ad035c2754007b1b3d08bb65546c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } health_report { overall_health: UNKNOWN } } }
W20250111 02:54:59.285944 1505 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48030: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:54:59.286085 1502 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48030: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:54:59.286689 1501 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48030: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
I20250111 02:54:59.296955 2333 raft_consensus.cc:1270] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Refusing update from remote peer 185ad035c2754007b1b3d08bb65546c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1648. Preceding OpId from leader: term: 2 index: 1650. (index mismatch)
I20250111 02:54:59.297816 3160 consensus_queue.cc:1035] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1649, Last known committed idx: 1647, Time since last communication: 0.000s
I20250111 02:54:59.306089 1545 raft_consensus.cc:1270] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 2 FOLLOWER]: Refusing update from remote peer 185ad035c2754007b1b3d08bb65546c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1648. Preceding OpId from leader: term: 2 index: 1650. (index mismatch)
I20250111 02:54:59.307169 3160 consensus_queue.cc:1035] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1649, Last known committed idx: 1648, Time since last communication: 0.000s
I20250111 02:55:00.097913 1262 tablet_service.cc:1422] Tablet server b530fdca53b74ee8b97452b96b4e9d9a set to quiescing
I20250111 02:55:00.098347 1262 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:00.195981 1394 tablet_service.cc:1422] Tablet server ceb745fac9b04980b45a6b9fa9416bc2 set to quiescing
I20250111 02:55:00.196262 1394 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:03.957494 1525 tablet_service.cc:1422] Tablet server 486f7fe875934cebb51c6717b5e4312d set to quiescing
I20250111 02:55:03.957804 1525 tablet_service.cc:1429] Tablet server has 0 leaders and 1 scanners
I20250111 02:55:07.409816 1525 tablet_service.cc:1422] Tablet server 486f7fe875934cebb51c6717b5e4312d set to quiescing
I20250111 02:55:07.410079 1525 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:09.300338 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 1200
I20250111 02:55:09.359114 1321 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:55:09.610530 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 1200
I20250111 02:55:09.665064 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.129:43009
--local_ip_for_outbound_sockets=127.1.19.129
--tserver_master_addrs=127.1.19.190:40247
--webserver_port=32843
--webserver_interface=127.1.19.129
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250111 02:55:10.423482 3254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:55:10.424024 3254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:55:10.424237 3254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:55:10.437160 3254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:55:10.437919 3254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.129
I20250111 02:55:10.455636 3254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.129:43009
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.19.129
--webserver_port=32843
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.129
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:55:10.458521 3254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:55:10.460994 3254 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:55:10.497217 3261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:55:10.501044 3262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:55:10.566984 3265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:55:10.568347 3254 server_base.cc:1034] running on GCE node
I20250111 02:55:10.570014 3254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:55:10.598873 3254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:55:10.600243 3254 hybrid_clock.cc:648] HybridClock initialized: now 1736564110600154 us; error 63 us; skew 500 ppm
I20250111 02:55:10.600802 3254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:55:10.637308 3254 webserver.cc:458] Webserver started at http://127.1.19.129:32843/ using document root <none> and password file <none>
I20250111 02:55:10.639010 3254 fs_manager.cc:362] Metadata directory not provided
I20250111 02:55:10.639302 3254 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:55:10.664295 3254 fs_manager.cc:714] Time spent opening directory manager: real 0.022s user 0.003s sys 0.007s
I20250111 02:55:10.675385 3271 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:55:10.677661 3254 fs_manager.cc:730] Time spent opening block manager: real 0.011s user 0.005s sys 0.003s
I20250111 02:55:10.677940 3254 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/wal
uuid: "b530fdca53b74ee8b97452b96b4e9d9a"
format_stamp: "Formatted at 2025-01-11 02:54:29 on dist-test-slave-th66"
I20250111 02:55:10.679276 3254 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:55:10.737632 3254 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:55:10.738777 3254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:55:10.739110 3254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:55:10.746695 3254 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:55:10.755180 3254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:55:10.755498 3254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250111 02:55:10.755664 3254 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:55:10.755788 3254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:55:11.773056 3254 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.129:43009
I20250111 02:55:11.773092 3385 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.129:43009 every 8 connection(s)
I20250111 02:55:11.776038 3254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data/info.pb
I20250111 02:55:11.778124 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 3254
I20250111 02:55:11.778321 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 1331
I20250111 02:55:11.833554 3386 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:55:11.833925 3386 heartbeater.cc:463] Registering TS with master...
I20250111 02:55:11.834969 3386 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:11.837180 1135 ts_manager.cc:194] Re-registered known tserver with Master: b530fdca53b74ee8b97452b96b4e9d9a (127.1.19.129:43009)
I20250111 02:55:11.838337 1135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.129:50165
I20250111 02:55:11.915699 1453 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:55:12.202171 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 1331
I20250111 02:55:12.245605 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.130:33299
--local_ip_for_outbound_sockets=127.1.19.130
--tserver_master_addrs=127.1.19.190:40247
--webserver_port=33123
--webserver_interface=127.1.19.130
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250111 02:55:12.847592 3386 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
W20250111 02:55:13.050496 3393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:55:13.050824 3393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:55:13.050984 3393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:55:13.062669 3393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:55:13.063047 3393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.130
I20250111 02:55:13.079427 3393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.130:33299
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.19.130
--webserver_port=33123
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.130
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:55:13.081862 3393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:55:13.084041 3393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:55:13.120394 3401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:55:13.121337 3400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:55:13.137889 3393 server_base.cc:1034] running on GCE node
W20250111 02:55:13.142431 3404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:55:13.150969 3393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:55:13.154687 3393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:55:13.158891 3393 hybrid_clock.cc:648] HybridClock initialized: now 1736564113158709 us; error 132 us; skew 500 ppm
I20250111 02:55:13.160333 3393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:55:13.207675 3393 webserver.cc:458] Webserver started at http://127.1.19.130:33123/ using document root <none> and password file <none>
I20250111 02:55:13.208649 3393 fs_manager.cc:362] Metadata directory not provided
I20250111 02:55:13.209012 3393 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:55:13.225548 3393 fs_manager.cc:714] Time spent opening directory manager: real 0.013s user 0.004s sys 0.000s
I20250111 02:55:13.239245 3409 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:55:13.242440 3393 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.001s sys 0.001s
I20250111 02:55:13.242745 3393 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/wal
uuid: "ceb745fac9b04980b45a6b9fa9416bc2"
format_stamp: "Formatted at 2025-01-11 02:54:30 on dist-test-slave-th66"
I20250111 02:55:13.244163 3393 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:55:13.328230 3393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:55:13.329813 3393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:55:13.330202 3393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:55:13.351230 3393 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:55:13.355825 3393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:55:13.356142 3393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250111 02:55:13.356338 3393 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:55:13.356469 3393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:55:14.397185 3393 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.130:33299
I20250111 02:55:14.397239 3524 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.130:33299 every 8 connection(s)
I20250111 02:55:14.399320 3393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data/info.pb
I20250111 02:55:14.409296 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 3393
I20250111 02:55:14.409550 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 1462
I20250111 02:55:14.507685 1585 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:55:14.514395 3525 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:55:14.514911 3525 heartbeater.cc:463] Registering TS with master...
I20250111 02:55:14.516368 3525 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:14.519251 1135 ts_manager.cc:194] Re-registered known tserver with Master: ceb745fac9b04980b45a6b9fa9416bc2 (127.1.19.130:33299)
I20250111 02:55:14.520372 1135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.130:50641
W20250111 02:55:14.856629 2415 log.cc:926] Time spent T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Append to log took a long time: real 0.345s user 0.000s sys 0.000s
W20250111 02:55:14.864993 3532 raft_consensus.cc:668] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: failed to trigger leader election: Illegal state: leader elections are disabled
I20250111 02:55:14.866403 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 1462
W20250111 02:55:14.950876 1611 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.19.131:37075 (error 108)
I20250111 02:55:14.952347 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.131:37075
--local_ip_for_outbound_sockets=127.1.19.131
--tserver_master_addrs=127.1.19.190:40247
--webserver_port=35523
--webserver_interface=127.1.19.131
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250111 02:55:14.966269 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.131:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:55:15.041173 2419 meta_cache.cc:1510] marking tablet server 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075) as failed
W20250111 02:55:15.041565 2419 meta_cache.cc:302] tablet a851ac2cebaa49d5b317fd02bc2766fa: replica 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.1.19.131:37075: connect: Connection refused (error 111)
W20250111 02:55:15.453037 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.131:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250111 02:55:15.511847 3535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:55:15.512271 3535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:55:15.512434 3535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250111 02:55:15.524482 3525 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
W20250111 02:55:15.525095 3535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:55:15.525440 3535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.131
I20250111 02:55:15.540760 3535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.131:37075
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.19.131
--webserver_port=35523
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.131
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:55:15.543098 3535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:55:15.545549 3535 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:55:15.578981 3543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:55:15.585867 3535 server_base.cc:1034] running on GCE node
W20250111 02:55:15.588093 3542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:55:15.597425 3545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:55:15.598652 3535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:55:15.607105 3535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:55:15.610726 3535 hybrid_clock.cc:648] HybridClock initialized: now 1736564115610221 us; error 450 us; skew 500 ppm
I20250111 02:55:15.611465 3535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:55:15.621097 3535 webserver.cc:458] Webserver started at http://127.1.19.131:35523/ using document root <none> and password file <none>
I20250111 02:55:15.622313 3535 fs_manager.cc:362] Metadata directory not provided
I20250111 02:55:15.622502 3535 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:55:15.630832 3535 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20250111 02:55:15.649914 3551 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:55:15.653532 3535 fs_manager.cc:730] Time spent opening block manager: real 0.020s user 0.002s sys 0.012s
I20250111 02:55:15.654848 3535 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/wal
uuid: "486f7fe875934cebb51c6717b5e4312d"
format_stamp: "Formatted at 2025-01-11 02:54:30 on dist-test-slave-th66"
I20250111 02:55:15.656490 3535 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:55:15.702167 3535 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:55:15.703400 3535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:55:15.716414 3535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:55:15.727396 3535 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:55:15.742873 3560 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250111 02:55:15.753091 3535 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250111 02:55:15.753326 3535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.023s user 0.003s sys 0.005s
I20250111 02:55:15.753535 3535 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250111 02:55:15.758613 3535 ts_tablet_manager.cc:610] Registered 1 tablets
I20250111 02:55:15.758816 3535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250111 02:55:15.759905 3560 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Bootstrap starting.
W20250111 02:55:15.930217 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.131:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250111 02:55:16.478295 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.131:37075: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250111 02:55:16.986498 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.131:37075: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250111 02:55:16.995903 3535 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.131:37075
I20250111 02:55:16.995903 3672 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.131:37075 every 8 connection(s)
I20250111 02:55:16.998373 3535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data/info.pb
I20250111 02:55:17.000331 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 3535
I20250111 02:55:17.052886 3673 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:55:17.053282 3673 heartbeater.cc:463] Registering TS with master...
I20250111 02:55:17.054523 3673 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:17.060374 1135 ts_manager.cc:194] Re-registered known tserver with Master: 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075)
I20250111 02:55:17.061986 1135 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.131:38647
W20250111 02:55:17.524302 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
W20250111 02:55:18.058307 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250111 02:55:18.068337 3673 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
I20250111 02:55:18.120596 3560 log.cc:826] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Log is configured to *not* fsync() on all Append() calls
W20250111 02:55:18.634048 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20250111 02:55:18.721208 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:18.767860 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:18.768632 1656 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250111 02:55:18.790067 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:18.797892 2313 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250111 02:55:18.810340 2182 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:18.830861 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:18.863308 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:18.934358 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:55:19.372319 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
I20250111 02:55:19.742813 3730 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 486f7fe875934cebb51c6717b5e4312d is lagging by at least 5 ops behind the committed index
W20250111 02:55:19.925057 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250111 02:55:20.436822 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
W20250111 02:55:20.953585 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
W20250111 02:55:21.472050 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
W20250111 02:55:21.928421 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
W20250111 02:55:22.450523 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
I20250111 02:55:22.838954 3730 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 486f7fe875934cebb51c6717b5e4312d is lagging by at least 328 ops behind the committed index [suppressed 29 similar messages]
W20250111 02:55:22.978928 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
I20250111 02:55:23.129520 3560 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Bootstrap replaying log segment 1/1 (1.65M/4.95M this segment, stats: ops{read=953 overwritten=0 applied=952 ignored=0} inserts{seen=47550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:55:23.459021 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250111 02:55:24.036691 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
W20250111 02:55:24.524204 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
W20250111 02:55:25.060789 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
W20250111 02:55:25.643852 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
I20250111 02:55:25.904989 3738 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 486f7fe875934cebb51c6717b5e4312d is lagging by at least 623 ops behind the committed index [suppressed 28 similar messages]
I20250111 02:55:26.056844 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:26.077383 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:26.084087 1656 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250111 02:55:26.108865 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:55:26.154102 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
I20250111 02:55:26.171001 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:26.211199 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:26.229779 2182 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:26.241783 2313 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250111 02:55:26.246897 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:55:26.718497 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
W20250111 02:55:27.180152 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
W20250111 02:55:27.728322 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
I20250111 02:55:28.132462 3560 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Bootstrap replaying log segment 1/1 (3.36M/4.95M this segment, stats: ops{read=1950 overwritten=0 applied=1950 ignored=0} inserts{seen=97400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:55:28.248809 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
W20250111 02:55:28.782701 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
I20250111 02:55:28.952147 3751 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 486f7fe875934cebb51c6717b5e4312d is lagging by at least 912 ops behind the committed index [suppressed 28 similar messages]
W20250111 02:55:29.309962 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
W20250111 02:55:29.802698 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
W20250111 02:55:30.375798 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
W20250111 02:55:30.913021 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
W20250111 02:55:31.404971 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 156: this message will repeat every 5th retry.
I20250111 02:55:31.895186 3560 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2888 overwritten=0 applied=2887 ignored=0} inserts{seen=144250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250111 02:55:31.896198 3560 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Bootstrap complete.
I20250111 02:55:31.899466 3560 ts_tablet_manager.cc:1397] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Time spent bootstrapping tablet: real 16.140s user 12.215s sys 0.633s
W20250111 02:55:31.909272 1611 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 161: this message will repeat every 5th retry.
I20250111 02:55:31.910982 3560 raft_consensus.cc:357] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:55:31.924206 3560 raft_consensus.cc:738] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486f7fe875934cebb51c6717b5e4312d, State: Initialized, Role: FOLLOWER
I20250111 02:55:31.925226 3560 consensus_queue.cc:260] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2887, Last appended: 2.2888, Last appended by leader: 2888, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:55:31.927431 3560 ts_tablet_manager.cc:1428] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Time spent starting tablet: real 0.028s user 0.016s sys 0.000s
I20250111 02:55:31.999533 3794 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 486f7fe875934cebb51c6717b5e4312d is lagging by at least 1221 ops behind the committed index [suppressed 28 similar messages]
I20250111 02:55:32.804869 3818 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 185ad035c2754007b1b3d08bb65546c2)
I20250111 02:55:32.805186 3818 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:55:32.807494 3818 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691), 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205)
I20250111 02:55:32.854586 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "486f7fe875934cebb51c6717b5e4312d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4163 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2" is_pre_election: true
I20250111 02:55:32.878578 2333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "486f7fe875934cebb51c6717b5e4312d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4163 } } ignore_live_leader: false dest_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" is_pre_election: true
I20250111 02:55:32.879253 3553 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 486f7fe875934cebb51c6717b5e4312d; no voters: 185ad035c2754007b1b3d08bb65546c2, 560d9bddebd54e0da0bf8d2da0a7a593
I20250111 02:55:32.879834 3818 raft_consensus.cc:2743] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250111 02:55:33.260599 3818 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 185ad035c2754007b1b3d08bb65546c2)
I20250111 02:55:33.260838 3818 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:55:33.263576 3818 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691), 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205)
I20250111 02:55:33.264007 2333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "486f7fe875934cebb51c6717b5e4312d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4578 } } ignore_live_leader: false dest_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" is_pre_election: true
I20250111 02:55:33.264174 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "486f7fe875934cebb51c6717b5e4312d" candidate_term: 3 candidate_status { last_received { term: 2 index: 4578 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2" is_pre_election: true
I20250111 02:55:33.264839 3554 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 486f7fe875934cebb51c6717b5e4312d; no voters: 185ad035c2754007b1b3d08bb65546c2, 560d9bddebd54e0da0bf8d2da0a7a593
I20250111 02:55:33.265451 3818 raft_consensus.cc:2743] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250111 02:55:33.535279 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:33.593420 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:33.601405 1656 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250111 02:55:33.673475 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:33.712401 2313 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250111 02:55:33.727507 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:33.751384 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:33.768836 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:33.785070 2182 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:36.133720 1138 ts_manager.cc:281] Unset tserver state for 486f7fe875934cebb51c6717b5e4312d from MAINTENANCE_MODE
I20250111 02:55:36.148530 1135 ts_manager.cc:281] Unset tserver state for ceb745fac9b04980b45a6b9fa9416bc2 from MAINTENANCE_MODE
I20250111 02:55:36.260413 1721 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:36.280268 3673 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:36.368829 2679 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:36.440374 2247 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:36.586166 3525 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:36.618542 2818 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:36.682387 2958 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:36.694777 2378 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:36.908264 3386 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:37.553159 1135 ts_manager.cc:281] Unset tserver state for b530fdca53b74ee8b97452b96b4e9d9a from MAINTENANCE_MODE
I20250111 02:55:37.594741 3525 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:37.623242 2818 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:37.686218 2958 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:37.703894 2378 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:37.913362 3386 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:38.272737 1721 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:38.292193 3673 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:38.397902 2679 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:38.445809 2247 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:55:40.135983 1102 maintenance_mode-itest.cc:745] Restarting batch of 2 tservers: 5d154c4e3a614e269ad20079b76f38d8,560d9bddebd54e0da0bf8d2da0a7a593
I20250111 02:55:41.592715 1135 ts_manager.cc:292] Set tserver state for 560d9bddebd54e0da0bf8d2da0a7a593 to MAINTENANCE_MODE
I20250111 02:55:42.313032 1135 ts_manager.cc:292] Set tserver state for 5d154c4e3a614e269ad20079b76f38d8 to MAINTENANCE_MODE
I20250111 02:55:45.215135 2313 tablet_service.cc:1422] Tablet server 560d9bddebd54e0da0bf8d2da0a7a593 set to quiescing
I20250111 02:55:45.215447 2313 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
W20250111 02:55:45.270920 2417 meta_cache.cc:1261] Time spent looking up entry by key: real 0.050s user 0.000s sys 0.000s
I20250111 02:55:47.598152 2182 tablet_service.cc:1422] Tablet server 5d154c4e3a614e269ad20079b76f38d8 set to quiescing
I20250111 02:55:47.598433 2182 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:55:49.162451 2417 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s user 0.003s sys 0.000s
I20250111 02:55:49.930545 2313 tablet_service.cc:1422] Tablet server 560d9bddebd54e0da0bf8d2da0a7a593 set to quiescing
I20250111 02:55:49.930805 2313 tablet_service.cc:1429] Tablet server has 0 leaders and 2 scanners
I20250111 02:55:53.394573 2313 tablet_service.cc:1422] Tablet server 560d9bddebd54e0da0bf8d2da0a7a593 set to quiescing
I20250111 02:55:53.394833 2313 tablet_service.cc:1429] Tablet server has 0 leaders and 2 scanners
I20250111 02:55:55.440457 2313 tablet_service.cc:1422] Tablet server 560d9bddebd54e0da0bf8d2da0a7a593 set to quiescing
I20250111 02:55:55.440701 2313 tablet_service.cc:1429] Tablet server has 0 leaders and 1 scanners
I20250111 02:55:58.987284 2313 tablet_service.cc:1422] Tablet server 560d9bddebd54e0da0bf8d2da0a7a593 set to quiescing
I20250111 02:55:58.987550 2313 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:55:59.235261 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 2119
I20250111 02:55:59.265534 2241 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:55:59.471453 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 2119
I20250111 02:55:59.506757 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.136:45497
--local_ip_for_outbound_sockets=127.1.19.136
--tserver_master_addrs=127.1.19.190:40247
--webserver_port=42199
--webserver_interface=127.1.19.136
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250111 02:56:00.156533 4117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:56:00.157061 4117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:56:00.157264 4117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:56:00.170921 4117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:56:00.171543 4117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.136
I20250111 02:56:00.188777 4117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.136:45497
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data/info.pb
--webserver_interface=127.1.19.136
--webserver_port=42199
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.136
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:56:00.191231 4117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:56:00.193648 4117 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:56:00.235107 4122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:56:00.236917 4123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:56:00.253041 4117 server_base.cc:1034] running on GCE node
W20250111 02:56:00.268187 4126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:56:00.269182 4117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:56:00.277150 4117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:56:00.293221 4117 hybrid_clock.cc:648] HybridClock initialized: now 1736564160286018 us; error 7146 us; skew 500 ppm
I20250111 02:56:00.297437 4117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:56:00.336687 4117 webserver.cc:458] Webserver started at http://127.1.19.136:42199/ using document root <none> and password file <none>
I20250111 02:56:00.337915 4117 fs_manager.cc:362] Metadata directory not provided
I20250111 02:56:00.338155 4117 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:56:00.385995 4117 fs_manager.cc:714] Time spent opening directory manager: real 0.045s user 0.003s sys 0.001s
I20250111 02:56:00.389508 4132 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:56:00.390997 4117 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250111 02:56:00.391155 4117 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/wal
uuid: "5d154c4e3a614e269ad20079b76f38d8"
format_stamp: "Formatted at 2025-01-11 02:54:33 on dist-test-slave-th66"
I20250111 02:56:00.392014 4117 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:56:00.406183 4117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:56:00.407195 4117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:56:00.407596 4117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:56:00.411577 4117 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:56:00.413727 4117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:56:00.413831 4117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:56:00.413980 4117 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:56:00.414057 4117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20250111 02:56:00.420019 2390 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.19.132:41691, user_credentials={real_user=slave}} blocked reactor thread for 48427us
I20250111 02:56:01.118238 4246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.136:45497 every 8 connection(s)
I20250111 02:56:01.121029 4117 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.136:45497
I20250111 02:56:01.123136 4117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data/info.pb
I20250111 02:56:01.128528 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 4117
I20250111 02:56:01.128783 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 2250
I20250111 02:56:01.202500 4247 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:56:01.202852 4247 heartbeater.cc:463] Registering TS with master...
I20250111 02:56:01.203891 4247 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:56:01.206024 1133 ts_manager.cc:194] Re-registered known tserver with Master: 5d154c4e3a614e269ad20079b76f38d8 (127.1.19.136:45497)
I20250111 02:56:01.207136 1133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.136:49353
I20250111 02:56:01.216753 2373 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:56:01.673557 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 2250
W20250111 02:56:01.803680 1610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.19.137:39205 (error 108) [suppressed 20 similar messages]
I20250111 02:56:01.812486 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.137:39205
--local_ip_for_outbound_sockets=127.1.19.137
--tserver_master_addrs=127.1.19.190:40247
--webserver_port=35313
--webserver_interface=127.1.19.137
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250111 02:56:01.823091 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.137:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:56:02.210587 4247 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
W20250111 02:56:02.304665 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.137:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250111 02:56:02.461256 4258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:56:02.461751 4258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:56:02.461934 4258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:56:02.474270 4258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:56:02.474642 4258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.137
I20250111 02:56:02.488451 4258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.137:39205
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data/info.pb
--webserver_interface=127.1.19.137
--webserver_port=35313
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.137
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:56:02.490880 4258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:56:02.493386 4258 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:56:02.539170 4267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:56:02.542732 4269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:56:02.552407 4266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:56:02.553917 4258 server_base.cc:1034] running on GCE node
I20250111 02:56:02.554911 4258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:56:02.566581 4258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:56:02.568090 4258 hybrid_clock.cc:648] HybridClock initialized: now 1736564162567990 us; error 73 us; skew 500 ppm
I20250111 02:56:02.568583 4258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:56:02.580638 4258 webserver.cc:458] Webserver started at http://127.1.19.137:35313/ using document root <none> and password file <none>
I20250111 02:56:02.581914 4258 fs_manager.cc:362] Metadata directory not provided
I20250111 02:56:02.582175 4258 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:56:02.610234 4258 fs_manager.cc:714] Time spent opening directory manager: real 0.025s user 0.008s sys 0.008s
I20250111 02:56:02.618501 4275 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:56:02.620929 4258 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20250111 02:56:02.621196 4258 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/wal
uuid: "560d9bddebd54e0da0bf8d2da0a7a593"
format_stamp: "Formatted at 2025-01-11 02:54:33 on dist-test-slave-th66"
I20250111 02:56:02.622892 4258 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:56:02.721167 4258 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:56:02.722342 4258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:56:02.722769 4258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:56:02.745685 4258 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:56:02.778134 4284 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20250111 02:56:02.806847 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.137:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250111 02:56:02.820690 4258 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250111 02:56:02.820935 4258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.056s user 0.006s sys 0.005s
I20250111 02:56:02.821120 4258 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250111 02:56:02.831679 4258 ts_tablet_manager.cc:610] Registered 1 tablets
I20250111 02:56:02.831876 4258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.004s sys 0.000s
I20250111 02:56:02.832214 4284 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Bootstrap starting.
W20250111 02:56:03.365842 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.137:39205: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250111 02:56:03.721650 4393 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.137:39205 every 8 connection(s)
I20250111 02:56:03.722048 4258 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.137:39205
I20250111 02:56:03.724123 4258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data/info.pb
I20250111 02:56:03.729851 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 4258
I20250111 02:56:03.781585 4394 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:56:03.781966 4394 heartbeater.cc:463] Registering TS with master...
I20250111 02:56:03.783524 4394 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:56:03.786358 1133 ts_manager.cc:194] Re-registered known tserver with Master: 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205)
I20250111 02:56:03.787972 1133 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.137:46543
W20250111 02:56:03.935696 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
I20250111 02:56:04.296947 4284 log.cc:826] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Log is configured to *not* fsync() on all Append() calls
W20250111 02:56:04.407493 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
I20250111 02:56:04.791433 4394 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
W20250111 02:56:04.882427 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250111 02:56:05.342550 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:05.432927 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:05.435449 1656 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250111 02:56:05.474700 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:05.554244 4329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:05.568536 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:05.579416 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:05.581230 4181 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:56:05.599635 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20250111 02:56:05.604445 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:05.933917 4315 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 560d9bddebd54e0da0bf8d2da0a7a593 is lagging by at least 11 ops behind the committed index
W20250111 02:56:06.080145 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20250111 02:56:06.598147 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250111 02:56:07.094349 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
W20250111 02:56:07.619885 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
W20250111 02:56:08.068733 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
W20250111 02:56:08.585109 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
I20250111 02:56:08.940867 4259 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 560d9bddebd54e0da0bf8d2da0a7a593 is lagging by at least 347 ops behind the committed index [suppressed 29 similar messages]
W20250111 02:56:09.059484 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
I20250111 02:56:09.307431 4284 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Bootstrap replaying log segment 1/2 (1.53M/8.00M this segment, stats: ops{read=906 overwritten=0 applied=903 ignored=0} inserts{seen=45100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:56:09.730022 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
W20250111 02:56:10.216755 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250111 02:56:10.733080 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
W20250111 02:56:11.226403 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
W20250111 02:56:11.796749 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
I20250111 02:56:11.947654 4450 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 560d9bddebd54e0da0bf8d2da0a7a593 is lagging by at least 651 ops behind the committed index [suppressed 27 similar messages]
W20250111 02:56:12.271378 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
I20250111 02:56:12.583019 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:12.649593 1656 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250111 02:56:12.674279 4329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:12.682860 4181 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:12.695287 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:12.750437 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:12.802035 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:12.810878 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:12.840830 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:56:12.902680 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20250111 02:56:13.525084 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
W20250111 02:56:13.953161 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
I20250111 02:56:14.308696 4284 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Bootstrap replaying log segment 1/2 (3.23M/8.00M this segment, stats: ops{read=1906 overwritten=0 applied=1904 ignored=0} inserts{seen=95100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:56:14.460734 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
W20250111 02:56:14.972194 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
I20250111 02:56:15.032393 4459 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 560d9bddebd54e0da0bf8d2da0a7a593 is lagging by at least 931 ops behind the committed index [suppressed 28 similar messages]
W20250111 02:56:15.613996 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
W20250111 02:56:16.144785 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
W20250111 02:56:16.640619 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
W20250111 02:56:17.128932 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
W20250111 02:56:17.604482 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
I20250111 02:56:18.125084 4450 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 560d9bddebd54e0da0bf8d2da0a7a593 is lagging by at least 1255 ops behind the committed index [suppressed 28 similar messages]
W20250111 02:56:18.160650 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 156: this message will repeat every 5th retry.
W20250111 02:56:18.671761 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 161: this message will repeat every 5th retry.
W20250111 02:56:19.215333 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 166: this message will repeat every 5th retry.
I20250111 02:56:19.308907 4284 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Bootstrap replaying log segment 1/2 (4.94M/8.00M this segment, stats: ops{read=2930 overwritten=0 applied=2928 ignored=0} inserts{seen=146300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:56:19.744434 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 171: this message will repeat every 5th retry.
W20250111 02:56:20.259595 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 176: this message will repeat every 5th retry.
I20250111 02:56:20.462234 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:20.469483 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:20.477756 1656 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250111 02:56:20.569300 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:20.623930 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:20.672186 4329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:20.723606 4181 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:20.742214 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:20.781062 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:56:20.920611 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 181: this message will repeat every 5th retry.
I20250111 02:56:21.147413 4544 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 560d9bddebd54e0da0bf8d2da0a7a593 is lagging by at least 1517 ops behind the committed index [suppressed 27 similar messages]
W20250111 02:56:21.612732 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 186: this message will repeat every 5th retry.
W20250111 02:56:22.071355 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 191: this message will repeat every 5th retry.
W20250111 02:56:22.552335 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 196: this message will repeat every 5th retry.
W20250111 02:56:23.018816 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 201: this message will repeat every 5th retry.
W20250111 02:56:23.640795 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 206: this message will repeat every 5th retry.
W20250111 02:56:24.104611 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 211: this message will repeat every 5th retry.
I20250111 02:56:24.188021 4583 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 560d9bddebd54e0da0bf8d2da0a7a593 is lagging by at least 1841 ops behind the committed index [suppressed 27 similar messages]
I20250111 02:56:24.309314 4284 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Bootstrap replaying log segment 1/2 (6.54M/8.00M this segment, stats: ops{read=3889 overwritten=0 applied=3888 ignored=0} inserts{seen=194300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:56:24.597149 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 216: this message will repeat every 5th retry.
W20250111 02:56:25.168393 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 221: this message will repeat every 5th retry.
W20250111 02:56:25.665227 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 226: this message will repeat every 5th retry.
W20250111 02:56:26.194698 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 231: this message will repeat every 5th retry.
W20250111 02:56:26.801936 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 236: this message will repeat every 5th retry.
I20250111 02:56:27.216774 4573 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 560d9bddebd54e0da0bf8d2da0a7a593 is lagging by at least 2148 ops behind the committed index [suppressed 28 similar messages]
W20250111 02:56:27.280731 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 241: this message will repeat every 5th retry.
W20250111 02:56:27.797737 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 246: this message will repeat every 5th retry.
I20250111 02:56:27.887826 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:27.911583 1656 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250111 02:56:27.939832 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:27.996244 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:28.005975 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:28.097106 4181 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:28.129565 4329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:28.180346 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:28.181133 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:28.256414 4284 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Bootstrap replayed 1/2 log segments. Stats: ops{read=4768 overwritten=0 applied=4767 ignored=0} inserts{seen=238250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20250111 02:56:28.256500 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 251: this message will repeat every 5th retry.
W20250111 02:56:28.776749 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 256: this message will repeat every 5th retry.
I20250111 02:56:29.310559 4284 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Bootstrap replaying log segment 2/2 (381.1K/3.37M this segment, stats: ops{read=4992 overwritten=0 applied=4985 ignored=0} inserts{seen=249150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:56:29.312731 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 261: this message will repeat every 5th retry.
I20250111 02:56:29.420709 1130 maintenance_manager.cc:418] P d713edd59d9544dc99edd0753427cb82: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033397
I20250111 02:56:29.514657 1129 maintenance_manager.cc:642] P d713edd59d9544dc99edd0753427cb82: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.093s user 0.023s sys 0.012s Metrics: {"bytes_written":7251,"cfile_init":1,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1179,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":13,"thread_start_us":4080,"threads_started":1,"wal-append.queue_time_us":4193}
I20250111 02:56:29.516793 1130 maintenance_manager.cc:418] P d713edd59d9544dc99edd0753427cb82: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 653 bytes on disk
I20250111 02:56:29.521092 1129 maintenance_manager.cc:642] P d713edd59d9544dc99edd0753427cb82: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.004s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":142,"lbm_reads_lt_1ms":4}
W20250111 02:56:29.736258 2390 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.19.132:41691, user_credentials={real_user=slave}} blocked reactor thread for 47073.2us
W20250111 02:56:29.833034 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 266: this message will repeat every 5th retry.
I20250111 02:56:30.302544 4595 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 560d9bddebd54e0da0bf8d2da0a7a593 is lagging by at least 2437 ops behind the committed index [suppressed 29 similar messages]
W20250111 02:56:30.337277 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 271: this message will repeat every 5th retry.
W20250111 02:56:30.892522 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 276: this message will repeat every 5th retry.
W20250111 02:56:31.420994 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 281: this message will repeat every 5th retry.
W20250111 02:56:31.887683 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 286: this message will repeat every 5th retry.
W20250111 02:56:32.469144 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 291: this message will repeat every 5th retry.
W20250111 02:56:32.985425 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 296: this message will repeat every 5th retry.
I20250111 02:56:33.313385 4643 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 560d9bddebd54e0da0bf8d2da0a7a593 is lagging by at least 2759 ops behind the committed index [suppressed 27 similar messages]
W20250111 02:56:33.571333 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 301: this message will repeat every 5th retry.
I20250111 02:56:33.876283 1722 maintenance_manager.cc:418] P 185ad035c2754007b1b3d08bb65546c2: Scheduling FlushMRSOp(a851ac2cebaa49d5b317fd02bc2766fa): perf score=0.051315
W20250111 02:56:34.190418 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 306: this message will repeat every 5th retry.
I20250111 02:56:34.315177 4284 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Bootstrap replaying log segment 2/2 (2.25M/3.37M this segment, stats: ops{read=6114 overwritten=0 applied=6114 ignored=0} inserts{seen=305600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:56:34.648762 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 311: this message will repeat every 5th retry.
W20250111 02:56:35.308888 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 316: this message will repeat every 5th retry.
I20250111 02:56:35.421495 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:35.473691 1656 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250111 02:56:35.560014 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:35.573627 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:35.638316 4181 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:35.693468 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:35.702564 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:35.711462 4329 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:35.755829 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:56:36.028800 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 321: this message will repeat every 5th retry.
I20250111 02:56:36.422029 4583 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 560d9bddebd54e0da0bf8d2da0a7a593 is lagging by at least 2995 ops behind the committed index [suppressed 25 similar messages]
W20250111 02:56:36.626227 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 326: this message will repeat every 5th retry.
W20250111 02:56:37.151325 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 331: this message will repeat every 5th retry.
W20250111 02:56:37.696590 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 336: this message will repeat every 5th retry.
W20250111 02:56:38.352780 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 341: this message will repeat every 5th retry.
W20250111 02:56:38.854426 1610 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 -> Peer 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): Couldn't send request to peer 560d9bddebd54e0da0bf8d2da0a7a593. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 346: this message will repeat every 5th retry.
I20250111 02:56:39.081442 4284 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Bootstrap replayed 2/2 log segments. Stats: ops{read=6779 overwritten=0 applied=6778 ignored=0} inserts{seen=338800 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250111 02:56:39.085103 4284 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Bootstrap complete.
I20250111 02:56:39.089887 4284 ts_tablet_manager.cc:1397] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Time spent bootstrapping tablet: real 36.258s user 26.623s sys 0.875s
I20250111 02:56:39.108749 4284 raft_consensus.cc:357] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:39.131083 4284 raft_consensus.cc:738] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 560d9bddebd54e0da0bf8d2da0a7a593, State: Initialized, Role: FOLLOWER
I20250111 02:56:39.132160 4284 consensus_queue.cc:260] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6778, Last appended: 2.6779, Last appended by leader: 6779, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:39.148358 4284 ts_tablet_manager.cc:1428] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593: Time spent starting tablet: real 0.058s user 0.026s sys 0.008s
I20250111 02:56:39.758234 4710 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 185ad035c2754007b1b3d08bb65546c2)
I20250111 02:56:39.758535 4710 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:39.763419 4710 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075), 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691)
I20250111 02:56:39.845881 4708 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 560d9bddebd54e0da0bf8d2da0a7a593 is lagging by at least 2848 ops behind the committed index [suppressed 24 similar messages]
I20250111 02:56:39.890278 3625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 7204 } } ignore_live_leader: false dest_uuid: "486f7fe875934cebb51c6717b5e4312d" is_pre_election: true
I20250111 02:56:39.901067 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 7204 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2" is_pre_election: true
I20250111 02:56:39.902057 4278 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 560d9bddebd54e0da0bf8d2da0a7a593; no voters: 185ad035c2754007b1b3d08bb65546c2, 486f7fe875934cebb51c6717b5e4312d
I20250111 02:56:39.902884 4710 raft_consensus.cc:2743] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250111 02:56:40.765219 4710 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 185ad035c2754007b1b3d08bb65546c2)
I20250111 02:56:40.765417 4710 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:40.766183 4710 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075), 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691)
I20250111 02:56:40.767254 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 7629 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2" is_pre_election: true
I20250111 02:56:40.767643 3625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 7629 } } ignore_live_leader: false dest_uuid: "486f7fe875934cebb51c6717b5e4312d" is_pre_election: true
I20250111 02:56:40.773512 4278 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 560d9bddebd54e0da0bf8d2da0a7a593; no voters: 185ad035c2754007b1b3d08bb65546c2, 486f7fe875934cebb51c6717b5e4312d
I20250111 02:56:40.773981 4710 raft_consensus.cc:2743] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250111 02:56:41.405045 4710 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 185ad035c2754007b1b3d08bb65546c2)
I20250111 02:56:41.405262 4710 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:41.406098 4710 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075), 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691)
I20250111 02:56:41.407001 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 8054 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2" is_pre_election: true
I20250111 02:56:41.408766 3625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 8054 } } ignore_live_leader: false dest_uuid: "486f7fe875934cebb51c6717b5e4312d" is_pre_election: true
I20250111 02:56:41.409435 4278 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 560d9bddebd54e0da0bf8d2da0a7a593; no voters: 185ad035c2754007b1b3d08bb65546c2, 486f7fe875934cebb51c6717b5e4312d
I20250111 02:56:41.409945 4710 raft_consensus.cc:2743] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250111 02:56:42.347385 4710 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 185ad035c2754007b1b3d08bb65546c2)
I20250111 02:56:42.347576 4710 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:42.348321 4710 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075), 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691)
I20250111 02:56:42.349390 3623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 8479 } } ignore_live_leader: false dest_uuid: "486f7fe875934cebb51c6717b5e4312d" is_pre_election: true
I20250111 02:56:42.349607 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 8479 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2" is_pre_election: true
I20250111 02:56:42.350837 4278 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 560d9bddebd54e0da0bf8d2da0a7a593; no voters: 185ad035c2754007b1b3d08bb65546c2, 486f7fe875934cebb51c6717b5e4312d
I20250111 02:56:42.351783 4710 raft_consensus.cc:2743] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250111 02:56:43.275480 4710 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 185ad035c2754007b1b3d08bb65546c2)
I20250111 02:56:43.275700 4710 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:43.276476 4710 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075), 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691)
I20250111 02:56:43.277987 3623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 8904 } } ignore_live_leader: false dest_uuid: "486f7fe875934cebb51c6717b5e4312d" is_pre_election: true
I20250111 02:56:43.279767 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 8904 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2" is_pre_election: true
I20250111 02:56:43.280462 4278 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 560d9bddebd54e0da0bf8d2da0a7a593; no voters: 185ad035c2754007b1b3d08bb65546c2, 486f7fe875934cebb51c6717b5e4312d
I20250111 02:56:43.281105 4710 raft_consensus.cc:2743] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250111 02:56:43.281595 4727 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer 560d9bddebd54e0da0bf8d2da0a7a593 is lagging by at least 1381 ops behind the committed index [suppressed 3 similar messages]
I20250111 02:56:44.012684 4752 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 185ad035c2754007b1b3d08bb65546c2)
I20250111 02:56:44.012912 4752 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:44.052783 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 9329 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2" is_pre_election: true
I20250111 02:56:44.053241 3623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 9329 } } ignore_live_leader: false dest_uuid: "486f7fe875934cebb51c6717b5e4312d" is_pre_election: true
I20250111 02:56:44.070443 4278 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 560d9bddebd54e0da0bf8d2da0a7a593; no voters: 185ad035c2754007b1b3d08bb65546c2, 486f7fe875934cebb51c6717b5e4312d
I20250111 02:56:44.071806 4752 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075), 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691)
I20250111 02:56:44.072045 4752 raft_consensus.cc:2743] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250111 02:56:44.111666 4278 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.1.19.132:41691, user_credentials={real_user=slave}} blocked reactor thread for 56348.9us
I20250111 02:56:44.262786 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:44.331496 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:44.344125 1656 tablet_service.cc:1429] Tablet server has 1 leaders and 2 scanners
I20250111 02:56:44.382942 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:44.398190 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:44.497339 4181 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:44.545751 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:44.549153 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:56:45.099252 4759 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 185ad035c2754007b1b3d08bb65546c2)
I20250111 02:56:45.099443 4759 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:45.100322 4759 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075), 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691)
I20250111 02:56:45.102545 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 9754 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2" is_pre_election: true
I20250111 02:56:45.106742 3623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 9754 } } ignore_live_leader: false dest_uuid: "486f7fe875934cebb51c6717b5e4312d" is_pre_election: true
I20250111 02:56:45.108928 4278 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 560d9bddebd54e0da0bf8d2da0a7a593; no voters: 185ad035c2754007b1b3d08bb65546c2, 486f7fe875934cebb51c6717b5e4312d
I20250111 02:56:45.109767 4329 tablet_service.cc:1429] Tablet server has 0 leaders and 1 scanners
I20250111 02:56:45.110159 4759 raft_consensus.cc:2743] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250111 02:56:45.112860 4349 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.19.132:59531 (request call id 336) took 1061 ms. Trace:
I20250111 02:56:45.113042 4349 rpcz_store.cc:276] 0111 02:56:44.050976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:56:44.055693 (+ 4717us) service_pool.cc:225] Handling call
0111 02:56:44.101221 (+ 45528us) raft_consensus.cc:1466] Updating replica for 425 ops
0111 02:56:44.254387 (+153166us) raft_consensus.cc:1529] Early marking committed up to index 0
0111 02:56:44.254392 (+ 5us) raft_consensus.cc:1534] Triggering prepare for 425 ops
0111 02:56:44.254500 (+ 108us) write_op.cc:270] Start()
0111 02:56:44.254532 (+ 32us) write_op.cc:276] Timestamp: P: 1736564186911831 usec, L: 0
0111 02:56:44.266164 (+ 11632us) write_op.cc:270] Start()
0111 02:56:44.266202 (+ 38us) write_op.cc:276] Timestamp: P: 1736564186928177 usec, L: 0
0111 02:56:44.266408 (+ 206us) write_op.cc:270] Start()
0111 02:56:44.266429 (+ 21us) write_op.cc:276] Timestamp: P: 1736564186932603 usec, L: 0
0111 02:56:44.266610 (+ 181us) write_op.cc:270] Start()
0111 02:56:44.266632 (+ 22us) write_op.cc:276] Timestamp: P: 1736564186963911 usec, L: 0
0111 02:56:44.266795 (+ 163us) write_op.cc:270] Start()
0111 02:56:44.266814 (+ 19us) write_op.cc:276] Timestamp: P: 1736564186967740 usec, L: 0
0111 02:56:44.266976 (+ 162us) write_op.cc:270] Start()
0111 02:56:44.266996 (+ 20us) write_op.cc:276] Timestamp: P: 1736564186981389 usec, L: 0
0111 02:56:44.277103 (+ 10107us) write_op.cc:270] Start()
0111 02:56:44.277154 (+ 51us) write_op.cc:276] Timestamp: P: 1736564187004168 usec, L: 0
0111 02:56:44.277351 (+ 197us) write_op.cc:270] Start()
0111 02:56:44.277374 (+ 23us) write_op.cc:276] Timestamp: P: 1736564187005337 usec, L: 0
0111 02:56:44.277596 (+ 222us) write_op.cc:270] Start()
0111 02:56:44.277619 (+ 23us) write_op.cc:276] Timestamp: P: 1736564187015042 usec, L: 0
0111 02:56:44.277772 (+ 153us) write_op.cc:270] Start()
0111 02:56:44.277793 (+ 21us) write_op.cc:276] Timestamp: P: 1736564187037046 usec, L: 0
0111 02:56:44.277929 (+ 136us) write_op.cc:270] Start()
0111 02:56:44.277947 (+ 18us) write_op.cc:276] Timestamp: P: 1736564187043978 usec, L: 0
0111 02:56:44.278094 (+ 147us) write_op.cc:270] Start()
0111 02:56:44.278131 (+ 37us) write_op.cc:276] Timestamp: P: 1736564187069135 usec, L: 0
0111 02:56:44.278264 (+ 133us) write_op.cc:270] Start()
0111 02:56:44.278282 (+ 18us) write_op.cc:276] Timestamp: P: 1736564187070271 usec, L: 0
0111 02:56:44.278408 (+ 126us) write_op.cc:270] Start()
0111 02:56:44.278425 (+ 17us) write_op.cc:276] Timestamp: P: 1736564187075921 usec, L: 0
0111 02:56:44.278548 (+ 123us) write_op.cc:270] Start()
0111 02:56:44.278566 (+ 18us) write_op.cc:276] Timestamp: P: 1736564187104700 usec, L: 0
0111 02:56:44.278698 (+ 132us) write_op.cc:270] Start()
0111 02:56:44.278716 (+ 18us) write_op.cc:276] Timestamp: P: 1736564187115347 usec, L: 0
0111 02:56:44.278849 (+ 133us) write_op.cc:270] Start()
0111 02:56:44.278876 (+ 27us) write_op.cc:276] Timestamp: P: 1736564187119216 usec, L: 0
0111 02:56:44.279041 (+ 165us) write_op.cc:270] Start()
0111 02:56:44.279061 (+ 20us) write_op.cc:276] Timestamp: P: 1736564187136490 usec, L: 0
0111 02:56:44.279198 (+ 137us) write_op.cc:270] Start()
0111 02:56:44.279217 (+ 19us) write_op.cc:276] Timestamp: P: 1736564187141106 usec, L: 0
0111 02:56:44.279380 (+ 163us) write_op.cc:270] Start()
0111 02:56:44.279400 (+ 20us) write_op.cc:276] Timestamp: P: 1736564187163354 usec, L: 0
0111 02:56:44.279532 (+ 132us) write_op.cc:270] Start()
0111 02:56:44.279551 (+ 19us) write_op.cc:276] Timestamp: P: 1736564187168394 usec, L: 0
0111 02:56:44.279693 (+ 142us) write_op.cc:270] Start()
0111 02:56:44.279711 (+ 18us) write_op.cc:276] Timestamp: P: 1736564187169816 usec, L: 0
0111 02:56:44.279838 (+ 127us) write_op.cc:270] Start()
0111 02:56:44.279867 (+ 29us) write_op.cc:276] Timestamp: P: 1736564187201804 usec, L: 0
0111 02:56:44.279993 (+ 126us) write_op.cc:270] Start()
0111 02:56:44.280009 (+ 16us) write_op.cc:276] Timestamp: P: 1736564187206112 usec, L: 0
0111 02:56:44.280137 (+ 128us) write_op.cc:270] Start()
0111 02:56:44.280154 (+ 17us) write_op.cc:276] Timestamp: P: 1736564187207732 usec, L: 0
0111 02:56:44.280258 (+ 104us) write_op.cc:270] Start()
0111 02:56:44.280273 (+ 15us) write_op.cc:276] Timestamp: P: 1736564187239586 usec, L: 0
0111 02:56:44.280389 (+ 116us) write_op.cc:270] Start()
0111 02:56:44.280406 (+ 17us) write_op.cc:276] Timestamp: P: 1736564187240759 usec, L: 0
0111 02:56:44.291136 (+ 10730us) write_op.cc:270] Start()
0111 02:56:44.291177 (+ 41us) write_op.cc:276] Timestamp: P: 1736564187277858 usec, L: 0
0111 02:56:44.291731 (+ 554us) write_op.cc:270] Start()
0111 02:56:44.291758 (+ 27us) write_op.cc:276] Timestamp: P: 1736564187281054 usec, L: 0
0111 02:56:44.291947 (+ 189us) write_op.cc:270] Start()
0111 02:56:44.291967 (+ 20us) write_op.cc:276] Timestamp: P: 1736564187320298 usec, L: 0
0111 02:56:44.292097 (+ 130us) write_op.cc:270] Start()
0111 02:56:44.292115 (+ 18us) write_op.cc:276] Timestamp: P: 1736564187336193 usec, L: 0
0111 02:56:44.292233 (+ 118us) write_op.cc:270] Start()
0111 02:56:44.292252 (+ 19us) write_op.cc:276] Timestamp: P: 1736564187363967 usec, L: 0
0111 02:56:44.292418 (+ 166us) write_op.cc:270] Start()
0111 02:56:44.293746 (+ 1328us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x619000061558. stack: 00007f4dc615cfd6 00007f4dc615cd90 00007f4dc513c48d 000000000096d544 000000000096d50d 000000000096ceda 00007f4dd2e9366a 00007f4dd2e9a74f 00007f4dd2c16fdc 00007f4dd2bcf75a 00007f4dd2d8b55e 00007f4dd2d59327 00007f4dd2d0486f 00007f4dd2d0392e 00007f4dd19c56b1 00007f4dd19e4208
0111 02:56:44.293768 (+ 22us) write_op.cc:276] Timestamp: P: 1736564187379565 usec, L: 0
0111 02:56:44.293999 (+ 231us) write_op.cc:270] Start()
0111 02:56:44.294023 (+ 24us) write_op.cc:276] Timestamp: P: 1736564187380774 usec, L: 0
0111 02:56:44.329667 (+ 35644us) write_op.cc:270] Start()
0111 02:56:44.329714 (+ 47us) write_op.cc:276] Timestamp: P: 1736564187399263 usec, L: 0
0111 02:56:44.330000 (+ 286us) write_op.cc:270] Start()
0111 02:56:44.330027 (+ 27us) write_op.cc:276] Timestamp: P: 1736564187415469 usec, L: 0
0111 02:56:44.330219 (+ 192us) write_op.cc:270] Start()
0111 02:56:44.330247 (+ 28us) write_op.cc:276] Timestamp: P: 1736564187417695 usec, L: 0
0111 02:56:44.330389 (+ 142us) write_op.cc:270] Start()
0111 02:56:44.330413 (+ 24us) write_op.cc:276] Timestamp: P: 1736564187444299 usec, L: 0
0111 02:56:44.330546 (+ 133us) write_op.cc:270] Start()
0111 02:56:44.330564 (+ 18us) write_op.cc:276] Timestamp: P: 1736564187460324 usec, L: 0
0111 02:56:44.330732 (+ 168us) write_op.cc:270] Start()
0111 02:56:44.330758 (+ 26us) write_op.cc:276] Timestamp: P: 1736564187472555 usec, L: 0
0111 02:56:44.330955 (+ 197us) write_op.cc:270] Start()
0111 02:56:44.330980 (+ 25us) write_op.cc:276] Timestamp: P: 1736564187485451 usec, L: 0
0111 02:56:44.331142 (+ 162us) write_op.cc:270] Start()
0111 02:56:44.331164 (+ 22us) write_op.cc:276] Timestamp: P: 1736564187497491 usec, L: 0
0111 02:56:44.331317 (+ 153us) write_op.cc:270] Start()
0111 02:56:44.331340 (+ 23us) write_op.cc:276] Timestamp: P: 1736564187525417 usec, L: 0
0111 02:56:44.331508 (+ 168us) write_op.cc:270] Start()
0111 02:56:44.331530 (+ 22us) write_op.cc:276] Timestamp: P: 1736564187536917 usec, L: 0
0111 02:56:44.331710 (+ 180us) write_op.cc:270] Start()
0111 02:56:44.331731 (+ 21us) write_op.cc:276] Timestamp: P: 1736564187549444 usec, L: 0
0111 02:56:44.331912 (+ 181us) write_op.cc:270] Start()
0111 02:56:44.331938 (+ 26us) write_op.cc:276] Timestamp: P: 1736564187554581 usec, L: 0
0111 02:56:44.332086 (+ 148us) write_op.cc:270] Start()
0111 02:56:44.332104 (+ 18us) write_op.cc:276] Timestamp: P: 1736564187562607 usec, L: 0
0111 02:56:44.332224 (+ 120us) write_op.cc:270] Start()
0111 02:56:44.332239 (+ 15us) write_op.cc:276] Timestamp: P: 1736564187572571 usec, L: 0
0111 02:56:44.332361 (+ 122us) write_op.cc:270] Start()
0111 02:56:44.332387 (+ 26us) write_op.cc:276] Timestamp: P: 1736564187573685 usec, L: 0
0111 02:56:44.332556 (+ 169us) write_op.cc:270] Start()
0111 02:56:44.380893 (+ 48337us) write_op.cc:276] Timestamp: P: 1736564187581498 usec, L: 0
0111 02:56:44.381154 (+ 261us) write_op.cc:270] Start()
0111 02:56:44.381188 (+ 34us) write_op.cc:276] Timestamp: P: 1736564187599936 usec, L: 0
0111 02:56:44.381373 (+ 185us) write_op.cc:270] Start()
0111 02:56:44.381394 (+ 21us) write_op.cc:276] Timestamp: P: 1736564187611761 usec, L: 0
0111 02:56:44.381542 (+ 148us) write_op.cc:270] Start()
0111 02:56:44.381562 (+ 20us) write_op.cc:276] Timestamp: P: 1736564187632867 usec, L: 0
0111 02:56:44.381733 (+ 171us) write_op.cc:270] Start()
0111 02:56:44.381757 (+ 24us) write_op.cc:276] Timestamp: P: 1736564187639862 usec, L: 0
0111 02:56:44.381908 (+ 151us) write_op.cc:270] Start()
0111 02:56:44.381929 (+ 21us) write_op.cc:276] Timestamp: P: 1736564187647108 usec, L: 0
0111 02:56:44.382088 (+ 159us) write_op.cc:270] Start()
0111 02:56:44.382111 (+ 23us) write_op.cc:276] Timestamp: P: 1736564187658035 usec, L: 0
0111 02:56:44.382249 (+ 138us) write_op.cc:270] Start()
0111 02:56:44.382289 (+ 40us) write_op.cc:276] Timestamp: P: 1736564187662974 usec, L: 0
0111 02:56:44.382444 (+ 155us) write_op.cc:270] Start()
0111 02:56:44.382464 (+ 20us) write_op.cc:276] Timestamp: P: 1736564187681047 usec, L: 0
0111 02:56:44.382655 (+ 191us) write_op.cc:270] Start()
0111 02:56:44.382678 (+ 23us) write_op.cc:276] Timestamp: P: 1736564187683843 usec, L: 0
0111 02:56:44.382872 (+ 194us) write_op.cc:270] Start()
0111 02:56:44.382897 (+ 25us) write_op.cc:276] Timestamp: P: 1736564187685247 usec, L: 0
0111 02:56:44.383043 (+ 146us) write_op.cc:270] Start()
0111 02:56:44.383067 (+ 24us) write_op.cc:276] Timestamp: P: 1736564187709726 usec, L: 0
0111 02:56:44.383231 (+ 164us) write_op.cc:270] Start()
0111 02:56:44.383255 (+ 24us) write_op.cc:276] Timestamp: P: 1736564187716784 usec, L: 0
0111 02:56:44.383404 (+ 149us) write_op.cc:270] Start()
0111 02:56:44.383423 (+ 19us) write_op.cc:276] Timestamp: P: 1736564187734876 usec, L: 0
0111 02:56:44.383548 (+ 125us) write_op.cc:270] Start()
0111 02:56:44.383565 (+ 17us) write_op.cc:276] Timestamp: P: 1736564187748217 usec, L: 0
0111 02:56:44.383733 (+ 168us) write_op.cc:270] Start()
0111 02:56:44.383751 (+ 18us) write_op.cc:276] Timestamp: P: 1736564187757672 usec, L: 0
0111 02:56:44.383876 (+ 125us) write_op.cc:270] Start()
0111 02:56:44.383893 (+ 17us) write_op.cc:276] Timestamp: P: 1736564187783764 usec, L: 0
0111 02:56:44.384015 (+ 122us) write_op.cc:270] Start()
0111 02:56:44.384033 (+ 18us) write_op.cc:276] Timestamp: P: 1736564187784930 usec, L: 0
0111 02:56:44.384167 (+ 134us) write_op.cc:270] Start()
0111 02:56:44.384189 (+ 22us) write_op.cc:276] Timestamp: P: 1736564187800759 usec, L: 0
0111 02:56:44.384334 (+ 145us) write_op.cc:270] Start()
0111 02:56:44.384356 (+ 22us) write_op.cc:276] Timestamp: P: 1736564187806725 usec, L: 0
0111 02:56:44.384502 (+ 146us) write_op.cc:270] Start()
0111 02:56:44.384529 (+ 27us) write_op.cc:276] Timestamp: P: 1736564187810981 usec, L: 0
0111 02:56:44.428409 (+ 43880us) write_op.cc:270] Start()
0111 02:56:44.428444 (+ 35us) write_op.cc:276] Timestamp: P: 1736564187859100 usec, L: 0
0111 02:56:44.428716 (+ 272us) write_op.cc:270] Start()
0111 02:56:44.428740 (+ 24us) write_op.cc:276] Timestamp: P: 1736564187921526 usec, L: 0
0111 02:56:44.431202 (+ 2462us) write_op.cc:270] Start()
0111 02:56:44.431227 (+ 25us) write_op.cc:276] Timestamp: P: 1736564187924800 usec, L: 0
0111 02:56:44.431411 (+ 184us) write_op.cc:270] Start()
0111 02:56:44.431432 (+ 21us) write_op.cc:276] Timestamp: P: 1736564187929604 usec, L: 0
0111 02:56:44.431588 (+ 156us) write_op.cc:270] Start()
0111 02:56:44.431610 (+ 22us) write_op.cc:276] Timestamp: P: 1736564187958913 usec, L: 0
0111 02:56:44.431799 (+ 189us) write_op.cc:270] Start()
0111 02:56:44.431824 (+ 25us) write_op.cc:276] Timestamp: P: 1736564187963444 usec, L: 0
0111 02:56:44.431993 (+ 169us) write_op.cc:270] Start()
0111 02:56:44.432015 (+ 22us) write_op.cc:276] Timestamp: P: 1736564187965297 usec, L: 0
0111 02:56:44.432152 (+ 137us) write_op.cc:270] Start()
0111 02:56:44.432171 (+ 19us) write_op.cc:276] Timestamp: P: 1736564187977245 usec, L: 0
0111 02:56:44.432314 (+ 143us) write_op.cc:270] Start()
0111 02:56:44.432333 (+ 19us) write_op.cc:276] Timestamp: P: 1736564188016704 usec, L: 0
0111 02:56:44.432471 (+ 138us) write_op.cc:270] Start()
0111 02:56:44.432489 (+ 18us) write_op.cc:276] Timestamp: P: 1736564188030682 usec, L: 0
0111 02:56:44.444058 (+ 11569us) write_op.cc:270] Start()
0111 02:56:44.444101 (+ 43us) write_op.cc:276] Timestamp: P: 1736564188032662 usec, L: 0
0111 02:56:44.444294 (+ 193us) write_op.cc:270] Start()
0111 02:56:44.444313 (+ 19us) write_op.cc:276] Timestamp: P: 1736564188047165 usec, L: 0
0111 02:56:44.444429 (+ 116us) write_op.cc:270] Start()
0111 02:56:44.444448 (+ 19us) write_op.cc:276] Timestamp: P: 1736564188075539 usec, L: 0
0111 02:56:44.444608 (+ 160us) write_op.cc:270] Start()
0111 02:56:44.444627 (+ 19us) write_op.cc:276] Timestamp: P: 1736564188081230 usec, L: 0
0111 02:56:44.444767 (+ 140us) write_op.cc:270] Start()
0111 02:56:44.444791 (+ 24us) write_op.cc:276] Timestamp: P: 1736564188090027 usec, L: 0
0111 02:56:44.444908 (+ 117us) write_op.cc:270] Start()
0111 02:56:44.444934 (+ 26us) write_op.cc:276] Timestamp: P: 1736564188141579 usec, L: 0
0111 02:56:44.445060 (+ 126us) write_op.cc:270] Start()
0111 02:56:44.445076 (+ 16us) write_op.cc:276] Timestamp: P: 1736564188142885 usec, L: 0
0111 02:56:44.445202 (+ 126us) write_op.cc:270] Start()
0111 02:56:44.445218 (+ 16us) write_op.cc:276] Timestamp: P: 1736564188161468 usec, L: 0
0111 02:56:44.445337 (+ 119us) write_op.cc:270] Start()
0111 02:56:44.445354 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188175620 usec, L: 0
0111 02:56:44.445480 (+ 126us) write_op.cc:270] Start()
0111 02:56:44.445497 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188196841 usec, L: 0
0111 02:56:44.445614 (+ 117us) write_op.cc:270] Start()
0111 02:56:44.445631 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188222591 usec, L: 0
0111 02:56:44.445757 (+ 126us) write_op.cc:270] Start()
0111 02:56:44.445774 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188224957 usec, L: 0
0111 02:56:44.445935 (+ 161us) write_op.cc:270] Start()
0111 02:56:44.445954 (+ 19us) write_op.cc:276] Timestamp: P: 1736564188253738 usec, L: 0
0111 02:56:44.446088 (+ 134us) write_op.cc:270] Start()
0111 02:56:44.446106 (+ 18us) write_op.cc:276] Timestamp: P: 1736564188260511 usec, L: 0
0111 02:56:44.446227 (+ 121us) write_op.cc:270] Start()
0111 02:56:44.446243 (+ 16us) write_op.cc:276] Timestamp: P: 1736564188278961 usec, L: 0
0111 02:56:44.446357 (+ 114us) write_op.cc:270] Start()
0111 02:56:44.446373 (+ 16us) write_op.cc:276] Timestamp: P: 1736564188296120 usec, L: 0
0111 02:56:44.446522 (+ 149us) write_op.cc:270] Start()
0111 02:56:44.446541 (+ 19us) write_op.cc:276] Timestamp: P: 1736564188303798 usec, L: 0
0111 02:56:44.446669 (+ 128us) write_op.cc:270] Start()
0111 02:56:44.446686 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188310092 usec, L: 0
0111 02:56:44.527576 (+ 80890us) write_op.cc:270] Start()
0111 02:56:44.527618 (+ 42us) write_op.cc:276] Timestamp: P: 1736564188323984 usec, L: 0
0111 02:56:44.528493 (+ 875us) write_op.cc:270] Start()
0111 02:56:44.528522 (+ 29us) write_op.cc:276] Timestamp: P: 1736564188331458 usec, L: 0
0111 02:56:44.529764 (+ 1242us) write_op.cc:270] Start()
0111 02:56:44.529986 (+ 222us) write_op.cc:276] Timestamp: P: 1736564188332575 usec, L: 0
0111 02:56:44.531151 (+ 1165us) write_op.cc:270] Start()
0111 02:56:44.531178 (+ 27us) write_op.cc:276] Timestamp: P: 1736564188347464 usec, L: 0
0111 02:56:44.541514 (+ 10336us) write_op.cc:270] Start()
0111 02:56:44.543931 (+ 2417us) write_op.cc:276] Timestamp: P: 1736564188357000 usec, L: 0
0111 02:56:44.546270 (+ 2339us) write_op.cc:270] Start()
0111 02:56:44.546300 (+ 30us) write_op.cc:276] Timestamp: P: 1736564188361310 usec, L: 0
0111 02:56:44.546458 (+ 158us) write_op.cc:270] Start()
0111 02:56:44.546479 (+ 21us) write_op.cc:276] Timestamp: P: 1736564188377356 usec, L: 0
0111 02:56:44.555428 (+ 8949us) write_op.cc:270] Start()
0111 02:56:44.555461 (+ 33us) write_op.cc:276] Timestamp: P: 1736564188394100 usec, L: 0
0111 02:56:44.555677 (+ 216us) write_op.cc:270] Start()
0111 02:56:44.555700 (+ 23us) write_op.cc:276] Timestamp: P: 1736564188404554 usec, L: 0
0111 02:56:44.555884 (+ 184us) write_op.cc:270] Start()
0111 02:56:44.555906 (+ 22us) write_op.cc:276] Timestamp: P: 1736564188426499 usec, L: 0
0111 02:56:44.556073 (+ 167us) write_op.cc:270] Start()
0111 02:56:44.556094 (+ 21us) write_op.cc:276] Timestamp: P: 1736564188437955 usec, L: 0
0111 02:56:44.556255 (+ 161us) write_op.cc:270] Start()
0111 02:56:44.556276 (+ 21us) write_op.cc:276] Timestamp: P: 1736564188439110 usec, L: 0
0111 02:56:44.556417 (+ 141us) write_op.cc:270] Start()
0111 02:56:44.556438 (+ 21us) write_op.cc:276] Timestamp: P: 1736564188460988 usec, L: 0
0111 02:56:44.556615 (+ 177us) write_op.cc:270] Start()
0111 02:56:44.556638 (+ 23us) write_op.cc:276] Timestamp: P: 1736564188470419 usec, L: 0
0111 02:56:44.556821 (+ 183us) write_op.cc:270] Start()
0111 02:56:44.556842 (+ 21us) write_op.cc:276] Timestamp: P: 1736564188471580 usec, L: 0
0111 02:56:44.557035 (+ 193us) write_op.cc:270] Start()
0111 02:56:44.557056 (+ 21us) write_op.cc:276] Timestamp: P: 1736564188498435 usec, L: 0
0111 02:56:44.557231 (+ 175us) write_op.cc:270] Start()
0111 02:56:44.557252 (+ 21us) write_op.cc:276] Timestamp: P: 1736564188502225 usec, L: 0
0111 02:56:44.557385 (+ 133us) write_op.cc:270] Start()
0111 02:56:44.557402 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188511924 usec, L: 0
0111 02:56:44.557525 (+ 123us) write_op.cc:270] Start()
0111 02:56:44.557542 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188515517 usec, L: 0
0111 02:56:44.557702 (+ 160us) write_op.cc:270] Start()
0111 02:56:44.557724 (+ 22us) write_op.cc:276] Timestamp: P: 1736564188531368 usec, L: 0
0111 02:56:44.557867 (+ 143us) write_op.cc:270] Start()
0111 02:56:44.557884 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188551159 usec, L: 0
0111 02:56:44.558008 (+ 124us) write_op.cc:270] Start()
0111 02:56:44.558025 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188553398 usec, L: 0
0111 02:56:44.558135 (+ 110us) write_op.cc:270] Start()
0111 02:56:44.558151 (+ 16us) write_op.cc:276] Timestamp: P: 1736564188569524 usec, L: 0
0111 02:56:44.558289 (+ 138us) write_op.cc:270] Start()
0111 02:56:44.558306 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188578624 usec, L: 0
0111 02:56:44.558429 (+ 123us) write_op.cc:270] Start()
0111 02:56:44.558447 (+ 18us) write_op.cc:276] Timestamp: P: 1736564188587250 usec, L: 0
0111 02:56:44.558568 (+ 121us) write_op.cc:270] Start()
0111 02:56:44.558585 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188590636 usec, L: 0
0111 02:56:44.558723 (+ 138us) write_op.cc:270] Start()
0111 02:56:44.558740 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188610741 usec, L: 0
0111 02:56:44.558886 (+ 146us) write_op.cc:270] Start()
0111 02:56:44.657884 (+ 98998us) spinlock_profiling.cc:243] Waited 98.8 ms on lock 0x619000061558. stack: 00007f4dc615cfd6 00007f4dc615cd90 00007f4dc513c48d 000000000096d544 000000000096d50d 000000000096ceda 00007f4dd2e9366a 00007f4dd2e9a74f 00007f4dd2c16fdc 00007f4dd2bcf75a 00007f4dd2d8b55e 00007f4dd2d59327 00007f4dd2d0486f 00007f4dd2d0392e 00007f4dd19c56b1 00007f4dd19e4208
0111 02:56:44.657911 (+ 27us) write_op.cc:276] Timestamp: P: 1736564188618810 usec, L: 0
0111 02:56:44.658175 (+ 264us) write_op.cc:270] Start()
0111 02:56:44.658203 (+ 28us) write_op.cc:276] Timestamp: P: 1736564188643791 usec, L: 0
0111 02:56:44.658497 (+ 294us) write_op.cc:270] Start()
0111 02:56:44.658528 (+ 31us) write_op.cc:276] Timestamp: P: 1736564188648676 usec, L: 0
0111 02:56:44.658714 (+ 186us) write_op.cc:270] Start()
0111 02:56:44.658737 (+ 23us) write_op.cc:276] Timestamp: P: 1736564188666108 usec, L: 0
0111 02:56:44.659099 (+ 362us) write_op.cc:270] Start()
0111 02:56:44.659128 (+ 29us) write_op.cc:276] Timestamp: P: 1736564188678995 usec, L: 0
0111 02:56:44.659329 (+ 201us) write_op.cc:270] Start()
0111 02:56:44.659352 (+ 23us) write_op.cc:276] Timestamp: P: 1736564188685662 usec, L: 0
0111 02:56:44.659493 (+ 141us) write_op.cc:270] Start()
0111 02:56:44.659522 (+ 29us) write_op.cc:276] Timestamp: P: 1736564188692878 usec, L: 0
0111 02:56:44.659657 (+ 135us) write_op.cc:270] Start()
0111 02:56:44.659676 (+ 19us) write_op.cc:276] Timestamp: P: 1736564188709586 usec, L: 0
0111 02:56:44.659806 (+ 130us) write_op.cc:270] Start()
0111 02:56:44.659825 (+ 19us) write_op.cc:276] Timestamp: P: 1736564188719047 usec, L: 0
0111 02:56:44.659949 (+ 124us) write_op.cc:270] Start()
0111 02:56:44.659970 (+ 21us) write_op.cc:276] Timestamp: P: 1736564188721703 usec, L: 0
0111 02:56:44.660114 (+ 144us) write_op.cc:270] Start()
0111 02:56:44.660135 (+ 21us) write_op.cc:276] Timestamp: P: 1736564188727511 usec, L: 0
0111 02:56:44.660286 (+ 151us) write_op.cc:270] Start()
0111 02:56:44.660306 (+ 20us) write_op.cc:276] Timestamp: P: 1736564188735055 usec, L: 0
0111 02:56:44.660686 (+ 380us) write_op.cc:270] Start()
0111 02:56:44.660710 (+ 24us) write_op.cc:276] Timestamp: P: 1736564188743906 usec, L: 0
0111 02:56:44.660859 (+ 149us) write_op.cc:270] Start()
0111 02:56:44.660882 (+ 23us) write_op.cc:276] Timestamp: P: 1736564188759850 usec, L: 0
0111 02:56:44.661045 (+ 163us) write_op.cc:270] Start()
0111 02:56:44.661064 (+ 19us) write_op.cc:276] Timestamp: P: 1736564188761006 usec, L: 0
0111 02:56:44.661193 (+ 129us) write_op.cc:270] Start()
0111 02:56:44.661211 (+ 18us) write_op.cc:276] Timestamp: P: 1736564188764294 usec, L: 0
0111 02:56:44.661331 (+ 120us) write_op.cc:270] Start()
0111 02:56:44.661350 (+ 19us) write_op.cc:276] Timestamp: P: 1736564188791316 usec, L: 0
0111 02:56:44.661474 (+ 124us) write_op.cc:270] Start()
0111 02:56:44.661493 (+ 19us) write_op.cc:276] Timestamp: P: 1736564188792773 usec, L: 0
0111 02:56:44.661654 (+ 161us) write_op.cc:270] Start()
0111 02:56:44.661675 (+ 21us) write_op.cc:276] Timestamp: P: 1736564188794401 usec, L: 0
0111 02:56:44.661827 (+ 152us) write_op.cc:270] Start()
0111 02:56:44.661847 (+ 20us) write_op.cc:276] Timestamp: P: 1736564188823722 usec, L: 0
0111 02:56:44.661990 (+ 143us) write_op.cc:270] Start()
0111 02:56:44.662007 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188824873 usec, L: 0
0111 02:56:44.662130 (+ 123us) write_op.cc:270] Start()
0111 02:56:44.662149 (+ 19us) write_op.cc:276] Timestamp: P: 1736564188843084 usec, L: 0
0111 02:56:44.662278 (+ 129us) write_op.cc:270] Start()
0111 02:56:44.662296 (+ 18us) write_op.cc:276] Timestamp: P: 1736564188851966 usec, L: 0
0111 02:56:44.662421 (+ 125us) write_op.cc:270] Start()
0111 02:56:44.662439 (+ 18us) write_op.cc:276] Timestamp: P: 1736564188853086 usec, L: 0
0111 02:56:44.662608 (+ 169us) write_op.cc:270] Start()
0111 02:56:44.662627 (+ 19us) write_op.cc:276] Timestamp: P: 1736564188872940 usec, L: 0
0111 02:56:44.662760 (+ 133us) write_op.cc:270] Start()
0111 02:56:44.662777 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188882511 usec, L: 0
0111 02:56:44.662910 (+ 133us) write_op.cc:270] Start()
0111 02:56:44.662927 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188891176 usec, L: 0
0111 02:56:44.663045 (+ 118us) write_op.cc:270] Start()
0111 02:56:44.663063 (+ 18us) write_op.cc:276] Timestamp: P: 1736564188896207 usec, L: 0
0111 02:56:44.663194 (+ 131us) write_op.cc:270] Start()
0111 02:56:44.663212 (+ 18us) write_op.cc:276] Timestamp: P: 1736564188908860 usec, L: 0
0111 02:56:44.663333 (+ 121us) write_op.cc:270] Start()
0111 02:56:44.663350 (+ 17us) write_op.cc:276] Timestamp: P: 1736564188910893 usec, L: 0
0111 02:56:44.663476 (+ 126us) write_op.cc:270] Start()
0111 02:56:44.663494 (+ 18us) write_op.cc:276] Timestamp: P: 1736564188940878 usec, L: 0
0111 02:56:44.663650 (+ 156us) write_op.cc:270] Start()
0111 02:56:44.663671 (+ 21us) write_op.cc:276] Timestamp: P: 1736564188961132 usec, L: 0
0111 02:56:44.663830 (+ 159us) write_op.cc:270] Start()
0111 02:56:44.663848 (+ 18us) write_op.cc:276] Timestamp: P: 1736564188962869 usec, L: 0
0111 02:56:44.663989 (+ 141us) write_op.cc:270] Start()
0111 02:56:44.664008 (+ 19us) write_op.cc:276] Timestamp: P: 1736564188976517 usec, L: 0
0111 02:56:44.664146 (+ 138us) write_op.cc:270] Start()
0111 02:56:44.664161 (+ 15us) write_op.cc:276] Timestamp: P: 1736564188981068 usec, L: 0
0111 02:56:44.664247 (+ 86us) write_op.cc:270] Start()
0111 02:56:44.664260 (+ 13us) write_op.cc:276] Timestamp: P: 1736564188983832 usec, L: 0
0111 02:56:44.664343 (+ 83us) write_op.cc:270] Start()
0111 02:56:44.664354 (+ 11us) write_op.cc:276] Timestamp: P: 1736564188992360 usec, L: 0
0111 02:56:44.664448 (+ 94us) write_op.cc:270] Start()
0111 02:56:44.664460 (+ 12us) write_op.cc:276] Timestamp: P: 1736564189013259 usec, L: 0
0111 02:56:44.669897 (+ 5437us) write_op.cc:270] Start()
0111 02:56:44.669934 (+ 37us) write_op.cc:276] Timestamp: P: 1736564189015004 usec, L: 0
0111 02:56:44.670219 (+ 285us) write_op.cc:270] Start()
0111 02:56:44.670248 (+ 29us) write_op.cc:276] Timestamp: P: 1736564189028216 usec, L: 0
0111 02:56:44.670428 (+ 180us) write_op.cc:270] Start()
0111 02:56:44.670468 (+ 40us) write_op.cc:276] Timestamp: P: 1736564189030058 usec, L: 0
0111 02:56:44.670896 (+ 428us) write_op.cc:270] Start()
0111 02:56:44.670920 (+ 24us) write_op.cc:276] Timestamp: P: 1736564189048194 usec, L: 0
0111 02:56:44.671124 (+ 204us) write_op.cc:270] Start()
0111 02:56:44.671149 (+ 25us) write_op.cc:276] Timestamp: P: 1736564189049810 usec, L: 0
0111 02:56:44.671307 (+ 158us) write_op.cc:270] Start()
0111 02:56:44.671329 (+ 22us) write_op.cc:276] Timestamp: P: 1736564189050886 usec, L: 0
0111 02:56:44.671485 (+ 156us) write_op.cc:270] Start()
0111 02:56:44.671505 (+ 20us) write_op.cc:276] Timestamp: P: 1736564189070259 usec, L: 0
0111 02:56:44.671649 (+ 144us) write_op.cc:270] Start()
0111 02:56:44.671669 (+ 20us) write_op.cc:276] Timestamp: P: 1736564189072066 usec, L: 0
0111 02:56:44.671792 (+ 123us) write_op.cc:270] Start()
0111 02:56:44.671810 (+ 18us) write_op.cc:276] Timestamp: P: 1736564189073172 usec, L: 0
0111 02:56:44.671941 (+ 131us) write_op.cc:270] Start()
0111 02:56:44.671960 (+ 19us) write_op.cc:276] Timestamp: P: 1736564189102819 usec, L: 0
0111 02:56:44.672088 (+ 128us) write_op.cc:270] Start()
0111 02:56:44.672105 (+ 17us) write_op.cc:276] Timestamp: P: 1736564189104097 usec, L: 0
0111 02:56:44.672217 (+ 112us) write_op.cc:270] Start()
0111 02:56:44.672233 (+ 16us) write_op.cc:276] Timestamp: P: 1736564189105267 usec, L: 0
0111 02:56:44.672418 (+ 185us) write_op.cc:270] Start()
0111 02:56:44.672441 (+ 23us) write_op.cc:276] Timestamp: P: 1736564189124433 usec, L: 0
0111 02:56:44.672662 (+ 221us) write_op.cc:270] Start()
0111 02:56:44.672684 (+ 22us) write_op.cc:276] Timestamp: P: 1736564189134615 usec, L: 0
0111 02:56:44.672842 (+ 158us) write_op.cc:270] Start()
0111 02:56:44.672863 (+ 21us) write_op.cc:276] Timestamp: P: 1736564189137107 usec, L: 0
0111 02:56:44.673022 (+ 159us) write_op.cc:270] Start()
0111 02:56:44.673046 (+ 24us) write_op.cc:276] Timestamp: P: 1736564189156613 usec, L: 0
0111 02:56:44.673208 (+ 162us) write_op.cc:270] Start()
0111 02:56:44.673234 (+ 26us) write_op.cc:276] Timestamp: P: 1736564189158175 usec, L: 0
0111 02:56:44.673431 (+ 197us) write_op.cc:270] Start()
0111 02:56:44.673456 (+ 25us) write_op.cc:276] Timestamp: P: 1736564189169430 usec, L: 0
0111 02:56:44.673627 (+ 171us) write_op.cc:270] Start()
0111 02:56:44.673646 (+ 19us) write_op.cc:276] Timestamp: P: 1736564189175684 usec, L: 0
0111 02:56:44.673773 (+ 127us) write_op.cc:270] Start()
0111 02:56:44.673792 (+ 19us) write_op.cc:276] Timestamp: P: 1736564189187293 usec, L: 0
0111 02:56:44.673928 (+ 136us) write_op.cc:270] Start()
0111 02:56:44.673945 (+ 17us) write_op.cc:276] Timestamp: P: 1736564189191583 usec, L: 0
0111 02:56:44.674067 (+ 122us) write_op.cc:270] Start()
0111 02:56:44.674084 (+ 17us) write_op.cc:276] Timestamp: P: 1736564189195099 usec, L: 0
0111 02:56:44.674209 (+ 125us) write_op.cc:270] Start()
0111 02:56:44.674225 (+ 16us) write_op.cc:276] Timestamp: P: 1736564189213850 usec, L: 0
0111 02:56:44.674364 (+ 139us) write_op.cc:270] Start()
0111 02:56:44.674400 (+ 36us) write_op.cc:276] Timestamp: P: 1736564189216250 usec, L: 0
0111 02:56:44.674576 (+ 176us) write_op.cc:270] Start()
0111 02:56:44.674598 (+ 22us) write_op.cc:276] Timestamp: P: 1736564189231063 usec, L: 0
0111 02:56:44.674740 (+ 142us) write_op.cc:270] Start()
0111 02:56:44.674760 (+ 20us) write_op.cc:276] Timestamp: P: 1736564189234023 usec, L: 0
0111 02:56:44.674904 (+ 144us) write_op.cc:270] Start()
0111 02:56:44.674924 (+ 20us) write_op.cc:276] Timestamp: P: 1736564189237010 usec, L: 0
0111 02:56:44.675063 (+ 139us) write_op.cc:270] Start()
0111 02:56:44.675081 (+ 18us) write_op.cc:276] Timestamp: P: 1736564189252616 usec, L: 0
0111 02:56:44.675260 (+ 179us) write_op.cc:270] Start()
0111 02:56:44.675285 (+ 25us) write_op.cc:276] Timestamp: P: 17
I20250111 02:56:46.277290 4759 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 185ad035c2754007b1b3d08bb65546c2)
I20250111 02:56:46.277472 4759 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:46.278188 4759 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075), 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691)
I20250111 02:56:46.278883 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 10604 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2" is_pre_election: true
I20250111 02:56:46.285857 3623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 10604 } } ignore_live_leader: false dest_uuid: "486f7fe875934cebb51c6717b5e4312d" is_pre_election: true
I20250111 02:56:46.286603 4278 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 560d9bddebd54e0da0bf8d2da0a7a593; no voters: 185ad035c2754007b1b3d08bb65546c2, 486f7fe875934cebb51c6717b5e4312d
I20250111 02:56:46.287110 4759 raft_consensus.cc:2743] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250111 02:56:47.046725 4759 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 185ad035c2754007b1b3d08bb65546c2)
I20250111 02:56:47.046942 4759 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:47.047755 4759 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075), 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691)
I20250111 02:56:47.048655 3625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 10913 } } ignore_live_leader: false dest_uuid: "486f7fe875934cebb51c6717b5e4312d" is_pre_election: true
I20250111 02:56:47.048803 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 10913 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2" is_pre_election: true
I20250111 02:56:47.049711 4278 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 560d9bddebd54e0da0bf8d2da0a7a593; no voters: 185ad035c2754007b1b3d08bb65546c2, 486f7fe875934cebb51c6717b5e4312d
I20250111 02:56:47.052911 4759 raft_consensus.cc:2743] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250111 02:56:47.195442 1133 ts_manager.cc:281] Unset tserver state for 560d9bddebd54e0da0bf8d2da0a7a593 from MAINTENANCE_MODE
I20250111 02:56:47.199582 4394 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:56:47.200142 3386 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:56:47.206740 1138 ts_manager.cc:281] Unset tserver state for 5d154c4e3a614e269ad20079b76f38d8 from MAINTENANCE_MODE
I20250111 02:56:47.339812 4247 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:56:47.490181 3673 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:56:47.570173 2679 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:56:47.800866 2818 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:56:47.803174 3525 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:56:47.896575 2958 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:56:47.924892 1721 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:56:51.049324 1102 maintenance_mode-itest.cc:745] Restarting batch of 1 tservers: 185ad035c2754007b1b3d08bb65546c2
I20250111 02:56:52.328281 1139 ts_manager.cc:292] Set tserver state for 185ad035c2754007b1b3d08bb65546c2 to MAINTENANCE_MODE
I20250111 02:56:57.100284 1656 tablet_service.cc:1422] Tablet server 185ad035c2754007b1b3d08bb65546c2 set to quiescing
I20250111 02:56:57.100582 1656 tablet_service.cc:1429] Tablet server has 1 leaders and 1 scanners
I20250111 02:56:57.101425 4865 raft_consensus.cc:988] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: : Instructing follower 486f7fe875934cebb51c6717b5e4312d to start an election
I20250111 02:56:57.101573 4865 raft_consensus.cc:1076] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 2 LEADER]: Signalling peer 486f7fe875934cebb51c6717b5e4312d to start an election
I20250111 02:56:57.101733 4875 raft_consensus.cc:988] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: : Instructing follower 560d9bddebd54e0da0bf8d2da0a7a593 to start an election
I20250111 02:56:57.101925 4875 raft_consensus.cc:1076] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 2 LEADER]: Signalling peer 560d9bddebd54e0da0bf8d2da0a7a593 to start an election
I20250111 02:56:57.102371 3624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa"
dest_uuid: "486f7fe875934cebb51c6717b5e4312d"
from {username='slave'} at 127.1.19.132:50139
I20250111 02:56:57.102669 3624 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:56:57.102797 3624 raft_consensus.cc:3054] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:56:57.105365 4349 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa"
dest_uuid: "560d9bddebd54e0da0bf8d2da0a7a593"
from {username='slave'} at 127.1.19.132:59531
I20250111 02:56:57.105633 4349 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:56:57.105731 4349 raft_consensus.cc:3054] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:56:57.115465 4349 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:57.119029 3624 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:57.119932 4348 raft_consensus.cc:1235] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 3 FOLLOWER]: Rejecting Update request from peer 185ad035c2754007b1b3d08bb65546c2 for earlier term 2. Current term is 3. Ops: []
I20250111 02:56:57.120846 4832 consensus_queue.cc:1046] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 }, Status: INVALID_TERM, Last received: 2.11496, Next index: 11497, Last known committed idx: 11495, Time since last communication: 0.000s
I20250111 02:56:57.121222 4832 raft_consensus.cc:3049] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 2 LEADER]: Stepping down as leader of term 2
I20250111 02:56:57.121327 4832 raft_consensus.cc:738] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 185ad035c2754007b1b3d08bb65546c2, State: Running, Role: LEADER
I20250111 02:56:57.121645 4832 consensus_queue.cc:260] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11496, Committed index: 11496, Last appended: 2.11497, Last appended by leader: 11497, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:57.121901 4832 logging.cc:424] LogThrottler TrackedPeer Lag: suppressed but not reported on 3 messages since previous log ~13 seconds ago
I20250111 02:56:57.121987 4832 logging.cc:424] LogThrottler TrackedPeer Lag: suppressed but not reported on 2 messages since previous log ~85 seconds ago
I20250111 02:56:57.122085 4832 raft_consensus.cc:3054] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:56:57.127589 4349 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 election: Requested vote from peers 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075), 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691)
I20250111 02:56:57.128096 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 11496 } } ignore_live_leader: true dest_uuid: "185ad035c2754007b1b3d08bb65546c2"
I20250111 02:56:57.128127 3625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 3 candidate_status { last_received { term: 2 index: 11496 } } ignore_live_leader: true dest_uuid: "486f7fe875934cebb51c6717b5e4312d"
I20250111 02:56:57.128398 1676 raft_consensus.cc:2405] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 560d9bddebd54e0da0bf8d2da0a7a593 for term 3 because replica has last-logged OpId of term: 2 index: 11497, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 11496.
I20250111 02:56:57.129328 3623 raft_consensus.cc:1235] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 3 FOLLOWER]: Rejecting Update request from peer 185ad035c2754007b1b3d08bb65546c2 for earlier term 2. Current term is 3. Ops: [2.11497-2.11497]
W20250111 02:56:57.129326 1635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
I20250111 02:56:57.128890 4278 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 560d9bddebd54e0da0bf8d2da0a7a593; no voters: 185ad035c2754007b1b3d08bb65546c2, 486f7fe875934cebb51c6717b5e4312d
I20250111 02:56:57.137197 3624 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [CANDIDATE]: Term 3 election: Requested vote from peers 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691), 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205)
W20250111 02:56:57.137107 4302 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59286: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.144834 4305 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59286: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.153391 1635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
I20250111 02:56:57.164160 4890 raft_consensus.cc:2743] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250111 02:56:57.175661 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "486f7fe875934cebb51c6717b5e4312d" candidate_term: 3 candidate_status { last_received { term: 2 index: 11496 } } ignore_live_leader: true dest_uuid: "185ad035c2754007b1b3d08bb65546c2"
I20250111 02:56:57.176030 1676 raft_consensus.cc:2405] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 486f7fe875934cebb51c6717b5e4312d for term 3 because replica has last-logged OpId of term: 2 index: 11497, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 11496.
I20250111 02:56:57.194041 4349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "486f7fe875934cebb51c6717b5e4312d" candidate_term: 3 candidate_status { last_received { term: 2 index: 11496 } } ignore_live_leader: true dest_uuid: "560d9bddebd54e0da0bf8d2da0a7a593"
I20250111 02:56:57.194450 4349 raft_consensus.cc:2388] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 486f7fe875934cebb51c6717b5e4312d in current term 3: Already voted for candidate 560d9bddebd54e0da0bf8d2da0a7a593 in this term.
I20250111 02:56:57.195003 3553 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 486f7fe875934cebb51c6717b5e4312d; no voters: 185ad035c2754007b1b3d08bb65546c2, 560d9bddebd54e0da0bf8d2da0a7a593
W20250111 02:56:57.217891 1632 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.218976 1634 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
I20250111 02:56:57.222548 4894 raft_consensus.cc:2743] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250111 02:56:57.262032 3583 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50222: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:56:57.262382 3582 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50222: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:56:57.273303 4305 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59286: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.273623 4306 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59286: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.304319 1632 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.305430 1634 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.317272 3583 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50222: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:56:57.317272 3582 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50222: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:56:57.328688 4305 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59286: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.330495 4305 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59286: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.341842 1634 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.346025 1634 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.357523 3582 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50222: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:56:57.357542 3583 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50222: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:56:57.374128 4306 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59286: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.374130 4305 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59286: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.392479 1632 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.392515 1634 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.410809 3583 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50222: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:56:57.412298 3583 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50222: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:56:57.427440 4301 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59286: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.428190 4304 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59286: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.451893 1632 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.455225 1632 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.471251 3583 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50222: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:56:57.476182 3583 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50222: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
W20250111 02:56:57.488035 4832 raft_consensus.cc:668] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: failed to trigger leader election: Illegal state: leader elections are disabled
W20250111 02:56:57.493474 4305 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59286: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.498728 4305 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59286: Illegal state: replica 560d9bddebd54e0da0bf8d2da0a7a593 is not leader of this config: current role FOLLOWER
W20250111 02:56:57.514786 1632 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
I20250111 02:56:57.527846 4890 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:56:57.528023 4890 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:57.528721 4890 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075), 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691)
W20250111 02:56:57.529194 1634 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Illegal state: replica 185ad035c2754007b1b3d08bb65546c2 is not leader of this config: current role FOLLOWER
I20250111 02:56:57.530575 3624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 4 candidate_status { last_received { term: 2 index: 11496 } } ignore_live_leader: false dest_uuid: "486f7fe875934cebb51c6717b5e4312d" is_pre_election: true
I20250111 02:56:57.530747 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 4 candidate_status { last_received { term: 2 index: 11496 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2" is_pre_election: true
I20250111 02:56:57.530874 3624 raft_consensus.cc:2463] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 560d9bddebd54e0da0bf8d2da0a7a593 in term 3.
I20250111 02:56:57.531008 1676 raft_consensus.cc:2405] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 560d9bddebd54e0da0bf8d2da0a7a593 for term 4 because replica has last-logged OpId of term: 2 index: 11497, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 11496.
I20250111 02:56:57.531811 4278 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 486f7fe875934cebb51c6717b5e4312d, 560d9bddebd54e0da0bf8d2da0a7a593; no voters: 185ad035c2754007b1b3d08bb65546c2
I20250111 02:56:57.533488 4890 raft_consensus.cc:2798] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250111 02:56:57.533588 4890 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:56:57.533653 4890 raft_consensus.cc:3054] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:56:57.536463 4890 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:57.537732 3624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 4 candidate_status { last_received { term: 2 index: 11496 } } ignore_live_leader: false dest_uuid: "486f7fe875934cebb51c6717b5e4312d"
I20250111 02:56:57.538002 3624 raft_consensus.cc:3054] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:56:57.538266 1676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" candidate_term: 4 candidate_status { last_received { term: 2 index: 11496 } } ignore_live_leader: false dest_uuid: "185ad035c2754007b1b3d08bb65546c2"
I20250111 02:56:57.538487 1676 raft_consensus.cc:3054] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:56:57.540966 3624 raft_consensus.cc:2463] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 560d9bddebd54e0da0bf8d2da0a7a593 in term 4.
I20250111 02:56:57.541630 1676 raft_consensus.cc:2405] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 560d9bddebd54e0da0bf8d2da0a7a593 for term 4 because replica has last-logged OpId of term: 2 index: 11497, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 11496.
W20250111 02:56:57.542702 3583 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50222: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
I20250111 02:56:57.543556 4278 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 486f7fe875934cebb51c6717b5e4312d, 560d9bddebd54e0da0bf8d2da0a7a593; no voters:
I20250111 02:56:57.544365 4890 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [CANDIDATE]: Term 4 election: Requested vote from peers 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075), 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691)
I20250111 02:56:57.544723 4890 raft_consensus.cc:2798] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 4 FOLLOWER]: Leader election won for term 4
I20250111 02:56:57.545053 4890 raft_consensus.cc:695] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [term 4 LEADER]: Becoming Leader. State: Replica: 560d9bddebd54e0da0bf8d2da0a7a593, State: Running, Role: LEADER
I20250111 02:56:57.545756 4890 consensus_queue.cc:237] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11495, Committed index: 11495, Last appended: 2.11496, Last appended by leader: 11496, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:56:57.551646 1138 catalog_manager.cc:5526] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 reported cstate change: term changed from 2 to 4, leader changed from 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132) to 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137). New cstate: current_term: 4 leader_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } health_report { overall_health: HEALTHY } } }
W20250111 02:56:57.553673 3583 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50222: Illegal state: replica 486f7fe875934cebb51c6717b5e4312d is not leader of this config: current role FOLLOWER
I20250111 02:56:57.629107 1676 raft_consensus.cc:1270] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 4 FOLLOWER]: Refusing update from remote peer 560d9bddebd54e0da0bf8d2da0a7a593: Log matching property violated. Preceding OpId in replica: term: 2 index: 11497. Preceding OpId from leader: term: 4 index: 11497. (term mismatch)
I20250111 02:56:57.629280 1676 pending_rounds.cc:77] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Aborting all ops after (but not including) 11496
I20250111 02:56:57.629354 1676 pending_rounds.cc:99] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Aborting uncommitted WRITE_OP operation due to leader change: 2.11497
W20250111 02:56:57.629673 1676 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48134: Aborted: Op aborted by new leader
I20250111 02:56:57.630846 4896 consensus_queue.cc:1035] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11497, Last known committed idx: 11496, Time since last communication: 0.000s
I20250111 02:56:57.639657 3624 raft_consensus.cc:1270] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d [term 4 FOLLOWER]: Refusing update from remote peer 560d9bddebd54e0da0bf8d2da0a7a593: Log matching property violated. Preceding OpId in replica: term: 2 index: 11496. Preceding OpId from leader: term: 4 index: 11497. (index mismatch)
I20250111 02:56:57.640265 4896 consensus_queue.cc:1035] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Connected to new peer: Peer: permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11497, Last known committed idx: 11496, Time since last communication: 0.000s
I20250111 02:56:59.313732 1656 tablet_service.cc:1422] Tablet server 185ad035c2754007b1b3d08bb65546c2 set to quiescing
I20250111 02:56:59.314000 1656 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:01.794134 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 1593
I20250111 02:57:01.855929 1715 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:57:02.472874 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 1593
W20250111 02:57:02.639837 4278 connection.cc:537] client connection to 127.1.19.132:41691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250111 02:57:02.640129 4278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250111 02:57:02.647969 1102 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
/tmp/dist-test-taskaChIIQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.19.132:41691
--local_ip_for_outbound_sockets=127.1.19.132
--tserver_master_addrs=127.1.19.190:40247
--webserver_port=34781
--webserver_interface=127.1.19.132
--builtin_ntp_servers=127.1.19.148:45373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250111 02:57:02.672317 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.132:41691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250111 02:57:03.118643 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.132:41691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250111 02:57:03.239718 4929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:57:03.240213 4929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250111 02:57:03.240422 4929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:57:03.252897 4929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250111 02:57:03.253212 4929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.19.132
I20250111 02:57:03.267961 4929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.19.148:45373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.19.132:41691
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.1.19.132
--webserver_port=34781
--tserver_master_addrs=127.1.19.190:40247
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.19.132
--log_dir=/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 11 Jan 2025 02:43:25 UTC on 24a791456cd2
build id 4119
ASAN enabled
I20250111 02:57:03.270303 4929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:57:03.272694 4929 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:57:03.350335 4935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:57:03.359282 4936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:57:03.369329 4939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:57:03.375463 4929 server_base.cc:1034] running on GCE node
I20250111 02:57:03.376739 4929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250111 02:57:03.388783 4929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250111 02:57:03.392238 4929 hybrid_clock.cc:648] HybridClock initialized: now 1736564223392088 us; error 112 us; skew 500 ppm
I20250111 02:57:03.393133 4929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:57:03.426091 4929 webserver.cc:458] Webserver started at http://127.1.19.132:34781/ using document root <none> and password file <none>
I20250111 02:57:03.426995 4929 fs_manager.cc:362] Metadata directory not provided
I20250111 02:57:03.427148 4929 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:57:03.460150 4929 fs_manager.cc:714] Time spent opening directory manager: real 0.029s user 0.006s sys 0.003s
I20250111 02:57:03.537218 4947 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:57:03.541955 4929 fs_manager.cc:730] Time spent opening block manager: real 0.073s user 0.007s sys 0.007s
I20250111 02:57:03.542315 4929 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data,/tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/wal
uuid: "185ad035c2754007b1b3d08bb65546c2"
format_stamp: "Formatted at 2025-01-11 02:54:31 on dist-test-slave-th66"
I20250111 02:57:03.543901 4929 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data/data
Total live blocks: 5
Total live bytes: 19664634
Total live bytes (after alignment): 19677184
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250111 02:57:03.608196 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.132:41691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250111 02:57:03.717250 4929 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:57:03.718271 4929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:57:03.718531 4929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:57:03.719818 4929 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:57:03.736198 4954 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250111 02:57:03.778484 4929 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250111 02:57:03.778657 4929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.050s user 0.000s sys 0.006s
I20250111 02:57:03.778795 4929 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250111 02:57:03.783241 4929 ts_tablet_manager.cc:610] Registered 1 tablets
I20250111 02:57:03.783440 4929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.004s
I20250111 02:57:03.783828 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap starting.
W20250111 02:57:04.066958 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.132:41691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250111 02:57:04.583355 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.132:41691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250111 02:57:04.741255 4868 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59286 (ReqId={client: 6c17f6e121da43538351c07a7b6cfcb5, seq_no=11496, attempt_no=19}) took 7174 ms. Trace:
I20250111 02:57:04.741421 4868 rpcz_store.cc:276] 0111 02:56:57.567205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:56:57.569252 (+ 2047us) service_pool.cc:225] Handling call
0111 02:57:04.741227 (+7171975us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:04.688309 (+ 0us) write_op.cc:183] PREPARE: starting on tablet a851ac2cebaa49d5b317fd02bc2766fa
0111 02:57:04.688443 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:04.688450 (+ 7us) write_op.cc:435] Acquired schema lock
0111 02:57:04.688454 (+ 4us) tablet.cc:659] Decoding operations
0111 02:57:04.691084 (+ 2630us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:04.691102 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:04.691111 (+ 9us) tablet.cc:682] Acquiring locks for 50 operations
0111 02:57:04.691626 (+ 515us) tablet.cc:698] Row locks acquired
0111 02:57:04.691629 (+ 3us) write_op.cc:260] PREPARE: finished
0111 02:57:04.691706 (+ 77us) write_op.cc:270] Start()
0111 02:57:04.691741 (+ 35us) write_op.cc:276] Timestamp: P: 1736564224691683 usec, L: 0
0111 02:57:04.691749 (+ 8us) op_driver.cc:348] REPLICATION: starting
0111 02:57:04.691982 (+ 233us) log.cc:844] Serialized 2467 byte log entry
0111 02:57:04.738404 (+ 46422us) op_driver.cc:464] REPLICATION: finished
0111 02:57:04.738530 (+ 126us) write_op.cc:301] APPLY: starting
0111 02:57:04.738552 (+ 22us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:04.738696 (+ 144us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:04.738726 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:04.740249 (+ 1523us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:04.740313 (+ 64us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0111 02:57:04.740322 (+ 9us) write_op.cc:312] APPLY: finished
0111 02:57:04.740486 (+ 164us) log.cc:844] Serialized 320 byte log entry
0111 02:57:04.740948 (+ 462us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:04.741005 (+ 57us) write_op.cc:454] Released schema lock
0111 02:57:04.741020 (+ 15us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":48,"num_ops":50,"prepare.queue_time_us":7118477,"prepare.run_cpu_time_us":1851,"prepare.run_wall_time_us":4102,"raft.queue_time_us":63,"raft.run_cpu_time_us":442,"raft.run_wall_time_us":825,"replication_time_us":46621,"spinlock_wait_cycles":1137792}]]}
I20250111 02:57:04.764392 5021 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59286 (ReqId={client: 6c17f6e121da43538351c07a7b6cfcb5, seq_no=11495, attempt_no=19}) took 7187 ms. Trace:
I20250111 02:57:04.764603 5021 rpcz_store.cc:276] 0111 02:56:57.576800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:56:57.578025 (+ 1225us) service_pool.cc:225] Handling call
0111 02:57:04.764363 (+7186338us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:04.692440 (+ 0us) write_op.cc:183] PREPARE: starting on tablet a851ac2cebaa49d5b317fd02bc2766fa
0111 02:57:04.692531 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:04.692536 (+ 5us) write_op.cc:435] Acquired schema lock
0111 02:57:04.692539 (+ 3us) tablet.cc:659] Decoding operations
0111 02:57:04.693272 (+ 733us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:04.693286 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:04.693294 (+ 8us) tablet.cc:682] Acquiring locks for 50 operations
0111 02:57:04.693821 (+ 527us) tablet.cc:698] Row locks acquired
0111 02:57:04.693823 (+ 2us) write_op.cc:260] PREPARE: finished
0111 02:57:04.693860 (+ 37us) write_op.cc:270] Start()
0111 02:57:04.693888 (+ 28us) write_op.cc:276] Timestamp: P: 1736564224693843 usec, L: 0
0111 02:57:04.693892 (+ 4us) op_driver.cc:348] REPLICATION: starting
0111 02:57:04.694061 (+ 169us) log.cc:844] Serialized 2467 byte log entry
0111 02:57:04.761792 (+ 67731us) op_driver.cc:464] REPLICATION: finished
0111 02:57:04.761988 (+ 196us) write_op.cc:301] APPLY: starting
0111 02:57:04.762014 (+ 26us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:04.762232 (+ 218us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:04.762266 (+ 34us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:04.763747 (+ 1481us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:04.763814 (+ 67us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0111 02:57:04.763823 (+ 9us) write_op.cc:312] APPLY: finished
0111 02:57:04.764052 (+ 229us) log.cc:844] Serialized 320 byte log entry
0111 02:57:04.764147 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:04.764229 (+ 82us) write_op.cc:454] Released schema lock
0111 02:57:04.764245 (+ 16us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":110,"num_ops":50,"prepare.queue_time_us":7113828,"prepare.run_cpu_time_us":1444,"prepare.run_wall_time_us":1799,"raft.queue_time_us":44,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":67867}]]}
I20250111 02:57:04.767367 5021 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:59286 (ReqId={client: 6c17f6e121da43538351c07a7b6cfcb5, seq_no=11494, attempt_no=1}) took 7130 ms. Trace:
I20250111 02:57:04.767468 5021 rpcz_store.cc:276] 0111 02:56:57.636640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0111 02:56:57.636691 (+ 51us) service_pool.cc:225] Handling call
0111 02:57:04.767351 (+7130660us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0111 02:57:04.694282 (+ 0us) write_op.cc:183] PREPARE: starting on tablet a851ac2cebaa49d5b317fd02bc2766fa
0111 02:57:04.694402 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0111 02:57:04.694410 (+ 8us) write_op.cc:435] Acquired schema lock
0111 02:57:04.694414 (+ 4us) tablet.cc:659] Decoding operations
0111 02:57:04.695232 (+ 818us) write_op.cc:620] Acquiring the partition lock for write op
0111 02:57:04.695241 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0111 02:57:04.695246 (+ 5us) tablet.cc:682] Acquiring locks for 50 operations
0111 02:57:04.695562 (+ 316us) tablet.cc:698] Row locks acquired
0111 02:57:04.695564 (+ 2us) write_op.cc:260] PREPARE: finished
0111 02:57:04.695612 (+ 48us) write_op.cc:270] Start()
0111 02:57:04.695644 (+ 32us) write_op.cc:276] Timestamp: P: 1736564224695594 usec, L: 0
0111 02:57:04.695648 (+ 4us) op_driver.cc:348] REPLICATION: starting
0111 02:57:04.695798 (+ 150us) log.cc:844] Serialized 2467 byte log entry
0111 02:57:04.761877 (+ 66079us) op_driver.cc:464] REPLICATION: finished
0111 02:57:04.765318 (+ 3441us) write_op.cc:301] APPLY: starting
0111 02:57:04.765330 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0111 02:57:04.765465 (+ 135us) tablet.cc:1367] finished BulkCheckPresence
0111 02:57:04.765487 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0111 02:57:04.766957 (+ 1470us) tablet.cc:1380] finished ApplyRowOperation cycle
0111 02:57:04.767011 (+ 54us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0111 02:57:04.767018 (+ 7us) write_op.cc:312] APPLY: finished
0111 02:57:04.767142 (+ 124us) log.cc:844] Serialized 320 byte log entry
0111 02:57:04.767209 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0111 02:57:04.767261 (+ 52us) write_op.cc:454] Released schema lock
0111 02:57:04.767270 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3404,"num_ops":50,"prepare.queue_time_us":7057109,"prepare.run_cpu_time_us":1276,"prepare.run_wall_time_us":1682,"raft.queue_time_us":35,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":66212}]]}
W20250111 02:57:05.083684 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.132:41691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20250111 02:57:05.341091 4954 log.cc:826] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Log is configured to *not* fsync() on all Append() calls
I20250111 02:57:05.464943 5069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.19.132:41691 every 8 connection(s)
I20250111 02:57:05.467590 4929 rpc_server.cc:307] RPC server started. Bound to: 127.1.19.132:41691
I20250111 02:57:05.469558 4929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data/info.pb
I20250111 02:57:05.477880 1102 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu as pid 4929
I20250111 02:57:05.584635 5070 heartbeater.cc:346] Connected to a master server at 127.1.19.190:40247
I20250111 02:57:05.584998 5070 heartbeater.cc:463] Registering TS with master...
I20250111 02:57:05.586129 5070 heartbeater.cc:510] Master 127.1.19.190:40247 requested a full tablet report, sending...
I20250111 02:57:05.588958 1138 ts_manager.cc:194] Re-registered known tserver with Master: 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691)
I20250111 02:57:05.590519 1138 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.19.132:50249
W20250111 02:57:05.637212 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
W20250111 02:57:06.154299 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20250111 02:57:06.594720 5070 heartbeater.cc:502] Master 127.1.19.190:40247 was elected leader, sending a full tablet report...
W20250111 02:57:06.646468 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20250111 02:57:07.179177 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
I20250111 02:57:07.578527 4996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:07.604310 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:57:07.651859 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
I20250111 02:57:07.667915 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:07.675379 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:07.690975 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 2 scanners
I20250111 02:57:07.713027 4329 tablet_service.cc:1429] Tablet server has 1 leaders and 1 scanners
I20250111 02:57:07.756408 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:07.785915 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:07.820147 4181 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:57:08.105619 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
W20250111 02:57:08.617642 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
W20250111 02:57:09.134646 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
W20250111 02:57:09.648913 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
I20250111 02:57:09.720681 5125 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 11 ops behind the committed index
W20250111 02:57:10.060103 2389 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.19.137:39205, user_credentials={real_user=slave}} blocked reactor thread for 86164.3us
W20250111 02:57:10.205014 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
I20250111 02:57:10.345932 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replaying log segment 1/3 (1.26M/8.00M this segment, stats: ops{read=749 overwritten=0 applied=747 ignored=0} inserts{seen=37300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:57:10.686059 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250111 02:57:11.180644 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
W20250111 02:57:11.773995 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
W20250111 02:57:12.319810 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
I20250111 02:57:12.765496 5076 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 289 ops behind the committed index [suppressed 27 similar messages]
W20250111 02:57:12.901661 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
W20250111 02:57:13.400321 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20250111 02:57:14.212289 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
W20250111 02:57:14.749622 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
I20250111 02:57:15.235720 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:15.348481 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replaying log segment 1/3 (2.53M/8.00M this segment, stats: ops{read=1499 overwritten=0 applied=1499 ignored=0} inserts{seen=74900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250111 02:57:15.412770 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:15.482375 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250111 02:57:15.485347 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:15.501935 4996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:57:15.522410 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
I20250111 02:57:15.537349 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:15.538444 4181 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:15.602511 4329 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250111 02:57:15.603540 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:15.991082 5176 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 551 ops behind the committed index [suppressed 23 similar messages]
W20250111 02:57:16.250073 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
W20250111 02:57:16.556592 2389 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.19.137:39205, user_credentials={real_user=slave}} blocked reactor thread for 98305.9us
W20250111 02:57:16.789306 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
W20250111 02:57:17.436769 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
W20250111 02:57:18.115571 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
W20250111 02:57:18.648756 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
I20250111 02:57:19.013079 5176 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 815 ops behind the committed index [suppressed 25 similar messages]
W20250111 02:57:19.251742 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
W20250111 02:57:19.778391 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 156: this message will repeat every 5th retry.
W20250111 02:57:20.309023 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 161: this message will repeat every 5th retry.
I20250111 02:57:20.351691 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replaying log segment 1/3 (4.25M/8.00M this segment, stats: ops{read=2496 overwritten=0 applied=2495 ignored=0} inserts{seen=124650 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:57:20.858515 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 166: this message will repeat every 5th retry.
W20250111 02:57:21.433974 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 171: this message will repeat every 5th retry.
W20250111 02:57:22.024340 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 176: this message will repeat every 5th retry.
I20250111 02:57:22.076141 5206 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 1088 ops behind the committed index [suppressed 26 similar messages]
W20250111 02:57:22.628500 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 181: this message will repeat every 5th retry.
I20250111 02:57:23.123944 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:23.162654 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250111 02:57:23.197949 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:57:23.212033 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 186: this message will repeat every 5th retry.
I20250111 02:57:23.226068 4996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:23.239411 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:23.283792 4329 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250111 02:57:23.329948 4181 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:23.353049 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:23.367290 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:57:23.983726 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 191: this message will repeat every 5th retry.
W20250111 02:57:24.601032 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 196: this message will repeat every 5th retry.
W20250111 02:57:25.048897 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 201: this message will repeat every 5th retry.
I20250111 02:57:25.133148 5249 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 1328 ops behind the committed index [suppressed 24 similar messages]
I20250111 02:57:25.355058 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replaying log segment 1/3 (5.88M/8.00M this segment, stats: ops{read=3435 overwritten=0 applied=3435 ignored=0} inserts{seen=171650 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:57:25.619302 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 206: this message will repeat every 5th retry.
W20250111 02:57:26.182884 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 211: this message will repeat every 5th retry.
W20250111 02:57:26.705307 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 216: this message will repeat every 5th retry.
W20250111 02:57:27.261161 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 221: this message will repeat every 5th retry.
W20250111 02:57:27.679584 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 226: this message will repeat every 5th retry.
I20250111 02:57:28.151153 5193 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 1640 ops behind the committed index [suppressed 28 similar messages]
W20250111 02:57:28.170746 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 231: this message will repeat every 5th retry.
W20250111 02:57:28.649050 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 236: this message will repeat every 5th retry.
W20250111 02:57:29.023938 2389 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.19.137:39205, user_credentials={real_user=slave}} blocked reactor thread for 79852.9us
W20250111 02:57:29.214803 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 241: this message will repeat every 5th retry.
W20250111 02:57:29.776669 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 246: this message will repeat every 5th retry.
W20250111 02:57:30.265303 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 251: this message will repeat every 5th retry.
I20250111 02:57:30.356209 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replaying log segment 1/3 (7.63M/8.00M this segment, stats: ops{read=4450 overwritten=0 applied=4449 ignored=0} inserts{seen=222350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250111 02:57:30.736728 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:30.762332 4996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:57:30.840976 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 256: this message will repeat every 5th retry.
I20250111 02:57:30.861644 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:30.900449 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:30.915438 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250111 02:57:31.000609 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:31.020843 4181 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:31.040174 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:31.043553 4329 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250111 02:57:31.167548 5206 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 1850 ops behind the committed index [suppressed 27 similar messages]
W20250111 02:57:31.563182 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 261: this message will repeat every 5th retry.
I20250111 02:57:31.988916 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replayed 1/3 log segments. Stats: ops{read=4664 overwritten=0 applied=4661 ignored=0} inserts{seen=232950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
W20250111 02:57:32.064632 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 266: this message will repeat every 5th retry.
I20250111 02:57:32.117615 3674 maintenance_manager.cc:418] P 486f7fe875934cebb51c6717b5e4312d: Scheduling FlushMRSOp(a851ac2cebaa49d5b317fd02bc2766fa): perf score=0.070694
W20250111 02:57:32.467160 2389 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.1.19.137:39205, user_credentials={real_user=slave}} blocked reactor thread for 61136.5us
W20250111 02:57:32.566655 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 271: this message will repeat every 5th retry.
W20250111 02:57:33.252612 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 276: this message will repeat every 5th retry.
W20250111 02:57:33.774670 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 281: this message will repeat every 5th retry.
I20250111 02:57:34.247905 5206 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 2100 ops behind the committed index [suppressed 26 similar messages]
W20250111 02:57:34.289620 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 286: this message will repeat every 5th retry.
W20250111 02:57:34.898561 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 291: this message will repeat every 5th retry.
I20250111 02:57:35.363253 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replaying log segment 2/3 (853.2K/8.00M this segment, stats: ops{read=5145 overwritten=0 applied=5144 ignored=0} inserts{seen=257100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:57:35.407709 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 296: this message will repeat every 5th retry.
W20250111 02:57:35.978374 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 301: this message will repeat every 5th retry.
W20250111 02:57:36.525707 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 306: this message will repeat every 5th retry.
W20250111 02:57:37.142138 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 311: this message will repeat every 5th retry.
I20250111 02:57:37.329425 5317 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 2322 ops behind the committed index [suppressed 26 similar messages]
W20250111 02:57:37.681406 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 316: this message will repeat every 5th retry.
W20250111 02:57:38.320013 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 321: this message will repeat every 5th retry.
W20250111 02:57:39.210335 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 326: this message will repeat every 5th retry.
W20250111 02:57:39.801450 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 331: this message will repeat every 5th retry.
I20250111 02:57:39.896786 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:39.975128 4996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:39.997319 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:40.014853 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250111 02:57:40.016069 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:40.034473 4329 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250111 02:57:40.115648 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:40.120846 4181 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:40.140842 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:40.367794 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replaying log segment 2/3 (1.81M/8.00M this segment, stats: ops{read=5707 overwritten=0 applied=5706 ignored=0} inserts{seen=285200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250111 02:57:40.388470 5349 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 2497 ops behind the committed index [suppressed 22 similar messages]
W20250111 02:57:40.426657 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 336: this message will repeat every 5th retry.
W20250111 02:57:40.924314 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 341: this message will repeat every 5th retry.
W20250111 02:57:41.574291 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 346: this message will repeat every 5th retry.
W20250111 02:57:42.198663 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 351: this message will repeat every 5th retry.
W20250111 02:57:42.853907 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 356: this message will repeat every 5th retry.
W20250111 02:57:43.448036 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 361: this message will repeat every 5th retry.
I20250111 02:57:43.475641 5206 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 2699 ops behind the committed index [suppressed 25 similar messages]
W20250111 02:57:43.993238 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 366: this message will repeat every 5th retry.
W20250111 02:57:44.502990 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 371: this message will repeat every 5th retry.
W20250111 02:57:45.142607 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 376: this message will repeat every 5th retry.
I20250111 02:57:45.369547 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replaying log segment 2/3 (3.10M/8.00M this segment, stats: ops{read=6452 overwritten=0 applied=6452 ignored=0} inserts{seen=322500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:57:45.696390 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 381: this message will repeat every 5th retry.
W20250111 02:57:46.343274 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 386: this message will repeat every 5th retry.
I20250111 02:57:46.578424 5398 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 2929 ops behind the committed index [suppressed 26 similar messages]
W20250111 02:57:46.930650 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 391: this message will repeat every 5th retry.
W20250111 02:57:47.963831 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 396: this message will repeat every 5th retry.
I20250111 02:57:47.988022 4996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:48.074371 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:48.149801 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:48.170675 4329 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250111 02:57:48.226347 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:48.234563 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250111 02:57:48.256760 4181 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:48.294683 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:48.302493 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:57:48.483884 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 401: this message will repeat every 5th retry.
W20250111 02:57:49.111122 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 406: this message will repeat every 5th retry.
I20250111 02:57:49.674860 5464 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 3123 ops behind the committed index [suppressed 21 similar messages]
W20250111 02:57:49.721663 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 411: this message will repeat every 5th retry.
W20250111 02:57:49.862685 2416 meta_cache.cc:1261] Time spent looking up entry by key: real 0.066s user 0.003s sys 0.000s
W20250111 02:57:50.247784 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 416: this message will repeat every 5th retry.
I20250111 02:57:50.376367 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replaying log segment 2/3 (4.43M/8.00M this segment, stats: ops{read=7225 overwritten=0 applied=7225 ignored=0} inserts{seen=361150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:57:50.706434 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 421: this message will repeat every 5th retry.
W20250111 02:57:51.290591 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 426: this message will repeat every 5th retry.
W20250111 02:57:51.843456 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 431: this message will repeat every 5th retry.
W20250111 02:57:52.386247 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 436: this message will repeat every 5th retry.
I20250111 02:57:52.744864 5464 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 3346 ops behind the committed index [suppressed 28 similar messages]
W20250111 02:57:52.898491 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 441: this message will repeat every 5th retry.
W20250111 02:57:53.423048 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 446: this message will repeat every 5th retry.
W20250111 02:57:53.923311 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 451: this message will repeat every 5th retry.
W20250111 02:57:54.409773 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 456: this message will repeat every 5th retry.
W20250111 02:57:55.064515 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 461: this message will repeat every 5th retry.
I20250111 02:57:55.378523 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replaying log segment 2/3 (5.60M/8.00M this segment, stats: ops{read=7900 overwritten=0 applied=7899 ignored=0} inserts{seen=394850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:57:55.722747 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 466: this message will repeat every 5th retry.
I20250111 02:57:55.793545 5475 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 3521 ops behind the committed index [suppressed 26 similar messages]
I20250111 02:57:56.280421 4996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:56.287503 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:56.307379 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:56.377712 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:56.378455 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:56.393172 4181 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:57:56.413818 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250111 02:57:56.468803 4329 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20250111 02:57:56.498631 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 471: this message will repeat every 5th retry.
I20250111 02:57:56.542966 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:57:56.946166 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 476: this message will repeat every 5th retry.
W20250111 02:57:57.624780 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 481: this message will repeat every 5th retry.
W20250111 02:57:58.164713 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 486: this message will repeat every 5th retry.
I20250111 02:57:58.823499 5530 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 3708 ops behind the committed index [suppressed 23 similar messages]
W20250111 02:57:58.895694 4278 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.1.19.131:37075, user_credentials={real_user=slave}} blocked reactor thread for 44043.2us
W20250111 02:57:58.982864 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 491: this message will repeat every 5th retry.
W20250111 02:57:59.541887 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 496: this message will repeat every 5th retry.
W20250111 02:58:00.236261 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 501: this message will repeat every 5th retry.
I20250111 02:58:00.385329 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replaying log segment 2/3 (7.01M/8.00M this segment, stats: ops{read=8710 overwritten=0 applied=8710 ignored=0} inserts{seen=435400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:58:00.966188 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 506: this message will repeat every 5th retry.
W20250111 02:58:01.569077 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 511: this message will repeat every 5th retry.
I20250111 02:58:01.859200 5538 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 3897 ops behind the committed index [suppressed 22 similar messages]
W20250111 02:58:02.129529 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 516: this message will repeat every 5th retry.
W20250111 02:58:02.774574 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 521: this message will repeat every 5th retry.
I20250111 02:58:03.088572 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replayed 2/3 log segments. Stats: ops{read=9285 overwritten=0 applied=9284 ignored=0} inserts{seen=464100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20250111 02:58:03.427695 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 526: this message will repeat every 5th retry.
W20250111 02:58:04.005180 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 531: this message will repeat every 5th retry.
W20250111 02:58:04.762236 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 536: this message will repeat every 5th retry.
I20250111 02:58:04.963510 5549 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 4080 ops behind the committed index [suppressed 23 similar messages]
W20250111 02:58:05.408912 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 541: this message will repeat every 5th retry.
I20250111 02:58:05.485270 2893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:58:05.537161 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replaying log segment 3/3 (583.7K/3.83M this segment, stats: ops{read=9614 overwritten=0 applied=9613 ignored=0} inserts{seen=480550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250111 02:58:05.543604 3605 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250111 02:58:05.548431 2614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:58:05.552191 4996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:58:05.571760 2753 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:58:05.613274 4329 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250111 02:58:05.664127 4181 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:58:05.776095 3319 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250111 02:58:05.815416 3457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250111 02:58:05.915972 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 546: this message will repeat every 5th retry.
W20250111 02:58:06.477666 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 551: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/maintenance_mode-itest.cc:753: Failure
Failed
Bad status: Timed out: cluster didn't become healthy: Running 'cluster ksck 127.1.19.190:40247' did not succeed in 60 seconds: Runtime error: error running 'cluster ksck 127.1.19.190:40247': stdout: Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
d713edd59d9544dc99edd0753427cb82 | 127.1.19.190:40247 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.19.148:45373 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server States
Server | State
----------------------------------+------------------
185ad035c2754007b1b3d08bb65546c2 | MAINTENANCE_MODE
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0eb58b1fa7c746f8abc51f33c199e832 | 127.1.19.133:43355 | HEALTHY | /L1 | 0 | 0
185ad035c2754007b1b3d08bb65546c2 | 127.1.19.132:41691 | HEALTHY | /L1 | 0 | 0
34d6de1ad631433e9b596bf231a8c9d5 | 127.1.19.135:41329 | HEALTHY | /L1 | 0 | 0
486f7fe875934cebb51c6717b5e4312d | 127.1.19.131:37075 | HEALTHY | /L0 | 0 | 3
51a314e6744a4a11ab63fa6e76d4f780 | 127.1.19.134:44209 | HEALTHY | /L1 | 0 | 0
560d9bddebd54e0da0bf8d2da0a7a593 | 127.1.19.137:39205 | HEALTHY | /L2 | 1 | 0
5d154c4e3a614e269ad20079b76f38d8 | 127.1.19.136:45497 | HEALTHY | /L2 | 0 | 0
b530fdca53b74ee8b97452b96b4e9d9a | 127.1.19.129:43009 | HEALTHY | /L0 | 0 | 0
ceb745fac9b04980b45a6b9fa9416bc2 | 127.1.19.130:33299 | HEALTHY | /L0 | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/L2 | 2
/L1 | 4
/L0 | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 9 server(s) checked
local_ip_for_outbound_sockets | 127.1.19.129 | experimental | 127.1.19.129:43009
local_ip_for_outbound_sockets | 127.1.19.130 | experimental | 127.1.19.130:33299
local_ip_for_outbound_sockets | 127.1.19.131 | experimental | 127.1.19.131:37075
local_ip_for_outbound_sockets | 127.1.19.132 | experimental | 127.1.19.132:41691
local_ip_for_outbound_sockets | 127.1.19.133 | experimental | 127.1.19.133:43355
local_ip_for_outbound_sockets | 127.1.19.134 | experimental | 127.1.19.134:44209
local_ip_for_outbound_sockets | 127.1.19.135 | experimental | 127.1.19.135:41329
local_ip_for_outbound_sockets | 127.1.19.136 | experimental | 127.1.19.136:45497
local_ip_for_outbound_sockets | 127.1.19.137 | experimental | 127.1.19.137:39205
never_fsync | true | unsafe,advanced | all 9 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 9 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked
server_dump_info_format | pb | hidden | all 9 server(s) checked
server_dump_info_path | /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-0/data/info.pb | hidden | 127.1.19.129:43009
server_dump_info_path | /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-1/data/info.pb | hidden | 127.1.19.130:33299
server_dump_info_path | /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/data/info.pb | hidden | 127.1.19.131:37075
server_dump_info_path | /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-3/data/info.pb | hidden | 127.1.19.132:41691
server_dump_info_path | /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-4/data/info.pb | hidden | 127.1.19.133:43355
server_dump_info_path | /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-5/data/info.pb | hidden | 127.1.19.134:44209
server_dump_info_path | /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-6/data/info.pb | hidden | 127.1.19.135:41329
server_dump_info_path | /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-7/data/info.pb | hidden | 127.1.19.136:45497
server_dump_info_path | /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-8/data/info.pb | hidden | 127.1.19.137:39205
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.19.148:45373 | all 9 server(s) checked
time_source | builtin | all 9 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------
1.18.0-SNAPSHOT | all 10 server(s) checked
Tablet Summary
Tablet a851ac2cebaa49d5b317fd02bc2766fa of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): RUNNING
185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): not running
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap replayed 2/3 log segments. Stats: ops{read=9285 overwritten=0 applied=9284 ignored=0} inserts{seen=464100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205): RUNNING [LEADER]
All reported replicas are:
A = 486f7fe875934cebb51c6717b5e4312d
B = 185ad035c2754007b1b3d08bb65546c2
C = 560d9bddebd54e0da0bf8d2da0a7a593
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B C* | | | Yes
A | A B C* | 4 | -1 | Yes
B | A B C | 4 | -1 | Yes
C | A B C* | 4 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 9
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
, stderr: W20250111 02:58:04.731452 5546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250111 02:58:04.731966 5546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250111 02:58:04.745088 5546 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Runtime error: ksck discovered errors
: /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu: process exited with non-zero status 1
W20250111 02:58:06.974553 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 556: this message will repeat every 5th retry.
I20250111 02:58:07.310216 3558 maintenance_manager.cc:642] P 486f7fe875934cebb51c6717b5e4312d: FlushMRSOp(a851ac2cebaa49d5b317fd02bc2766fa) complete. Timing: real 35.192s user 21.587s sys 0.325s Metrics: {"bytes_written":19223300,"cfile_init":1,"dirs.queue_time_us":9139,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":7966,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":138596,"lbm_writes_1-10_ms":17,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":821,"peak_mem_usage":0,"rows_written":692800,"thread_start_us":9331,"threads_started":1}
I20250111 02:58:07.312014 3674 maintenance_manager.cc:418] P 486f7fe875934cebb51c6717b5e4312d: Scheduling UndoDeltaBlockGCOp(a851ac2cebaa49d5b317fd02bc2766fa): 7988227 bytes on disk
I20250111 02:58:07.312959 3558 maintenance_manager.cc:642] P 486f7fe875934cebb51c6717b5e4312d: UndoDeltaBlockGCOp(a851ac2cebaa49d5b317fd02bc2766fa) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
W20250111 02:58:07.540876 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 561: this message will repeat every 5th retry.
I20250111 02:58:08.022604 5599 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 4205 ops behind the committed index [suppressed 27 similar messages]
W20250111 02:58:08.040963 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 566: this message will repeat every 5th retry.
W20250111 02:58:08.554597 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 571: this message will repeat every 5th retry.
W20250111 02:58:09.072800 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 576: this message will repeat every 5th retry.
W20250111 02:58:09.592640 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 581: this message will repeat every 5th retry.
W20250111 02:58:10.123661 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 586: this message will repeat every 5th retry.
I20250111 02:58:10.538480 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replaying log segment 3/3 (2.44M/3.83M this segment, stats: ops{read=10696 overwritten=0 applied=10695 ignored=0} inserts{seen=534650 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250111 02:58:10.616055 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 591: this message will repeat every 5th retry.
I20250111 02:58:11.129889 5605 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 4205 ops behind the committed index [suppressed 29 similar messages]
W20250111 02:58:11.147204 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 596: this message will repeat every 5th retry.
W20250111 02:58:11.645578 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 601: this message will repeat every 5th retry.
W20250111 02:58:12.191293 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 606: this message will repeat every 5th retry.
W20250111 02:58:12.713963 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 611: this message will repeat every 5th retry.
W20250111 02:58:13.211038 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 616: this message will repeat every 5th retry.
I20250111 02:58:13.435725 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap replayed 3/3 log segments. Stats: ops{read=11498 overwritten=1 applied=11497 ignored=0} inserts{seen=574700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:58:13.436370 4954 tablet_bootstrap.cc:492] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Bootstrap complete.
I20250111 02:58:13.441637 4954 ts_tablet_manager.cc:1397] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Time spent bootstrapping tablet: real 69.658s user 45.059s sys 1.618s
I20250111 02:58:13.446956 4954 raft_consensus.cc:357] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:58:13.447698 4954 raft_consensus.cc:738] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 185ad035c2754007b1b3d08bb65546c2, State: Initialized, Role: FOLLOWER
I20250111 02:58:13.448396 4954 consensus_queue.cc:260] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11497, Last appended: 4.11497, Last appended by leader: 11497, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:58:13.449986 4954 ts_tablet_manager.cc:1428] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20250111 02:58:13.832082 3674 maintenance_manager.cc:418] P 486f7fe875934cebb51c6717b5e4312d: Scheduling LogGCOp(a851ac2cebaa49d5b317fd02bc2766fa): free 16783002 bytes of WAL
I20250111 02:58:13.832512 3558 log_reader.cc:385] T a851ac2cebaa49d5b317fd02bc2766fa: removed 2 log segments from log reader
I20250111 02:58:13.832651 3558 log.cc:1078] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Deleting log segment in path: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/wal/wals/a851ac2cebaa49d5b317fd02bc2766fa/wal-000000001 (ops 1-4952)
I20250111 02:58:13.832770 3558 log.cc:1078] T a851ac2cebaa49d5b317fd02bc2766fa P 486f7fe875934cebb51c6717b5e4312d: Deleting log segment in path: /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0/minicluster-data/ts-2/wal/wals/a851ac2cebaa49d5b317fd02bc2766fa/wal-000000002 (ops 4953-9623)
I20250111 02:58:13.837214 3558 maintenance_manager.cc:642] P 486f7fe875934cebb51c6717b5e4312d: LogGCOp(a851ac2cebaa49d5b317fd02bc2766fa) complete. Timing: real 0.005s user 0.000s sys 0.005s Metrics: {}
I20250111 02:58:13.871568 1102 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20250111 02:58:13.871788 1102 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID b530fdca53b74ee8b97452b96b4e9d9a and pid 3254
I20250111 02:58:14.303597 5616 consensus_queue.cc:786] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 [LEADER]: Peer 185ad035c2754007b1b3d08bb65546c2 is lagging by at least 2505 ops behind the committed index [suppressed 26 similar messages]
************************ BEGIN STACKS **************************
[New LWP 3257]
[New LWP 3258]
[New LWP 3259]
[New LWP 3260]
[New LWP 3267]
[New LWP 3268]
[New LWP 3269]
[New LWP 3272]
[New LWP 3273]
[New LWP 3274]
[New LWP 3275]
[New LWP 3276]
[New LWP 3277]
[New LWP 3278]
[New LWP 3279]
[New LWP 3280]
[New LWP 3281]
[New LWP 3282]
[New LWP 3283]
[New LWP 3284]
[New LWP 3285]
[New LWP 3286]
[New LWP 3287]
[New LWP 3288]
[New LWP 3289]
[New LWP 3290]
[New LWP 3291]
[New LWP 3292]
[New LWP 3293]
[New LWP 3294]
[New LWP 3295]
[New LWP 3296]
[New LWP 3297]
[New LWP 3298]
[New LWP 3299]
[New LWP 3300]
[New LWP 3301]
[New LWP 3302]
[New LWP 3303]
[New LWP 3304]
[New LWP 3305]
[New LWP 3306]
[New LWP 3307]
[New LWP 3308]
[New LWP 3309]
[New LWP 3310]
[New LWP 3311]
[New LWP 3312]
[New LWP 3313]
[New LWP 3314]
[New LWP 3315]
[New LWP 3316]
[New LWP 3317]
[New LWP 3318]
[New LWP 3319]
[New LWP 3320]
[New LWP 3321]
[New LWP 3322]
[New LWP 3323]
[New LWP 3324]
[New LWP 3325]
[New LWP 3326]
[New LWP 3327]
[New LWP 3328]
[New LWP 3329]
[New LWP 3330]
[New LWP 3331]
[New LWP 3332]
[New LWP 3333]
[New LWP 3334]
[New LWP 3335]
[New LWP 3336]
[New LWP 3337]
[New LWP 3338]
[New LWP 3339]
[New LWP 3340]
[New LWP 3341]
[New LWP 3343]
[New LWP 3344]
[New LWP 3345]
[New LWP 3346]
[New LWP 3347]
[New LWP 3348]
[New LWP 3349]
[New LWP 3350]
[New LWP 3351]
[New LWP 3352]
[New LWP 3353]
[New LWP 3354]
[New LWP 3355]
[New LWP 3356]
[New LWP 3357]
[New LWP 3358]
[New LWP 3359]
[New LWP 3360]
[New LWP 3361]
[New LWP 3362]
[New LWP 3363]
[New LWP 3364]
[New LWP 3365]
[New LWP 3367]
[New LWP 3368]
[New LWP 3369]
[New LWP 3370]
[New LWP 3371]
[New LWP 3372]
[New LWP 3373]
[New LWP 3374]
[New LWP 3375]
[New LWP 3376]
[New LWP 3377]
[New LWP 3378]
[New LWP 3379]
[New LWP 3380]
[New LWP 3381]
[New LWP 3382]
[New LWP 3383]
[New LWP 3384]
[New LWP 3385]
[New LWP 3386]
[New LWP 3389]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007f4a28ecad50 in ?? ()
Id Target Id Frame
* 1 LWP 3254 "kudu" 0x00007f4a28ecad50 in ?? ()
2 LWP 3257 "kudu" 0x00007f4a28ec6fb9 in ?? ()
3 LWP 3258 "kudu" 0x00007f4a28ec6fb9 in ?? ()
4 LWP 3259 "kudu" 0x00007f4a28ec6fb9 in ?? ()
5 LWP 3260 "kernel-watcher-" 0x00007f4a28ec6fb9 in ?? ()
6 LWP 3267 "ntp client-3267" 0x00007f4a28eca9e2 in ?? ()
7 LWP 3268 "file cache-evic" 0x00007f4a28ec6fb9 in ?? ()
8 LWP 3269 "sq_acceptor" 0x00007f4a21c23cb9 in ?? ()
9 LWP 3272 "rpc reactor-327" 0x00007f4a21c30a47 in ?? ()
10 LWP 3273 "rpc reactor-327" 0x00007f4a21c30a47 in ?? ()
11 LWP 3274 "rpc reactor-327" 0x00007f4a21c30a47 in ?? ()
12 LWP 3275 "rpc reactor-327" 0x00007f4a21c30a47 in ?? ()
13 LWP 3276 "MaintenanceMgr " 0x00007f4a28ec6ad3 in ?? ()
14 LWP 3277 "txn-status-mana" 0x00007f4a28ec6fb9 in ?? ()
15 LWP 3278 "collect_and_rem" 0x00007f4a28ec6fb9 in ?? ()
16 LWP 3279 "tc-session-exp-" 0x00007f4a28ec6fb9 in ?? ()
17 LWP 3280 "rpc worker-3280" 0x00007f4a28ec6ad3 in ?? ()
18 LWP 3281 "rpc worker-3281" 0x00007f4a28ec6ad3 in ?? ()
19 LWP 3282 "rpc worker-3282" 0x00007f4a28ec6ad3 in ?? ()
20 LWP 3283 "rpc worker-3283" 0x00007f4a28ec6ad3 in ?? ()
21 LWP 3284 "rpc worker-3284" 0x00007f4a28ec6ad3 in ?? ()
22 LWP 3285 "rpc worker-3285" 0x00007f4a28ec6ad3 in ?? ()
23 LWP 3286 "rpc worker-3286" 0x00007f4a28ec6ad3 in ?? ()
24 LWP 3287 "rpc worker-3287" 0x00007f4a28ec6ad3 in ?? ()
25 LWP 3288 "rpc worker-3288" 0x00007f4a28ec6ad3 in ?? ()
26 LWP 3289 "rpc worker-3289" 0x00007f4a28ec6ad3 in ?? ()
27 LWP 3290 "rpc worker-3290" 0x00007f4a28ec6ad3 in ?? ()
28 LWP 3291 "rpc worker-3291" 0x00007f4a28ec6ad3 in ?? ()
29 LWP 3292 "rpc worker-3292" 0x00007f4a28ec6ad3 in ?? ()
30 LWP 3293 "rpc worker-3293" 0x00007f4a28ec6ad3 in ?? ()
31 LWP 3294 "rpc worker-3294" 0x00007f4a28ec6ad3 in ?? ()
32 LWP 3295 "rpc worker-3295" 0x00007f4a28ec6ad3 in ?? ()
33 LWP 3296 "rpc worker-3296" 0x00007f4a28ec6ad3 in ?? ()
34 LWP 3297 "rpc worker-3297" 0x00007f4a28ec6ad3 in ?? ()
35 LWP 3298 "rpc worker-3298" 0x00007f4a28ec6ad3 in ?? ()
36 LWP 3299 "rpc worker-3299" 0x00007f4a28ec6ad3 in ?? ()
37 LWP 3300 "rpc worker-3300" 0x00007f4a28ec6ad3 in ?? ()
38 LWP 3301 "rpc worker-3301" 0x00007f4a28ec6ad3 in ?? ()
39 LWP 3302 "rpc worker-3302" 0x00007f4a28ec6ad3 in ?? ()
40 LWP 3303 "rpc worker-3303" 0x00007f4a28ec6ad3 in ?? ()
41 LWP 3304 "rpc worker-3304" 0x00007f4a28ec6ad3 in ?? ()
42 LWP 3305 "rpc worker-3305" 0x00007f4a28ec6ad3 in ?? ()
43 LWP 3306 "rpc worker-3306" 0x00007f4a28ec6ad3 in ?? ()
44 LWP 3307 "rpc worker-3307" 0x00007f4a28ec6ad3 in ?? ()
45 LWP 3308 "rpc worker-3308" 0x00007f4a28ec6ad3 in ?? ()
46 LWP 3309 "rpc worker-3309" 0x00007f4a28ec6ad3 in ?? ()
47 LWP 3310 "rpc worker-3310" 0x00007f4a28ec6ad3 in ?? ()
48 LWP 3311 "rpc worker-3311" 0x00007f4a28ec6ad3 in ?? ()
49 LWP 3312 "rpc worker-3312" 0x00007f4a28ec6ad3 in ?? ()
50 LWP 3313 "rpc worker-3313" 0x00007f4a28ec6ad3 in ?? ()
51 LWP 3314 "rpc worker-3314" 0x00007f4a28ec6ad3 in ?? ()
52 LWP 3315 "rpc worker-3315" 0x00007f4a28ec6ad3 in ?? ()
53 LWP 3316 "rpc worker-3316" 0x00007f4a28ec6ad3 in ?? ()
54 LWP 3317 "rpc worker-3317" 0x00007f4a28ec6ad3 in ?? ()
55 LWP 3318 "rpc worker-3318" 0x00007f4a28ec6ad3 in ?? ()
56 LWP 3319 "rpc worker-3319" 0x00007f4a28ec6ad3 in ?? ()
57 LWP 3320 "rpc worker-3320" 0x00007f4a28ec6ad3 in ?? ()
58 LWP 3321 "rpc worker-3321" 0x00007f4a28ec6ad3 in ?? ()
59 LWP 3322 "rpc worker-3322" 0x00007f4a28ec6ad3 in ?? ()
60 LWP 3323 "rpc worker-3323" 0x00007f4a28ec6ad3 in ?? ()
61 LWP 3324 "rpc worker-3324" 0x00007f4a28ec6ad3 in ?? ()
62 LWP 3325 "rpc worker-3325" 0x00007f4a28ec6ad3 in ?? ()
63 LWP 3326 "rpc worker-3326" 0x00007f4a28ec6ad3 in ?? ()
64 LWP 3327 "rpc worker-3327" 0x00007f4a28ec6ad3 in ?? ()
65 LWP 3328 "rpc worker-3328" 0x00007f4a28ec6ad3 in ?? ()
66 LWP 3329 "rpc worker-3329" 0x00007f4a28ec6ad3 in ?? ()
67 LWP 3330 "rpc worker-3330" 0x00007f4a28ec6ad3 in ?? ()
68 LWP 3331 "rpc worker-3331" 0x00007f4a28ec6ad3 in ?? ()
69 LWP 3332 "rpc worker-3332" 0x00007f4a28ec6ad3 in ?? ()
70 LWP 3333 "rpc worker-3333" 0x00007f4a28ec6ad3 in ?? ()
71 LWP 3334 "rpc worker-3334" 0x00007f4a28ec6ad3 in ?? ()
72 LWP 3335 "rpc worker-3335" 0x00007f4a28ec6ad3 in ?? ()
73 LWP 3336 "rpc worker-3336" 0x00007f4a28ec6ad3 in ?? ()
74 LWP 3337 "rpc worker-3337" 0x00007f4a28ec6ad3 in ?? ()
75 LWP 3338 "rpc worker-3338" 0x00007f4a28ec6ad3 in ?? ()
76 LWP 3339 "rpc worker-3339" 0x00007f4a28ec6ad3 in ?? ()
77 LWP 3340 "rpc worker-3340" 0x00007f4a28ec6ad3 in ?? ()
78 LWP 3341 "rpc worker-3341" 0x00007f4a28ec6ad3 in ?? ()
79 LWP 3343 "rpc worker-3343" 0x00007f4a28ec6ad3 in ?? ()
80 LWP 3344 "rpc worker-3344" 0x00007f4a28ec6ad3 in ?? ()
81 LWP 3345 "rpc worker-3345" 0x00007f4a28ec6ad3 in ?? ()
82 LWP 3346 "rpc worker-3346" 0x00007f4a28ec6ad3 in ?? ()
83 LWP 3347 "rpc worker-3347" 0x00007f4a28ec6ad3 in ?? ()
84 LWP 3348 "rpc worker-3348" 0x00007f4a28ec6ad3 in ?? ()
85 LWP 3349 "rpc worker-3349" 0x00007f4a28ec6ad3 in ?? ()
86 LWP 3350 "rpc worker-3350" 0x00007f4a28ec6ad3 in ?? ()
87 LWP 3351 "rpc worker-3351" 0x00007f4a28ec6ad3 in ?? ()
88 LWP 3352 "rpc worker-3352" 0x00007f4a28ec6ad3 in ?? ()
89 LWP 3353 "rpc worker-3353" 0x00007f4a28ec6ad3 in ?? ()
90 LWP 3354 "rpc worker-3354" 0x00007f4a28ec6ad3 in ?? ()
91 LWP 3355 "rpc worker-3355" 0x00007f4a28ec6ad3 in ?? ()
92 LWP 3356 "rpc worker-3356" 0x00007f4a28ec6ad3 in ?? ()
93 LWP 3357 "rpc worker-3357" 0x00007f4a28ec6ad3 in ?? ()
94 LWP 3358 "rpc worker-3358" 0x00007f4a28ec6ad3 in ?? ()
95 LWP 3359 "rpc worker-3359" 0x00007f4a28ec6ad3 in ?? ()
96 LWP 3360 "rpc worker-3360" 0x00007f4a28ec6ad3 in ?? ()
97 LWP 3361 "rpc worker-3361" 0x00007f4a28ec6ad3 in ?? ()
98 LWP 3362 "rpc worker-3362" 0x00007f4a28ec6ad3 in ?? ()
99 LWP 3363 "rpc worker-3363" 0x00007f4a28ec6ad3 in ?? ()
100 LWP 3364 "rpc worker-3364" 0x00007f4a28ec6ad3 in ?? ()
101 LWP 3365 "rpc worker-3365" 0x00007f4a28ec6ad3 in ?? ()
102 LWP 3367 "rpc worker-3367" 0x00007f4a28ec6ad3 in ?? ()
103 LWP 3368 "rpc worker-3368" 0x00007f4a28ec6ad3 in ?? ()
104 LWP 3369 "rpc worker-3369" 0x00007f4a28ec6ad3 in ?? ()
105 LWP 3370 "rpc worker-3370" 0x00007f4a28ec6ad3 in ?? ()
106 LWP 3371 "rpc worker-3371" 0x00007f4a28ec6ad3 in ?? ()
107 LWP 3372 "rpc worker-3372" 0x00007f4a28ec6ad3 in ?? ()
108 LWP 3373 "rpc worker-3373" 0x00007f4a28ec6ad3 in ?? ()
109 LWP 3374 "rpc worker-3374" 0x00007f4a28ec6ad3 in ?? ()
110 LWP 3375 "rpc worker-3375" 0x00007f4a28ec6ad3 in ?? ()
111 LWP 3376 "rpc worker-3376" 0x00007f4a28ec6ad3 in ?? ()
112 LWP 3377 "rpc worker-3377" 0x00007f4a28ec6ad3 in ?? ()
113 LWP 3378 "rpc worker-3378" 0x00007f4a28ec6ad3 in ?? ()
114 LWP 3379 "rpc worker-3379" 0x00007f4a28ec6ad3 in ?? ()
115 LWP 3380 "rpc worker-3380" 0x00007f4a28ec6ad3 in ?? ()
116 LWP 3381 "rpc worker-3381" 0x00007f4a28ec6ad3 in ?? ()
117 LWP 3382 "diag-logger-338" 0x00007f4a28ec6fb9 in ?? ()
118 LWP 3383 "result-tracker-" 0x00007f4a28ec6fb9 in ?? ()
119 LWP 3384 "excess-log-dele" 0x00007f4a28ec6fb9 in ?? ()
120 LWP 3385 "acceptor-3385" 0x00007f4a21c320c7 in ?? ()
121 LWP 3386 "heartbeat-3386" 0x00007f4a28ec6fb9 in ?? ()
122 LWP 3389 "maintenance_sch" 0x00007f4a28ec6fb9 in ?? ()
Thread 122 (LWP 3389):
#0 0x00007f4a28ec6fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000002dc in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007f49de7aa550 in ?? ()
#6 0x00000000000005b8 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 3386):
#0 0x00007f4a28ec6fb9 in ?? ()
#1 0xabd77187b7907576 in ?? ()
#2 0x00000000000000b6 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007f49defc25b0 in ?? ()
#6 0x000000000000016c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 3385):
#0 0x00007f4a21c320c7 in ?? ()
#1 0x00006140000218a8 in ?? ()
#2 0x00007f49df7e9d50 in ?? ()
#3 0x00007f49df7e9d80 in ?? ()
#4 0x00007f49df7e9e80 in ?? ()
#5 0x00007f49df7e9d70 in ?? ()
#6 0x00007f49df7e9de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007f4a25abc620 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 3384):
#0 0x00007f4a28ec6fb9 in ?? ()
#1 0x00007f49e0003f00 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffde4337950 in ?? ()
#5 0x00007f49e0003ef0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 3383):
#0 0x00007f4a28ec6fb9 in ?? ()
#1 0x00007f49e081c0c0 in ?? ()
#2 0x00000000000000b7 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007f49e081c0b0 in ?? ()
#6 0x000000000000016e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 3382):
#0 0x00007f4a28ec6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 3381):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x000000000000002b in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007f49e184be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f49e184be70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007f4a28ec6770 in ?? ()
#10 0x00007f49e184be70 in ?? ()
#11 0x00000fe93c3097d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 3380):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000019 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007f49e2063e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f49e2063e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007f4a28ec6770 in ?? ()
#10 0x00007f49e2063e70 in ?? ()
#11 0x00000fe93c40c7d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 3379):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 3378):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 3377):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 3376):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 3375):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 3374):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 3373):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 3372):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 3371):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 3370):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 3369):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 3368):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 3367):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 3365):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 3364):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 3363):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 3362):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 3361):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 3360):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 3359):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 3358):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 3357):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 3356):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 3355):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 3354):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 3353):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 3352):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 3351):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 3350):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 3349):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 3348):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 3347):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 3346):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 3345):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 3344):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 3343):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 3341):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 3340):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 3339):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007f49f5c17e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f49f5c17e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007f4a28ec6770 in ?? ()
#10 0x00007f49f5c17e70 in ?? ()
#11 0x00000fe93eb82fd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 3338):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 3337):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 3336):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 3335):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 3334):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 3333):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 3332):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 3331):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 3330):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 3329):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 3328):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 3327):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 3326):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 3325):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 3324):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 3323):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 3322):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 3321):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 3320):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 3319):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007f49ff5e5e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f49ff5e5e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007f4a28ec6770 in ?? ()
#10 0x00007f49ff5e5e70 in ?? ()
#11 0x00000fe93febcbd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 3318):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 3317):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 3316):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 3315):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 3314):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 3313):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 3312):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 3311):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 3310):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 3309):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 3308):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 3307):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 3306):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 3305):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 3304):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 3303):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 3302):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 3301):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 3300):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 3299):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007f4a097c6e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4a097c6e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007f4a28ec6770 in ?? ()
#10 0x00007f4a097c6e70 in ?? ()
#11 0x00000fe9412f8dd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 3298):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 3297):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 3296):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 3295):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 3294):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 3293):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 3292):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 3291):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 3290):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 3289):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 3288):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 3287):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 3286):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 3285):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 3284):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 3283):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 3282):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 3281):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 3280):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 3279):
#0 0x00007f4a28ec6fb9 in ?? ()
#1 0x00007f4a139adc80 in ?? ()
#2 0x0000000000000012 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001b360 in ?? ()
#5 0x00007f4a139adc70 in ?? ()
#6 0x0000000000000024 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 3278):
#0 0x00007f4a28ec6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 3277):
#0 0x00007f4a28ec6fb9 in ?? ()
#1 0x00007f4a149af1c0 in ?? ()
#2 0x0000000000000012 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007f4a149af1b0 in ?? ()
#6 0x0000000000000024 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 3276):
#0 0x00007f4a28ec6ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 3275):
#0 0x00007f4a21c30a47 in ?? ()
#1 0x00007f4a159b22e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007f4a159b22d0 in ?? ()
#4 0x00007f4a159b24e0 in ?? ()
#5 0x00007f4a159b2320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007f4a159b23a0 in ?? ()
#8 0x00007f4a2429025d in ?? ()
#9 0x3fb962f05878f000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000078000000 in ?? ()
#14 0x00007f4a00000000 in ?? ()
#15 0x41d9e077162c110c in ?? ()
#16 0x00000fe9c2b2e474 in ?? ()
#17 0x00007f4a159b2380 in ?? ()
#18 0x00007f4a24294ba3 in ?? ()
#19 0x00007f4a159b2350 in ?? ()
#20 0x3fb962f05878f000 in ?? ()
#21 0x00000000159b23a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb962f05878f000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 3274):
#0 0x00007f4a21c30a47 in ?? ()
#1 0x00007f4a161b32e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007f4a161b32d0 in ?? ()
#4 0x00007f4a161b34e0 in ?? ()
#5 0x00007f4a161b3320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007f4a161b33a0 in ?? ()
#8 0x00007f4a2429025d in ?? ()
#9 0x3fb958c071df0000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x0000000024800000 in ?? ()
#14 0x00007f4a00000000 in ?? ()
#15 0x41d9e077162c110b in ?? ()
#16 0x00000fe9c2c2e674 in ?? ()
#17 0x00007f4a161b3380 in ?? ()
#18 0x00007f4a24294ba3 in ?? ()
#19 0x00007f4a161b3350 in ?? ()
#20 0x3fb958c071df0000 in ?? ()
#21 0x00000000161b33a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb958c071df0000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 3273):
#0 0x00007f4a21c30a47 in ?? ()
#1 0x00007f4a189ad2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007f4a189ad2d0 in ?? ()
#4 0x00007f4a189ad4e0 in ?? ()
#5 0x00007f4a189ad320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007f4a189ad3a0 in ?? ()
#8 0x00007f4a2429025d in ?? ()
#9 0x3fb953fd3e98f000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x00000000d2000000 in ?? ()
#14 0x00007f4a00000000 in ?? ()
#15 0x41d9e077162c1109 in ?? ()
#16 0x00000fe9c312da74 in ?? ()
#17 0x00007f4a189ad380 in ?? ()
#18 0x00007f4a24294ba3 in ?? ()
#19 0x00007f4a189ad350 in ?? ()
#20 0x3fb953fd3e98f000 in ?? ()
#21 0x00000000189ad3a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb953fd3e98f000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 3272):
#0 0x00007f4a21c30a47 in ?? ()
#1 0x00007f4a169b42e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007f4a169b42d0 in ?? ()
#4 0x00007f4a169b44e0 in ?? ()
#5 0x00007f4a169b4320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f4a169b43a0 in ?? ()
#8 0x00007f4a2429025d in ?? ()
#9 0x3fb971f3d687b000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x00000000cc000000 in ?? ()
#14 0x00007f4a00000000 in ?? ()
#15 0x41d9e077162c110c in ?? ()
#16 0x00000fe9c2d2e874 in ?? ()
#17 0x00007f4a169b4380 in ?? ()
#18 0x00007f4a24294ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 3269):
#0 0x00007f4a21c23cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f4a1a3b7788 in ?? ()
#3 0x0000602000017cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 3268):
#0 0x00007f4a28ec6fb9 in ?? ()
#1 0x00007f4a19bb6080 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007f4a19bb6070 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 3267):
#0 0x00007f4a28eca9e2 in ?? ()
#1 0x00007f4a193b3b60 in ?? ()
#2 0x00007f4a193b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f4a193b3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007f4a193b3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007f4a193b3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007f4a2e7eef0c in ?? ()
#12 0x00007f4a2e7de7b9 in ?? ()
#13 0x00007f4a2e7e2da6 in ?? ()
#14 0x00007f4a2e7e77e0 in ?? ()
#15 0x00007f4a2e7e7609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007f4a25add0be in ?? ()
#18 0x00007f4a28ec06db in ?? ()
#19 0x00000fe9c326e67c in ?? ()
#20 0x00007f4a193b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 3260):
#0 0x00007f4a28ec6fb9 in ?? ()
#1 0x00007f4a1b3b8c40 in ?? ()
#2 0x0000000000000398 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007f4a1b3b8c30 in ?? ()
#6 0x0000000000000730 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 3259):
#0 0x00007f4a28ec6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 3258):
#0 0x00007f4a28ec6fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 3257):
#0 0x00007f4a28ec6fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f4a1cbbc410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 3254):
#0 0x00007f4a28ecad50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250111 02:58:15.250357 1102 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID ceb745fac9b04980b45a6b9fa9416bc2 and pid 3393
************************ BEGIN STACKS **************************
I20250111 02:58:15.872936 5632 raft_consensus.cc:491] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 560d9bddebd54e0da0bf8d2da0a7a593)
I20250111 02:58:15.873287 5632 raft_consensus.cc:513] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "486f7fe875934cebb51c6717b5e4312d" member_type: VOTER last_known_addr { host: "127.1.19.131" port: 37075 } } peers { permanent_uuid: "185ad035c2754007b1b3d08bb65546c2" member_type: VOTER last_known_addr { host: "127.1.19.132" port: 41691 } } peers { permanent_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" member_type: VOTER last_known_addr { host: "127.1.19.137" port: 39205 } }
I20250111 02:58:15.875514 5632 leader_election.cc:290] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075), 560d9bddebd54e0da0bf8d2da0a7a593 (127.1.19.137:39205)
I20250111 02:58:15.908834 4349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "185ad035c2754007b1b3d08bb65546c2" candidate_term: 5 candidate_status { last_received { term: 4 index: 16127 } } ignore_live_leader: false dest_uuid: "560d9bddebd54e0da0bf8d2da0a7a593" is_pre_election: true
I20250111 02:58:15.927217 3624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a851ac2cebaa49d5b317fd02bc2766fa" candidate_uuid: "185ad035c2754007b1b3d08bb65546c2" candidate_term: 5 candidate_status { last_received { term: 4 index: 16127 } } ignore_live_leader: false dest_uuid: "486f7fe875934cebb51c6717b5e4312d" is_pre_election: true
I20250111 02:58:15.928186 4950 leader_election.cc:304] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 185ad035c2754007b1b3d08bb65546c2; no voters: 486f7fe875934cebb51c6717b5e4312d, 560d9bddebd54e0da0bf8d2da0a7a593
I20250111 02:58:15.928741 5632 raft_consensus.cc:2743] T a851ac2cebaa49d5b317fd02bc2766fa P 185ad035c2754007b1b3d08bb65546c2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
[New LWP 3395]
[New LWP 3396]
[New LWP 3397]
[New LWP 3399]
[New LWP 3405]
[New LWP 3406]
[New LWP 3407]
[New LWP 3410]
[New LWP 3411]
[New LWP 3412]
[New LWP 3413]
[New LWP 3414]
[New LWP 3415]
[New LWP 3416]
[New LWP 3417]
[New LWP 3418]
[New LWP 3419]
[New LWP 3420]
[New LWP 3421]
[New LWP 3422]
[New LWP 3423]
[New LWP 3424]
[New LWP 3425]
[New LWP 3426]
[New LWP 3427]
[New LWP 3428]
[New LWP 3429]
[New LWP 3430]
[New LWP 3431]
[New LWP 3432]
[New LWP 3433]
[New LWP 3434]
[New LWP 3435]
[New LWP 3436]
[New LWP 3437]
[New LWP 3438]
[New LWP 3439]
[New LWP 3440]
[New LWP 3441]
[New LWP 3442]
[New LWP 3443]
[New LWP 3444]
[New LWP 3445]
[New LWP 3446]
[New LWP 3447]
[New LWP 3448]
[New LWP 3449]
[New LWP 3450]
[New LWP 3451]
[New LWP 3452]
[New LWP 3453]
[New LWP 3454]
[New LWP 3455]
[New LWP 3456]
[New LWP 3457]
[New LWP 3458]
[New LWP 3459]
[New LWP 3460]
[New LWP 3461]
[New LWP 3462]
[New LWP 3463]
[New LWP 3464]
[New LWP 3465]
[New LWP 3466]
[New LWP 3467]
[New LWP 3468]
[New LWP 3469]
[New LWP 3470]
[New LWP 3471]
[New LWP 3472]
[New LWP 3473]
[New LWP 3474]
[New LWP 3475]
[New LWP 3476]
[New LWP 3477]
[New LWP 3479]
[New LWP 3480]
[New LWP 3481]
[New LWP 3482]
[New LWP 3483]
[New LWP 3484]
[New LWP 3485]
[New LWP 3486]
[New LWP 3487]
[New LWP 3488]
[New LWP 3489]
[New LWP 3490]
[New LWP 3491]
[New LWP 3492]
[New LWP 3493]
[New LWP 3495]
[New LWP 3496]
[New LWP 3497]
[New LWP 3498]
[New LWP 3499]
[New LWP 3500]
[New LWP 3501]
[New LWP 3502]
[New LWP 3503]
[New LWP 3504]
[New LWP 3505]
[New LWP 3507]
[New LWP 3508]
[New LWP 3509]
[New LWP 3510]
[New LWP 3511]
[New LWP 3512]
[New LWP 3513]
[New LWP 3514]
[New LWP 3515]
[New LWP 3516]
[New LWP 3517]
[New LWP 3518]
[New LWP 3519]
[New LWP 3520]
[New LWP 3521]
[New LWP 3522]
[New LWP 3523]
[New LWP 3524]
[New LWP 3525]
[New LWP 3526]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007f445939cd50 in ?? ()
Id Target Id Frame
* 1 LWP 3393 "kudu" 0x00007f445939cd50 in ?? ()
2 LWP 3395 "kudu" 0x00007f4459398fb9 in ?? ()
3 LWP 3396 "kudu" 0x00007f4459398fb9 in ?? ()
4 LWP 3397 "kudu" 0x00007f4459398fb9 in ?? ()
5 LWP 3399 "kernel-watcher-" 0x00007f4459398fb9 in ?? ()
6 LWP 3405 "ntp client-3405" 0x00007f445939c9e2 in ?? ()
7 LWP 3406 "file cache-evic" 0x00007f4459398fb9 in ?? ()
8 LWP 3407 "sq_acceptor" 0x00007f44520f5cb9 in ?? ()
9 LWP 3410 "rpc reactor-341" 0x00007f4452102a47 in ?? ()
10 LWP 3411 "rpc reactor-341" 0x00007f4452102a47 in ?? ()
11 LWP 3412 "rpc reactor-341" 0x00007f4452102a47 in ?? ()
12 LWP 3413 "rpc reactor-341" 0x00007f4452102a47 in ?? ()
13 LWP 3414 "MaintenanceMgr " 0x00007f4459398ad3 in ?? ()
14 LWP 3415 "txn-status-mana" 0x00007f4459398fb9 in ?? ()
15 LWP 3416 "collect_and_rem" 0x00007f4459398fb9 in ?? ()
16 LWP 3417 "tc-session-exp-" 0x00007f4459398fb9 in ?? ()
17 LWP 3418 "rpc worker-3418" 0x00007f4459398ad3 in ?? ()
18 LWP 3419 "rpc worker-3419" 0x00007f4459398ad3 in ?? ()
19 LWP 3420 "rpc worker-3420" 0x00007f4459398ad3 in ?? ()
20 LWP 3421 "rpc worker-3421" 0x00007f4459398ad3 in ?? ()
21 LWP 3422 "rpc worker-3422" 0x00007f4459398ad3 in ?? ()
22 LWP 3423 "rpc worker-3423" 0x00007f4459398ad3 in ?? ()
23 LWP 3424 "rpc worker-3424" 0x00007f4459398ad3 in ?? ()
24 LWP 3425 "rpc worker-3425" 0x00007f4459398ad3 in ?? ()
25 LWP 3426 "rpc worker-3426" 0x00007f4459398ad3 in ?? ()
26 LWP 3427 "rpc worker-3427" 0x00007f4459398ad3 in ?? ()
27 LWP 3428 "rpc worker-3428" 0x00007f4459398ad3 in ?? ()
28 LWP 3429 "rpc worker-3429" 0x00007f4459398ad3 in ?? ()
29 LWP 3430 "rpc worker-3430" 0x00007f4459398ad3 in ?? ()
30 LWP 3431 "rpc worker-3431" 0x00007f4459398ad3 in ?? ()
31 LWP 3432 "rpc worker-3432" 0x00007f4459398ad3 in ?? ()
32 LWP 3433 "rpc worker-3433" 0x00007f4459398ad3 in ?? ()
33 LWP 3434 "rpc worker-3434" 0x00007f4459398ad3 in ?? ()
34 LWP 3435 "rpc worker-3435" 0x00007f4459398ad3 in ?? ()
35 LWP 3436 "rpc worker-3436" 0x00007f4459398ad3 in ?? ()
36 LWP 3437 "rpc worker-3437" 0x00007f4459398ad3 in ?? ()
37 LWP 3438 "rpc worker-3438" 0x00007f4459398ad3 in ?? ()
38 LWP 3439 "rpc worker-3439" 0x00007f4459398ad3 in ?? ()
39 LWP 3440 "rpc worker-3440" 0x00007f4459398ad3 in ?? ()
40 LWP 3441 "rpc worker-3441" 0x00007f4459398ad3 in ?? ()
41 LWP 3442 "rpc worker-3442" 0x00007f4459398ad3 in ?? ()
42 LWP 3443 "rpc worker-3443" 0x00007f4459398ad3 in ?? ()
43 LWP 3444 "rpc worker-3444" 0x00007f4459398ad3 in ?? ()
44 LWP 3445 "rpc worker-3445" 0x00007f4459398ad3 in ?? ()
45 LWP 3446 "rpc worker-3446" 0x00007f4459398ad3 in ?? ()
46 LWP 3447 "rpc worker-3447" 0x00007f4459398ad3 in ?? ()
47 LWP 3448 "rpc worker-3448" 0x00007f4459398ad3 in ?? ()
48 LWP 3449 "rpc worker-3449" 0x00007f4459398ad3 in ?? ()
49 LWP 3450 "rpc worker-3450" 0x00007f4459398ad3 in ?? ()
50 LWP 3451 "rpc worker-3451" 0x00007f4459398ad3 in ?? ()
51 LWP 3452 "rpc worker-3452" 0x00007f4459398ad3 in ?? ()
52 LWP 3453 "rpc worker-3453" 0x00007f4459398ad3 in ?? ()
53 LWP 3454 "rpc worker-3454" 0x00007f4459398ad3 in ?? ()
54 LWP 3455 "rpc worker-3455" 0x00007f4459398ad3 in ?? ()
55 LWP 3456 "rpc worker-3456" 0x00007f4459398ad3 in ?? ()
56 LWP 3457 "rpc worker-3457" 0x00007f4459398ad3 in ?? ()
57 LWP 3458 "rpc worker-3458" 0x00007f4459398ad3 in ?? ()
58 LWP 3459 "rpc worker-3459" 0x00007f4459398ad3 in ?? ()
59 LWP 3460 "rpc worker-3460" 0x00007f4459398ad3 in ?? ()
60 LWP 3461 "rpc worker-3461" 0x00007f4459398ad3 in ?? ()
61 LWP 3462 "rpc worker-3462" 0x00007f4459398ad3 in ?? ()
62 LWP 3463 "rpc worker-3463" 0x00007f4459398ad3 in ?? ()
63 LWP 3464 "rpc worker-3464" 0x00007f4459398ad3 in ?? ()
64 LWP 3465 "rpc worker-3465" 0x00007f4459398ad3 in ?? ()
65 LWP 3466 "rpc worker-3466" 0x00007f4459398ad3 in ?? ()
66 LWP 3467 "rpc worker-3467" 0x00007f4459398ad3 in ?? ()
67 LWP 3468 "rpc worker-3468" 0x00007f4459398ad3 in ?? ()
68 LWP 3469 "rpc worker-3469" 0x00007f4459398ad3 in ?? ()
69 LWP 3470 "rpc worker-3470" 0x00007f4459398ad3 in ?? ()
70 LWP 3471 "rpc worker-3471" 0x00007f4459398ad3 in ?? ()
71 LWP 3472 "rpc worker-3472" 0x00007f4459398ad3 in ?? ()
72 LWP 3473 "rpc worker-3473" 0x00007f4459398ad3 in ?? ()
73 LWP 3474 "rpc worker-3474" 0x00007f4459398ad3 in ?? ()
74 LWP 3475 "rpc worker-3475" 0x00007f4459398ad3 in ?? ()
75 LWP 3476 "rpc worker-3476" 0x00007f4459398ad3 in ?? ()
76 LWP 3477 "rpc worker-3477" 0x00007f4459398ad3 in ?? ()
77 LWP 3479 "rpc worker-3479" 0x00007f4459398ad3 in ?? ()
78 LWP 3480 "rpc worker-3480" 0x00007f4459398ad3 in ?? ()
79 LWP 3481 "rpc worker-3481" 0x00007f4459398ad3 in ?? ()
80 LWP 3482 "rpc worker-3482" 0x00007f4459398ad3 in ?? ()
81 LWP 3483 "rpc worker-3483" 0x00007f4459398ad3 in ?? ()
82 LWP 3484 "rpc worker-3484" 0x00007f4459398ad3 in ?? ()
83 LWP 3485 "rpc worker-3485" 0x00007f4459398ad3 in ?? ()
84 LWP 3486 "rpc worker-3486" 0x00007f4459398ad3 in ?? ()
85 LWP 3487 "rpc worker-3487" 0x00007f4459398ad3 in ?? ()
86 LWP 3488 "rpc worker-3488" 0x00007f4459398ad3 in ?? ()
87 LWP 3489 "rpc worker-3489" 0x00007f4459398ad3 in ?? ()
88 LWP 3490 "rpc worker-3490" 0x00007f4459398ad3 in ?? ()
89 LWP 3491 "rpc worker-3491" 0x00007f4459398ad3 in ?? ()
90 LWP 3492 "rpc worker-3492" 0x00007f4459398ad3 in ?? ()
91 LWP 3493 "rpc worker-3493" 0x00007f4459398ad3 in ?? ()
92 LWP 3495 "rpc worker-3495" 0x00007f4459398ad3 in ?? ()
93 LWP 3496 "rpc worker-3496" 0x00007f4459398ad3 in ?? ()
94 LWP 3497 "rpc worker-3497" 0x00007f4459398ad3 in ?? ()
95 LWP 3498 "rpc worker-3498" 0x00007f4459398ad3 in ?? ()
96 LWP 3499 "rpc worker-3499" 0x00007f4459398ad3 in ?? ()
97 LWP 3500 "rpc worker-3500" 0x00007f4459398ad3 in ?? ()
98 LWP 3501 "rpc worker-3501" 0x00007f4459398ad3 in ?? ()
99 LWP 3502 "rpc worker-3502" 0x00007f4459398ad3 in ?? ()
100 LWP 3503 "rpc worker-3503" 0x00007f4459398ad3 in ?? ()
101 LWP 3504 "rpc worker-3504" 0x00007f4459398ad3 in ?? ()
102 LWP 3505 "rpc worker-3505" 0x00007f4459398ad3 in ?? ()
103 LWP 3507 "rpc worker-3507" 0x00007f4459398ad3 in ?? ()
104 LWP 3508 "rpc worker-3508" 0x00007f4459398ad3 in ?? ()
105 LWP 3509 "rpc worker-3509" 0x00007f4459398ad3 in ?? ()
106 LWP 3510 "rpc worker-3510" 0x00007f4459398ad3 in ?? ()
107 LWP 3511 "rpc worker-3511" 0x00007f4459398ad3 in ?? ()
108 LWP 3512 "rpc worker-3512" 0x00007f4459398ad3 in ?? ()
109 LWP 3513 "rpc worker-3513" 0x00007f4459398ad3 in ?? ()
110 LWP 3514 "rpc worker-3514" 0x00007f4459398ad3 in ?? ()
111 LWP 3515 "rpc worker-3515" 0x00007f4459398ad3 in ?? ()
112 LWP 3516 "rpc worker-3516" 0x00007f4459398ad3 in ?? ()
113 LWP 3517 "rpc worker-3517" 0x00007f4459398ad3 in ?? ()
114 LWP 3518 "rpc worker-3518" 0x00007f4459398ad3 in ?? ()
115 LWP 3519 "rpc worker-3519" 0x00007f4459398ad3 in ?? ()
116 LWP 3520 "rpc worker-3520" 0x00007f4459398ad3 in ?? ()
117 LWP 3521 "diag-logger-352" 0x00007f4459398fb9 in ?? ()
118 LWP 3522 "result-tracker-" 0x00007f4459398fb9 in ?? ()
119 LWP 3523 "excess-log-dele" 0x00007f4459398fb9 in ?? ()
120 LWP 3524 "acceptor-3524" 0x00007f44521040c7 in ?? ()
121 LWP 3525 "heartbeat-3525" 0x00007f4459398fb9 in ?? ()
122 LWP 3526 "maintenance_sch" 0x00007f4459398fb9 in ?? ()
Thread 122 (LWP 3526):
#0 0x00007f4459398fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000002d7 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007f440ec7c550 in ?? ()
#6 0x00000000000005ae in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 3525):
#0 0x00007f4459398fb9 in ?? ()
#1 0xabd77187b7907576 in ?? ()
#2 0x00000000000000b5 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007f440f4945b0 in ?? ()
#6 0x000000000000016a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 3524):
#0 0x00007f44521040c7 in ?? ()
#1 0x0000614000021ca8 in ?? ()
#2 0x00007f440fcbbd50 in ?? ()
#3 0x00007f440fcbbd80 in ?? ()
#4 0x00007f440fcbbe80 in ?? ()
#5 0x00007f440fcbbd70 in ?? ()
#6 0x00007f440fcbbde0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007f4455f8e620 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 3523):
#0 0x00007f4459398fb9 in ?? ()
#1 0x00007f44104d5f00 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fffe68f0e10 in ?? ()
#5 0x00007f44104d5ef0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 3522):
#0 0x00007f4459398fb9 in ?? ()
#1 0x00007f4410cee0c0 in ?? ()
#2 0x00000000000000b6 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007f4410cee0b0 in ?? ()
#6 0x000000000000016c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 3521):
#0 0x00007f4459398fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 3520):
#0 0x00007f4459398ad3 in ?? ()
#1 0x000000000000001d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007f4411d1de50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4411d1de70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007f4459398770 in ?? ()
#10 0x00007f4411d1de70 in ?? ()
#11 0x00000fe8823a3bd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 3519):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000027 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007f4412535e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4412535e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007f4459398770 in ?? ()
#10 0x00007f4412535e70 in ?? ()
#11 0x00000fe8824a6bd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 3518):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 3517):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 3516):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 3515):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 3514):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 3513):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 3512):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 3511):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 3510):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 3509):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 3508):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 3507):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 3505):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 3504):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 3503):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 3502):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 3501):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 3500):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 3499):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 3498):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 3497):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 3496):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 3495):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 3493):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 3492):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 3491):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 3490):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 3489):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 3488):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 3487):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 3486):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 3485):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 3484):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 3483):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 3482):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 3481):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 3480):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 3479):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 3477):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007f44260eee50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f44260eee70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007f4459398770 in ?? ()
#10 0x00007f44260eee70 in ?? ()
#11 0x00000fe884c1ddd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 3476):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 3475):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 3474):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 3473):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 3472):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 3471):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 3470):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 3469):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 3468):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 3467):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 3466):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 3465):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 3464):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 3463):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 3462):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 3461):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 3460):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 3459):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 3458):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 3457):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007f442fab7e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f442fab7e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007f4459398770 in ?? ()
#10 0x00007f442fab7e70 in ?? ()
#11 0x00000fe885f56fd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 3456):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 3455):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 3454):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 3453):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 3452):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 3451):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 3450):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 3449):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 3448):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 3447):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 3446):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 3445):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 3444):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 3443):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 3442):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 3441):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 3440):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 3439):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 3438):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 3437):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007f4439c98e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4439c98e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007f4459398770 in ?? ()
#10 0x00007f4439c98e70 in ?? ()
#11 0x00000fe8873931d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 3436):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 3435):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 3434):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 3433):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 3432):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 3431):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 3430):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 3429):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 3428):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 3427):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 3426):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 3425):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 3424):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 3423):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 3422):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 3421):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 3420):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 3419):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 3418):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 3417):
#0 0x00007f4459398fb9 in ?? ()
#1 0x00007f4443cc3c80 in ?? ()
#2 0x0000000000000012 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001b360 in ?? ()
#5 0x00007f4443cc3c70 in ?? ()
#6 0x0000000000000024 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 3416):
#0 0x00007f4459398fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 3415):
#0 0x00007f4459398fb9 in ?? ()
#1 0x00007f4444cc51c0 in ?? ()
#2 0x0000000000000012 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007f4444cc51b0 in ?? ()
#6 0x0000000000000024 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 3414):
#0 0x00007f4459398ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 3413):
#0 0x00007f4452102a47 in ?? ()
#1 0x00007f4445cc82e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007f4445cc82d0 in ?? ()
#4 0x00007f4445cc84e0 in ?? ()
#5 0x00007f4445cc8320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007f4445cc83a0 in ?? ()
#8 0x00007f445476225d in ?? ()
#9 0x3fb972390f907000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x00000000fc000000 in ?? ()
#14 0x00007f4400000000 in ?? ()
#15 0x41d9e077162c110b in ?? ()
#16 0x00000fe908b91074 in ?? ()
#17 0x00007f4445cc8380 in ?? ()
#18 0x00007f4454766ba3 in ?? ()
#19 0x00007f4445cc8350 in ?? ()
#20 0x3fb972390f907000 in ?? ()
#21 0x0000000045cc83a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb972390f907000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 3412):
#0 0x00007f4452102a47 in ?? ()
#1 0x00007f44464c92e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007f44464c92d0 in ?? ()
#4 0x00007f44464c94e0 in ?? ()
#5 0x00007f44464c9320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007f44464c93a0 in ?? ()
#8 0x00007f445476225d in ?? ()
#9 0x3fb98bb218a1f000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x00000000f3000000 in ?? ()
#14 0x00007f4400000000 in ?? ()
#15 0x41d9e077162c110a in ?? ()
#16 0x00000fe908c91274 in ?? ()
#17 0x00007f44464c9380 in ?? ()
#18 0x00007f4454766ba3 in ?? ()
#19 0x00007f44464c9350 in ?? ()
#20 0x3fb98bb218a1f000 in ?? ()
#21 0x00000000464c93a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb98bb218a1f000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 3411):
#0 0x00007f4452102a47 in ?? ()
#1 0x00007f4446cca2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007f4446cca2d0 in ?? ()
#4 0x00007f4446cca4e0 in ?? ()
#5 0x00007f4446cca320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007f4446cca3a0 in ?? ()
#8 0x00007f445476225d in ?? ()
#9 0x3fb9745de4d45000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000082800000 in ?? ()
#14 0x00007f4400000000 in ?? ()
#15 0x41d9e077162c110b in ?? ()
#16 0x00000fe908d91474 in ?? ()
#17 0x00007f4446cca380 in ?? ()
#18 0x00007f4454766ba3 in ?? ()
#19 0x00007f4446cca350 in ?? ()
#20 0x3fb9745de4d45000 in ?? ()
#21 0x0000000046cca3a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb9745de4d45000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 3410):
#0 0x00007f4452102a47 in ?? ()
#1 0x00007f44488b32e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007f44488b32d0 in ?? ()
#4 0x00007f44488b34e0 in ?? ()
#5 0x00007f44488b3320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f44488b33a0 in ?? ()
#8 0x00007f445476225d in ?? ()
#9 0x3fb964ee5b2ea000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x000000003c000000 in ?? ()
#14 0x00007f4400000000 in ?? ()
#15 0x41d9e077162c110c in ?? ()
#16 0x00000fe90910e674 in ?? ()
#17 0x00007f44488b3380 in ?? ()
#18 0x00007f4454766ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 3407):
#0 0x00007f44520f5cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f444a8b7788 in ?? ()
#3 0x0000602000015a10 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 3406):
#0 0x00007f4459398fb9 in ?? ()
#1 0x00007f444a0b6080 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007f444a0b6070 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 3405):
#0 0x00007f445939c9e2 in ?? ()
#1 0x00007f44490b2b60 in ?? ()
#2 0x00007f44490b2c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f44490b2bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007f44490b2c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007f44490b2c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007f445ecc0f0c in ?? ()
#12 0x00007f445ecb07b9 in ?? ()
#13 0x00007f445ecb4da6 in ?? ()
#14 0x00007f445ecb97e0 in ?? ()
#15 0x00007f445ecb9609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007f4455faf0be in ?? ()
#18 0x00007f44593926db in ?? ()
#19 0x00000fe90920e47c in ?? ()
#20 0x00007f44490b2400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 3399):
#0 0x00007f4459398fb9 in ?? ()
#1 0x00007f444b8b8c40 in ?? ()
#2 0x0000000000000391 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007f444b8b8c30 in ?? ()
#6 0x0000000000000722 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 3397):
#0 0x00007f4459398fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 3396):
#0 0x00007f4459398fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 3395):
#0 0x00007f4459398fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f444d0bc410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 3393):
#0 0x00007f445939cd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250111 02:58:16.589187 1102 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID 486f7fe875934cebb51c6717b5e4312d and pid 3535
************************ BEGIN STACKS **************************
[New LWP 3538]
[New LWP 3539]
[New LWP 3540]
[New LWP 3541]
[New LWP 3547]
[New LWP 3548]
[New LWP 3549]
[New LWP 3552]
[New LWP 3553]
[New LWP 3554]
[New LWP 3555]
[New LWP 3558]
[New LWP 3559]
[New LWP 3561]
[New LWP 3562]
[New LWP 3563]
[New LWP 3564]
[New LWP 3565]
[New LWP 3566]
[New LWP 3567]
[New LWP 3568]
[New LWP 3569]
[New LWP 3570]
[New LWP 3571]
[New LWP 3573]
[New LWP 3574]
[New LWP 3575]
[New LWP 3576]
[New LWP 3577]
[New LWP 3578]
[New LWP 3579]
[New LWP 3580]
[New LWP 3581]
[New LWP 3582]
[New LWP 3583]
[New LWP 3584]
[New LWP 3585]
[New LWP 3586]
[New LWP 3587]
[New LWP 3588]
[New LWP 3589]
[New LWP 3590]
[New LWP 3591]
[New LWP 3592]
[New LWP 3594]
[New LWP 3595]
[New LWP 3597]
[New LWP 3598]
[New LWP 3599]
[New LWP 3600]
[New LWP 3601]
[New LWP 3602]
[New LWP 3603]
[New LWP 3604]
[New LWP 3605]
[New LWP 3606]
[New LWP 3607]
[New LWP 3608]
[New LWP 3609]
[New LWP 3610]
[New LWP 3611]
[New LWP 3612]
[New LWP 3613]
[New LWP 3614]
[New LWP 3615]
[New LWP 3616]
[New LWP 3617]
[New LWP 3618]
[New LWP 3619]
[New LWP 3620]
[New LWP 3621]
[New LWP 3622]
[New LWP 3623]
[New LWP 3624]
[New LWP 3625]
[New LWP 3626]
[New LWP 3627]
[New LWP 3628]
[New LWP 3629]
[New LWP 3630]
[New LWP 3631]
[New LWP 3632]
[New LWP 3633]
[New LWP 3637]
[New LWP 3638]
[New LWP 3639]
[New LWP 3640]
[New LWP 3641]
[New LWP 3642]
[New LWP 3643]
[New LWP 3644]
[New LWP 3645]
[New LWP 3646]
[New LWP 3647]
[New LWP 3648]
[New LWP 3649]
[New LWP 3650]
[New LWP 3651]
[New LWP 3652]
[New LWP 3653]
[New LWP 3654]
[New LWP 3655]
[New LWP 3656]
[New LWP 3657]
[New LWP 3658]
[New LWP 3659]
[New LWP 3660]
[New LWP 3661]
[New LWP 3662]
[New LWP 3663]
[New LWP 3664]
[New LWP 3665]
[New LWP 3666]
[New LWP 3667]
[New LWP 3668]
[New LWP 3669]
[New LWP 3670]
[New LWP 3671]
[New LWP 3672]
[New LWP 3673]
[New LWP 3674]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007fcf56637d50 in ?? ()
Id Target Id Frame
* 1 LWP 3535 "kudu" 0x00007fcf56637d50 in ?? ()
2 LWP 3538 "kudu" 0x00007fcf56633fb9 in ?? ()
3 LWP 3539 "kudu" 0x00007fcf56633fb9 in ?? ()
4 LWP 3540 "kudu" 0x00007fcf56633fb9 in ?? ()
5 LWP 3541 "kernel-watcher-" 0x00007fcf56633fb9 in ?? ()
6 LWP 3547 "ntp client-3547" 0x00007fcf566379e2 in ?? ()
7 LWP 3548 "file cache-evic" 0x00007fcf56633fb9 in ?? ()
8 LWP 3549 "sq_acceptor" 0x00007fcf4f390cb9 in ?? ()
9 LWP 3552 "rpc reactor-355" 0x00007fcf4f39da47 in ?? ()
10 LWP 3553 "rpc reactor-355" 0x00007fcf4f39da47 in ?? ()
11 LWP 3554 "rpc reactor-355" 0x00007fcf4f39da47 in ?? ()
12 LWP 3555 "rpc reactor-355" 0x00007fcf4f39da47 in ?? ()
13 LWP 3558 "MaintenanceMgr " 0x00007fcf56633ad3 in ?? ()
14 LWP 3559 "txn-status-mana" 0x00007fcf56633fb9 in ?? ()
15 LWP 3561 "collect_and_rem" 0x00007fcf56633fb9 in ?? ()
16 LWP 3562 "tc-session-exp-" 0x00007fcf56633fb9 in ?? ()
17 LWP 3563 "rpc worker-3563" 0x00007fcf56633ad3 in ?? ()
18 LWP 3564 "rpc worker-3564" 0x00007fcf56633ad3 in ?? ()
19 LWP 3565 "rpc worker-3565" 0x00007fcf56633ad3 in ?? ()
20 LWP 3566 "rpc worker-3566" 0x00007fcf56633ad3 in ?? ()
21 LWP 3567 "rpc worker-3567" 0x00007fcf56633ad3 in ?? ()
22 LWP 3568 "rpc worker-3568" 0x00007fcf56633ad3 in ?? ()
23 LWP 3569 "rpc worker-3569" 0x00007fcf56633ad3 in ?? ()
24 LWP 3570 "rpc worker-3570" 0x00007fcf56633ad3 in ?? ()
25 LWP 3571 "rpc worker-3571" 0x00007fcf56633ad3 in ?? ()
26 LWP 3573 "rpc worker-3573" 0x00007fcf56633ad3 in ?? ()
27 LWP 3574 "rpc worker-3574" 0x00007fcf56633ad3 in ?? ()
28 LWP 3575 "rpc worker-3575" 0x00007fcf56633ad3 in ?? ()
29 LWP 3576 "rpc worker-3576" 0x00007fcf56633ad3 in ?? ()
30 LWP 3577 "rpc worker-3577" 0x00007fcf56633ad3 in ?? ()
31 LWP 3578 "rpc worker-3578" 0x00007fcf56633ad3 in ?? ()
32 LWP 3579 "rpc worker-3579" 0x00007fcf56633ad3 in ?? ()
33 LWP 3580 "rpc worker-3580" 0x00007fcf56633ad3 in ?? ()
34 LWP 3581 "rpc worker-3581" 0x00007fcf56633ad3 in ?? ()
35 LWP 3582 "rpc worker-3582" 0x00007fcf56633ad3 in ?? ()
36 LWP 3583 "rpc worker-3583" 0x00007fcf56633ad3 in ?? ()
37 LWP 3584 "rpc worker-3584" 0x00007fcf56633ad3 in ?? ()
38 LWP 3585 "rpc worker-3585" 0x00007fcf56633ad3 in ?? ()
39 LWP 3586 "rpc worker-3586" 0x00007fcf56633ad3 in ?? ()
40 LWP 3587 "rpc worker-3587" 0x00007fcf56633ad3 in ?? ()
41 LWP 3588 "rpc worker-3588" 0x00007fcf56633ad3 in ?? ()
42 LWP 3589 "rpc worker-3589" 0x00007fcf56633ad3 in ?? ()
43 LWP 3590 "rpc worker-3590" 0x00007fcf56633ad3 in ?? ()
44 LWP 3591 "rpc worker-3591" 0x00007fcf56633ad3 in ?? ()
45 LWP 3592 "rpc worker-3592" 0x00007fcf56633ad3 in ?? ()
46 LWP 3594 "rpc worker-3594" 0x00007fcf56633ad3 in ?? ()
47 LWP 3595 "rpc worker-3595" 0x00007fcf56633ad3 in ?? ()
48 LWP 3597 "rpc worker-3597" 0x00007fcf56633ad3 in ?? ()
49 LWP 3598 "rpc worker-3598" 0x00007fcf56633ad3 in ?? ()
50 LWP 3599 "rpc worker-3599" 0x00007fcf56633ad3 in ?? ()
51 LWP 3600 "rpc worker-3600" 0x00007fcf56633ad3 in ?? ()
52 LWP 3601 "rpc worker-3601" 0x00007fcf56633ad3 in ?? ()
53 LWP 3602 "rpc worker-3602" 0x00007fcf56633ad3 in ?? ()
54 LWP 3603 "rpc worker-3603" 0x00007fcf56633ad3 in ?? ()
55 LWP 3604 "rpc worker-3604" 0x00007fcf56633ad3 in ?? ()
56 LWP 3605 "rpc worker-3605" 0x00007fcf56633ad3 in ?? ()
57 LWP 3606 "rpc worker-3606" 0x00007fcf56633ad3 in ?? ()
58 LWP 3607 "rpc worker-3607" 0x00007fcf56633ad3 in ?? ()
59 LWP 3608 "rpc worker-3608" 0x00007fcf56633ad3 in ?? ()
60 LWP 3609 "rpc worker-3609" 0x00007fcf56633ad3 in ?? ()
61 LWP 3610 "rpc worker-3610" 0x00007fcf56633ad3 in ?? ()
62 LWP 3611 "rpc worker-3611" 0x00007fcf56633ad3 in ?? ()
63 LWP 3612 "rpc worker-3612" 0x00007fcf56633ad3 in ?? ()
64 LWP 3613 "rpc worker-3613" 0x00007fcf56633ad3 in ?? ()
65 LWP 3614 "rpc worker-3614" 0x00007fcf56633ad3 in ?? ()
66 LWP 3615 "rpc worker-3615" 0x00007fcf56633ad3 in ?? ()
67 LWP 3616 "rpc worker-3616" 0x00007fcf56633ad3 in ?? ()
68 LWP 3617 "rpc worker-3617" 0x00007fcf56633ad3 in ?? ()
69 LWP 3618 "rpc worker-3618" 0x00007fcf56633ad3 in ?? ()
70 LWP 3619 "rpc worker-3619" 0x00007fcf56633ad3 in ?? ()
71 LWP 3620 "rpc worker-3620" 0x00007fcf56633ad3 in ?? ()
72 LWP 3621 "rpc worker-3621" 0x00007fcf56633ad3 in ?? ()
73 LWP 3622 "rpc worker-3622" 0x00007fcf56633ad3 in ?? ()
74 LWP 3623 "rpc worker-3623" 0x00007fcf56633ad3 in ?? ()
75 LWP 3624 "rpc worker-3624" 0x00007fcf56633ad3 in ?? ()
76 LWP 3625 "rpc worker-3625" 0x00007fcf56633ad3 in ?? ()
77 LWP 3626 "rpc worker-3626" 0x00007fcf56633ad3 in ?? ()
78 LWP 3627 "rpc worker-3627" 0x00007fcf56633ad3 in ?? ()
79 LWP 3628 "rpc worker-3628" 0x00007fcf56633ad3 in ?? ()
80 LWP 3629 "rpc worker-3629" 0x00007fcf56633ad3 in ?? ()
81 LWP 3630 "rpc worker-3630" 0x00007fcf56633ad3 in ?? ()
82 LWP 3631 "rpc worker-3631" 0x00007fcf56633ad3 in ?? ()
83 LWP 3632 "rpc worker-3632" 0x00007fcf56633ad3 in ?? ()
84 LWP 3633 "rpc worker-3633" 0x00007fcf56633ad3 in ?? ()
85 LWP 3637 "rpc worker-3637" 0x00007fcf56633ad3 in ?? ()
86 LWP 3638 "rpc worker-3638" 0x00007fcf56633ad3 in ?? ()
87 LWP 3639 "rpc worker-3639" 0x00007fcf56633ad3 in ?? ()
88 LWP 3640 "rpc worker-3640" 0x00007fcf56633ad3 in ?? ()
89 LWP 3641 "rpc worker-3641" 0x00007fcf56633ad3 in ?? ()
90 LWP 3642 "rpc worker-3642" 0x00007fcf56633ad3 in ?? ()
91 LWP 3643 "rpc worker-3643" 0x00007fcf56633ad3 in ?? ()
92 LWP 3644 "rpc worker-3644" 0x00007fcf56633ad3 in ?? ()
93 LWP 3645 "rpc worker-3645" 0x00007fcf56633ad3 in ?? ()
94 LWP 3646 "rpc worker-3646" 0x00007fcf56633ad3 in ?? ()
95 LWP 3647 "rpc worker-3647" 0x00007fcf56633ad3 in ?? ()
96 LWP 3648 "rpc worker-3648" 0x00007fcf56633ad3 in ?? ()
97 LWP 3649 "rpc worker-3649" 0x00007fcf56633ad3 in ?? ()
98 LWP 3650 "rpc worker-3650" 0x00007fcf56633ad3 in ?? ()
99 LWP 3651 "rpc worker-3651" 0x00007fcf56633ad3 in ?? ()
100 LWP 3652 "rpc worker-3652" 0x00007fcf56633ad3 in ?? ()
101 LWP 3653 "rpc worker-3653" 0x00007fcf56633ad3 in ?? ()
102 LWP 3654 "rpc worker-3654" 0x00007fcf56633ad3 in ?? ()
103 LWP 3655 "rpc worker-3655" 0x00007fcf56633ad3 in ?? ()
104 LWP 3656 "rpc worker-3656" 0x00007fcf56633ad3 in ?? ()
105 LWP 3657 "rpc worker-3657" 0x00007fcf56633ad3 in ?? ()
106 LWP 3658 "rpc worker-3658" 0x00007fcf56633ad3 in ?? ()
107 LWP 3659 "rpc worker-3659" 0x00007fcf56633ad3 in ?? ()
108 LWP 3660 "rpc worker-3660" 0x00007fcf56633ad3 in ?? ()
109 LWP 3661 "rpc worker-3661" 0x00007fcf56633ad3 in ?? ()
110 LWP 3662 "rpc worker-3662" 0x00007fcf56633ad3 in ?? ()
111 LWP 3663 "rpc worker-3663" 0x00007fcf56633ad3 in ?? ()
112 LWP 3664 "rpc worker-3664" 0x00007fcf56633ad3 in ?? ()
113 LWP 3665 "rpc worker-3665" 0x00007fcf56633ad3 in ?? ()
114 LWP 3666 "rpc worker-3666" 0x00007fcf56633ad3 in ?? ()
115 LWP 3667 "rpc worker-3667" 0x00007fcf56633ad3 in ?? ()
116 LWP 3668 "rpc worker-3668" 0x00007fcf56633ad3 in ?? ()
117 LWP 3669 "diag-logger-366" 0x00007fcf56633fb9 in ?? ()
118 LWP 3670 "result-tracker-" 0x00007fcf56633fb9 in ?? ()
119 LWP 3671 "excess-log-dele" 0x00007fcf56633fb9 in ?? ()
120 LWP 3672 "acceptor-3672" 0x00007fcf4f39f0c7 in ?? ()
121 LWP 3673 "heartbeat-3673" 0x00007fcf56633fb9 in ?? ()
122 LWP 3674 "maintenance_sch" 0x00007fcf56633fb9 in ?? ()
Thread 122 (LWP 3674):
#0 0x00007fcf56633fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000002d3 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x0000616000017584 in ?? ()
#5 0x00007fcf0b6bf550 in ?? ()
#6 0x00000000000005a7 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fcf0b6bf570 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0xffffffff0000006e in ?? ()
#12 0x00007fcf0b6c0030 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 121 (LWP 3673):
#0 0x00007fcf56633fb9 in ?? ()
#1 0xabd77187b7907576 in ?? ()
#2 0x00000000000000b5 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x000061300001aac4 in ?? ()
#5 0x00007fcf0bed75b0 in ?? ()
#6 0x000000000000016b in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fcf0bed75d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007fcf0bed7650 in ?? ()
#12 0x00000ff9e17daecc in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 120 (LWP 3672):
#0 0x00007fcf4f39f0c7 in ?? ()
#1 0x0000614000021aa8 in ?? ()
#2 0x00007fcf0c6fed50 in ?? ()
#3 0x00007fcf0c6fed80 in ?? ()
#4 0x00007fcf0c6fee80 in ?? ()
#5 0x00007fcf0c6fed70 in ?? ()
#6 0x00007fcf0c6fede0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007fcf53229620 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 3671):
#0 0x00007fcf56633fb9 in ?? ()
#1 0x00007fcf0cf17f00 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffcf63db5d0 in ?? ()
#5 0x00007fcf0cf17ef0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 3670):
#0 0x00007fcf56633fb9 in ?? ()
#1 0x00007fcf0d7310c0 in ?? ()
#2 0x00000000000000b4 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007fcf0d7310b0 in ?? ()
#6 0x0000000000000168 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 3669):
#0 0x00007fcf56633fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 3668):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000023 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002f680c in ?? ()
#4 0x00007fcf0e760e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fcf0e760e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002f67c0 in ?? ()
#9 0x00007fcf56633770 in ?? ()
#10 0x00007fcf0e760e70 in ?? ()
#11 0x00000ff9e1cec1d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 3667):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000021 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002f000c in ?? ()
#4 0x00007fcf0ef78e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fcf0ef78e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002effc0 in ?? ()
#9 0x00007fcf56633770 in ?? ()
#10 0x00007fcf0ef78e70 in ?? ()
#11 0x00000ff9e1def1d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 3666):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 3665):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 3664):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 3663):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 3662):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 3661):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 3660):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 3659):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 3658):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 3657):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 3656):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 3655):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 3654):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 3653):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 3652):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 3651):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 3650):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 3649):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 3648):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 3647):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 3646):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 3645):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 3644):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 3643):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 3642):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 3641):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 3640):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 3639):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 3638):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 3637):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 3633):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 3632):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 3631):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 3630):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 3629):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 3628):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 3627):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 3626):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 3625):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000001dcf in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001e684c in ?? ()
#4 0x00007fcf22b31e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fcf22b31e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001e6800 in ?? ()
#9 0x00007fcf56633770 in ?? ()
#10 0x00007fcf22b31e70 in ?? ()
#11 0x00000ff9e45663d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 3624):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x00000000000018af in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001e004c in ?? ()
#4 0x00007fcf23349e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fcf23349e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001e0000 in ?? ()
#9 0x00007fcf56633770 in ?? ()
#10 0x00007fcf23349e70 in ?? ()
#11 0x00000ff9e46693d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 74 (LWP 3623):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x00000000000010f0 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001d6858 in ?? ()
#4 0x00007fcf23b61e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fcf23b61e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 73 (LWP 3622):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 3621):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 3620):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 3619):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 3618):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 3617):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 3616):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 3615):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 3614):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 3613):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 3612):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 3611):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 3610):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 3609):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 3608):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 3607):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 3606):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 3605):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00015d00c in ?? ()
#4 0x00007fcf2cd12e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fcf2cd12e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00015cfc0 in ?? ()
#9 0x00007fcf56633770 in ?? ()
#10 0x00007fcf2cd12e70 in ?? ()
#11 0x00000ff9e59a25d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 3604):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 3603):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 3602):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 3601):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 3600):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 3599):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 3598):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 3597):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 3595):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 3594):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 3592):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 3591):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 3590):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 3589):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 3588):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 3587):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 3586):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 3585):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 3584):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 3583):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x00000000000022cb in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000c67cc in ?? ()
#4 0x00007fcf36ef3e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fcf36ef3e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000c6780 in ?? ()
#9 0x00007fcf56633770 in ?? ()
#10 0x00007fcf36ef3e70 in ?? ()
#11 0x00000ff9e6dde7d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 3582):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000001528 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000b67d8 in ?? ()
#4 0x00007fcf3770be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fcf3770be70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 34 (LWP 3581):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000001888 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000affd8 in ?? ()
#4 0x00007fcf37f23e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fcf37f23e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 33 (LWP 3580):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000002beb in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000acfec in ?? ()
#4 0x00007fcf3873be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fcf3873be70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000acfa0 in ?? ()
#9 0x00007fcf56633770 in ?? ()
#10 0x00007fcf3873be70 in ?? ()
#11 0x00000ff9e70e77d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 32 (LWP 3579):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000cb6 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007fcf38f53e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fcf38f53e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 31 (LWP 3578):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000001ed6 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00009ffe8 in ?? ()
#4 0x00007fcf483b7e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fcf483b7e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 30 (LWP 3577):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 3576):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 3575):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 3574):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 3573):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 3571):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 3570):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 3569):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 3568):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 3567):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 3566):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 3565):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 3564):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 3563):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 3562):
#0 0x00007fcf56633fb9 in ?? ()
#1 0x00007fcf408e0c80 in ?? ()
#2 0x0000000000000012 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ba60 in ?? ()
#5 0x00007fcf408e0c70 in ?? ()
#6 0x0000000000000024 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 3561):
#0 0x00007fcf56633fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 3559):
#0 0x00007fcf56633fb9 in ?? ()
#1 0x00007fcf421781c0 in ?? ()
#2 0x0000000000000012 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007fcf421781b0 in ?? ()
#6 0x0000000000000024 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 3558):
#0 0x00007fcf56633ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x00007fcf429a0b24 in ?? ()
#4 0x00007fcf429a0830 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fcf429a0850 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00006160000177c0 in ?? ()
#9 0x00007fcf56633770 in ?? ()
#10 0x00007fcf429a0850 in ?? ()
#11 0x00007fcf429a0850 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 12 (LWP 3555):
#0 0x00007fcf4f39da47 in ?? ()
#1 0x00007fcf431b22e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007fcf431b22d0 in ?? ()
#4 0x00007fcf431b24e0 in ?? ()
#5 0x00007fcf431b2320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007fcf431b23a0 in ?? ()
#8 0x00007fcf519fd25d in ?? ()
#9 0x3fb930b1ac6b2000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000073300000 in ?? ()
#14 0x00007fcf00000000 in ?? ()
#15 0x41d9e077162c110c in ?? ()
#16 0x00000ffa6862e474 in ?? ()
#17 0x00007fcf431b2380 in ?? ()
#18 0x00007fcf51a01ba3 in ?? ()
#19 0x00007fcf431b2350 in ?? ()
#20 0x3fb930b1ac6b2000 in ?? ()
#21 0x00000000431b23a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb930b1ac6b2000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 3554):
#0 0x00007fcf4f39da47 in ?? ()
#1 0x00007fcf439b32e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007fcf439b32d0 in ?? ()
#4 0x00007fcf439b34e0 in ?? ()
#5 0x00007fcf439b3320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007fcf439b33a0 in ?? ()
#8 0x00007fcf519fd25d in ?? ()
#9 0x3fb9813cc1291000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x00000000f4800000 in ?? ()
#14 0x00007fcf00000000 in ?? ()
#15 0x41d9e077162c110b in ?? ()
#16 0x00000ffa6872e674 in ?? ()
#17 0x00007fcf439b3380 in ?? ()
#18 0x00007fcf51a01ba3 in ?? ()
#19 0x00007fcf439b3350 in ?? ()
#20 0x3fb9813cc1291000 in ?? ()
#21 0x00000000439b33a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb9813cc1291000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 3553):
#0 0x00007fcf4f39da47 in ?? ()
#1 0x00007fcf441b42e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007fcf441b42d0 in ?? ()
#4 0x00007fcf441b44e0 in ?? ()
#5 0x00007fcf441b4320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007fcf441b43a0 in ?? ()
#8 0x00007fcf519fd25d in ?? ()
#9 0x3f7745c6efd00000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x000000009c000000 in ?? ()
#14 0x00007fcf00000000 in ?? ()
#15 0x41d9e077162c110a in ?? ()
#16 0x00000ffa6882e874 in ?? ()
#17 0x00007fcf441b4380 in ?? ()
#18 0x00007fcf51a01ba3 in ?? ()
#19 0x00007fcf441b4350 in ?? ()
#20 0x3f7745c6efd00000 in ?? ()
#21 0x00000000441b43a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3f7745c6efd00000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 3552):
#0 0x00007fcf4f39da47 in ?? ()
#1 0x00007fcf461ad2e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007fcf461ad2d0 in ?? ()
#4 0x00007fcf461ad4e0 in ?? ()
#5 0x00007fcf461ad320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fcf461ad3a0 in ?? ()
#8 0x00007fcf519fd25d in ?? ()
#9 0x3fb95993a7364000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x00000000a5000000 in ?? ()
#14 0x00007fcf00000000 in ?? ()
#15 0x41d9e077162c110a in ?? ()
#16 0x00000ffa68c2da74 in ?? ()
#17 0x00007fcf461ad380 in ?? ()
#18 0x00007fcf51a01ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 3549):
#0 0x00007fcf4f390cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fcf47bb7788 in ?? ()
#3 0x0000602000016cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 3548):
#0 0x00007fcf56633fb9 in ?? ()
#1 0x00007fcf473b6080 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007fcf473b6070 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 3547):
#0 0x00007fcf566379e2 in ?? ()
#1 0x00007fcf46bb3b60 in ?? ()
#2 0x00007fcf46bb3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fcf46bb3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007fcf46bb3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007fcf46bb3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007fcf5bf5bf0c in ?? ()
#12 0x00007fcf5bf4b7b9 in ?? ()
#13 0x00007fcf5bf4fda6 in ?? ()
#14 0x00007fcf5bf547e0 in ?? ()
#15 0x00007fcf5bf54609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007fcf5324a0be in ?? ()
#18 0x00007fcf5662d6db in ?? ()
#19 0x00000ffa68d6e67c in ?? ()
#20 0x00007fcf46bb3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 3541):
#0 0x00007fcf56633fb9 in ?? ()
#1 0x00007fcf48bb8c40 in ?? ()
#2 0x000000000000038b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007fcf48bb8c30 in ?? ()
#6 0x0000000000000716 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 3540):
#0 0x00007fcf56633fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 3539):
#0 0x00007fcf56633fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 3538):
#0 0x00007fcf56633fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fcf4a3bc410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 3535):
#0 0x00007fcf56637d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250111 02:58:17.786319 1102 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 3 with UUID 185ad035c2754007b1b3d08bb65546c2 and pid 4929
************************ BEGIN STACKS **************************
[New LWP 4931]
[New LWP 4932]
[New LWP 4933]
[New LWP 4934]
[New LWP 4940]
[New LWP 4941]
[New LWP 4942]
[New LWP 4948]
[New LWP 4949]
[New LWP 4950]
[New LWP 4951]
[New LWP 4952]
[New LWP 4953]
[New LWP 4955]
[New LWP 4956]
[New LWP 4957]
[New LWP 4958]
[New LWP 4959]
[New LWP 4960]
[New LWP 4961]
[New LWP 4962]
[New LWP 4963]
[New LWP 4964]
[New LWP 4965]
[New LWP 4966]
[New LWP 4967]
[New LWP 4968]
[New LWP 4969]
[New LWP 4970]
[New LWP 4971]
[New LWP 4972]
[New LWP 4973]
[New LWP 4974]
[New LWP 4975]
[New LWP 4976]
[New LWP 4977]
[New LWP 4978]
[New LWP 4979]
[New LWP 4980]
[New LWP 4981]
[New LWP 4982]
[New LWP 4983]
[New LWP 4984]
[New LWP 4985]
[New LWP 4986]
[New LWP 4987]
[New LWP 4988]
[New LWP 4989]
[New LWP 4990]
[New LWP 4991]
[New LWP 4992]
[New LWP 4993]
[New LWP 4994]
[New LWP 4995]
[New LWP 4996]
[New LWP 4997]
[New LWP 4998]
[New LWP 4999]
[New LWP 5000]
[New LWP 5001]
[New LWP 5002]
[New LWP 5003]
[New LWP 5004]
[New LWP 5005]
[New LWP 5006]
[New LWP 5007]
[New LWP 5008]
[New LWP 5009]
[New LWP 5010]
[New LWP 5011]
[New LWP 5012]
[New LWP 5013]
[New LWP 5014]
[New LWP 5015]
[New LWP 5016]
[New LWP 5017]
[New LWP 5018]
[New LWP 5019]
[New LWP 5020]
[New LWP 5022]
[New LWP 5023]
[New LWP 5024]
[New LWP 5025]
[New LWP 5026]
[New LWP 5027]
[New LWP 5028]
[New LWP 5030]
[New LWP 5031]
[New LWP 5034]
[New LWP 5036]
[New LWP 5037]
[New LWP 5038]
[New LWP 5039]
[New LWP 5042]
[New LWP 5043]
[New LWP 5044]
[New LWP 5045]
[New LWP 5047]
[New LWP 5048]
[New LWP 5049]
[New LWP 5050]
[New LWP 5051]
[New LWP 5052]
[New LWP 5053]
[New LWP 5054]
[New LWP 5055]
[New LWP 5056]
[New LWP 5057]
[New LWP 5058]
[New LWP 5059]
[New LWP 5060]
[New LWP 5061]
[New LWP 5062]
[New LWP 5064]
[New LWP 5065]
[New LWP 5066]
[New LWP 5067]
[New LWP 5068]
[New LWP 5069]
[New LWP 5070]
[New LWP 5071]
[New LWP 5609]
[New LWP 5610]
[New LWP 5611]
[New LWP 5612]
[New LWP 5613]
[New LWP 5614]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007fb31be34d50 in ?? ()
Id Target Id Frame
* 1 LWP 4929 "kudu" 0x00007fb31be34d50 in ?? ()
2 LWP 4931 "kudu" 0x00007fb31be30fb9 in ?? ()
3 LWP 4932 "kudu" 0x00007fb31be30fb9 in ?? ()
4 LWP 4933 "kudu" 0x00007fb31be30fb9 in ?? ()
5 LWP 4934 "kernel-watcher-" 0x00007fb31be30fb9 in ?? ()
6 LWP 4940 "ntp client-4940" 0x00007fb31be349e2 in ?? ()
7 LWP 4941 "file cache-evic" 0x00007fb31be30fb9 in ?? ()
8 LWP 4942 "sq_acceptor" 0x00007fb314b8dcb9 in ?? ()
9 LWP 4948 "rpc reactor-494" 0x00007fb314b9aa47 in ?? ()
10 LWP 4949 "rpc reactor-494" 0x00007fb314b9aa47 in ?? ()
11 LWP 4950 "rpc reactor-495" 0x00007fb314b9aa47 in ?? ()
12 LWP 4951 "rpc reactor-495" 0x00007fb314b9aa47 in ?? ()
13 LWP 4952 "MaintenanceMgr " 0x00007fb31be30ad3 in ?? ()
14 LWP 4953 "txn-status-mana" 0x00007fb31be30fb9 in ?? ()
15 LWP 4955 "collect_and_rem" 0x00007fb31be30fb9 in ?? ()
16 LWP 4956 "tc-session-exp-" 0x00007fb31be30fb9 in ?? ()
17 LWP 4957 "rpc worker-4957" 0x00007fb31be30ad3 in ?? ()
18 LWP 4958 "rpc worker-4958" 0x00007fb31be30ad3 in ?? ()
19 LWP 4959 "rpc worker-4959" 0x00007fb31be30ad3 in ?? ()
20 LWP 4960 "rpc worker-4960" 0x00007fb31be30ad3 in ?? ()
21 LWP 4961 "rpc worker-4961" 0x00007fb31be30ad3 in ?? ()
22 LWP 4962 "rpc worker-4962" 0x00007fb31be30ad3 in ?? ()
23 LWP 4963 "rpc worker-4963" 0x00007fb31be30ad3 in ?? ()
24 LWP 4964 "rpc worker-4964" 0x00007fb31be30ad3 in ?? ()
25 LWP 4965 "rpc worker-4965" 0x00007fb31be30ad3 in ?? ()
26 LWP 4966 "rpc worker-4966" 0x00007fb31be30ad3 in ?? ()
27 LWP 4967 "rpc worker-4967" 0x00007fb31be30ad3 in ?? ()
28 LWP 4968 "rpc worker-4968" 0x00007fb31be30ad3 in ?? ()
29 LWP 4969 "rpc worker-4969" 0x00007fb31be30ad3 in ?? ()
30 LWP 4970 "rpc worker-4970" 0x00007fb31be30ad3 in ?? ()
31 LWP 4971 "rpc worker-4971" 0x00007fb31be30ad3 in ?? ()
32 LWP 4972 "rpc worker-4972" 0x00007fb31be30ad3 in ?? ()
33 LWP 4973 "rpc worker-4973" 0x00007fb31be30ad3 in ?? ()
34 LWP 4974 "rpc worker-4974" 0x00007fb31be30ad3 in ?? ()
35 LWP 4975 "rpc worker-4975" 0x00007fb31be30ad3 in ?? ()
36 LWP 4976 "rpc worker-4976" 0x00007fb31be30ad3 in ?? ()
37 LWP 4977 "rpc worker-4977" 0x00007fb31be30ad3 in ?? ()
38 LWP 4978 "rpc worker-4978" 0x00007fb31be30ad3 in ?? ()
39 LWP 4979 "rpc worker-4979" 0x00007fb31be30ad3 in ?? ()
40 LWP 4980 "rpc worker-4980" 0x00007fb31be30ad3 in ?? ()
41 LWP 4981 "rpc worker-4981" 0x00007fb31be30ad3 in ?? ()
42 LWP 4982 "rpc worker-4982" 0x00007fb31be30ad3 in ?? ()
43 LWP 4983 "rpc worker-4983" 0x00007fb31be30ad3 in ?? ()
44 LWP 4984 "rpc worker-4984" 0x00007fb31be30ad3 in ?? ()
45 LWP 4985 "rpc worker-4985" 0x00007fb31be30ad3 in ?? ()
46 LWP 4986 "rpc worker-4986" 0x00007fb31be30ad3 in ?? ()
47 LWP 4987 "rpc worker-4987" 0x00007fb31be30ad3 in ?? ()
48 LWP 4988 "rpc worker-4988" 0x00007fb31be30ad3 in ?? ()
49 LWP 4989 "rpc worker-4989" 0x00007fb31be30ad3 in ?? ()
50 LWP 4990 "rpc worker-4990" 0x00007fb31be30ad3 in ?? ()
51 LWP 4991 "rpc worker-4991" 0x00007fb31be30ad3 in ?? ()
52 LWP 4992 "rpc worker-4992" 0x00007fb31be30ad3 in ?? ()
53 LWP 4993 "rpc worker-4993" 0x00007fb31be30ad3 in ?? ()
54 LWP 4994 "rpc worker-4994" 0x00007fb31be30ad3 in ?? ()
55 LWP 4995 "rpc worker-4995" 0x00007fb31be30ad3 in ?? ()
56 LWP 4996 "rpc worker-4996" 0x00007fb31be30ad3 in ?? ()
57 LWP 4997 "rpc worker-4997" 0x00007fb31be30ad3 in ?? ()
58 LWP 4998 "rpc worker-4998" 0x00007fb31be30ad3 in ?? ()
59 LWP 4999 "rpc worker-4999" 0x00007fb31be30ad3 in ?? ()
60 LWP 5000 "rpc worker-5000" 0x00007fb31be30ad3 in ?? ()
61 LWP 5001 "rpc worker-5001" 0x00007fb31be30ad3 in ?? ()
62 LWP 5002 "rpc worker-5002" 0x00007fb31be30ad3 in ?? ()
63 LWP 5003 "rpc worker-5003" 0x00007fb31be30ad3 in ?? ()
64 LWP 5004 "rpc worker-5004" 0x00007fb31be30ad3 in ?? ()
65 LWP 5005 "rpc worker-5005" 0x00007fb31be30ad3 in ?? ()
66 LWP 5006 "rpc worker-5006" 0x00007fb31be30ad3 in ?? ()
67 LWP 5007 "rpc worker-5007" 0x00007fb31be30ad3 in ?? ()
68 LWP 5008 "rpc worker-5008" 0x00007fb31be30ad3 in ?? ()
69 LWP 5009 "rpc worker-5009" 0x00007fb31be30ad3 in ?? ()
70 LWP 5010 "rpc worker-5010" 0x00007fb31be30ad3 in ?? ()
71 LWP 5011 "rpc worker-5011" 0x00007fb31be30ad3 in ?? ()
72 LWP 5012 "rpc worker-5012" 0x00007fb31be30ad3 in ?? ()
73 LWP 5013 "rpc worker-5013" 0x00007fb31be30ad3 in ?? ()
74 LWP 5014 "rpc worker-5014" 0x00007fb31be30ad3 in ?? ()
75 LWP 5015 "rpc worker-5015" 0x00007fb31be30ad3 in ?? ()
76 LWP 5016 "rpc worker-5016" 0x00007fb31be30ad3 in ?? ()
77 LWP 5017 "rpc worker-5017" 0x00007fb31be30ad3 in ?? ()
78 LWP 5018 "rpc worker-5018" 0x00007fb31be30ad3 in ?? ()
79 LWP 5019 "rpc worker-5019" 0x00007fb31be30ad3 in ?? ()
80 LWP 5020 "rpc worker-5020" 0x00007fb31be30ad3 in ?? ()
81 LWP 5022 "rpc worker-5022" 0x00007fb31be30ad3 in ?? ()
82 LWP 5023 "rpc worker-5023" 0x00007fb31be30ad3 in ?? ()
83 LWP 5024 "rpc worker-5024" 0x00007fb31be30ad3 in ?? ()
84 LWP 5025 "rpc worker-5025" 0x00007fb31be30ad3 in ?? ()
85 LWP 5026 "rpc worker-5026" 0x00007fb31be30ad3 in ?? ()
86 LWP 5027 "rpc worker-5027" 0x00007fb31be30ad3 in ?? ()
87 LWP 5028 "rpc worker-5028" 0x00007fb31be30ad3 in ?? ()
88 LWP 5030 "rpc worker-5030" 0x00007fb31be30ad3 in ?? ()
89 LWP 5031 "rpc worker-5031" 0x00007fb31be30ad3 in ?? ()
90 LWP 5034 "rpc worker-5034" 0x00007fb31be30ad3 in ?? ()
91 LWP 5036 "rpc worker-5036" 0x00007fb31be30ad3 in ?? ()
92 LWP 5037 "rpc worker-5037" 0x00007fb31be30ad3 in ?? ()
93 LWP 5038 "rpc worker-5038" 0x00007fb31be30ad3 in ?? ()
94 LWP 5039 "rpc worker-5039" 0x00007fb31be30ad3 in ?? ()
95 LWP 5042 "rpc worker-5042" 0x00007fb31be30ad3 in ?? ()
96 LWP 5043 "rpc worker-5043" 0x00007fb31be30ad3 in ?? ()
97 LWP 5044 "rpc worker-5044" 0x00007fb31be30ad3 in ?? ()
98 LWP 5045 "rpc worker-5045" 0x00007fb31be30ad3 in ?? ()
99 LWP 5047 "rpc worker-5047" 0x00007fb31be30ad3 in ?? ()
100 LWP 5048 "rpc worker-5048" 0x00007fb31be30ad3 in ?? ()
101 LWP 5049 "rpc worker-5049" 0x00007fb31be30ad3 in ?? ()
102 LWP 5050 "rpc worker-5050" 0x00007fb31be30ad3 in ?? ()
103 LWP 5051 "rpc worker-5051" 0x00007fb31be30ad3 in ?? ()
104 LWP 5052 "rpc worker-5052" 0x00007fb31be30ad3 in ?? ()
105 LWP 5053 "rpc worker-5053" 0x00007fb31be30ad3 in ?? ()
106 LWP 5054 "rpc worker-5054" 0x00007fb31be30ad3 in ?? ()
107 LWP 5055 "rpc worker-5055" 0x00007fb31be30ad3 in ?? ()
108 LWP 5056 "rpc worker-5056" 0x00007fb31be30ad3 in ?? ()
109 LWP 5057 "rpc worker-5057" 0x00007fb31be30ad3 in ?? ()
110 LWP 5058 "rpc worker-5058" 0x00007fb31be30ad3 in ?? ()
111 LWP 5059 "rpc worker-5059" 0x00007fb31be30ad3 in ?? ()
112 LWP 5060 "rpc worker-5060" 0x00007fb31be30ad3 in ?? ()
113 LWP 5061 "rpc worker-5061" 0x00007fb31be30ad3 in ?? ()
114 LWP 5062 "rpc worker-5062" 0x00007fb31be30ad3 in ?? ()
115 LWP 5064 "rpc worker-5064" 0x00007fb31be30ad3 in ?? ()
116 LWP 5065 "rpc worker-5065" 0x00007fb31be30ad3 in ?? ()
117 LWP 5066 "diag-logger-506" 0x00007fb31be30fb9 in ?? ()
118 LWP 5067 "result-tracker-" 0x00007fb31be30fb9 in ?? ()
119 LWP 5068 "excess-log-dele" 0x00007fb31be30fb9 in ?? ()
120 LWP 5069 "acceptor-5069" 0x00007fb314b9c0c7 in ?? ()
121 LWP 5070 "heartbeat-5070" 0x00007fb31be30fb9 in ?? ()
122 LWP 5071 "maintenance_sch" 0x00007fb31be30fb9 in ?? ()
123 LWP 5609 "prepare [worker" 0x00007fb31a9553f0 in ?? ()
124 LWP 5610 "wal-append [wor" 0x00007fb31be30fb9 in ?? ()
125 LWP 5611 "apply [worker]-" 0x00007fb3254e16b0 in ?? ()
126 LWP 5612 "apply [worker]-" 0x00007fb317a7fa2b in ?? ()
127 LWP 5613 "apply [worker]-" 0x00007fb31be314fc in ?? ()
128 LWP 5614 "apply [worker]-" 0x00007fb32550ccb6 in ?? ()
Thread 128 (LWP 5614):
#0 0x00007fb32550ccb6 in ?? ()
#1 0x0000000041b58ab3 in ?? ()
#2 0x00007fb324fe27f7 in ?? ()
#3 0x00007fb32550cc20 in ?? ()
#4 0x00007fb2c5ca7ee0 in ?? ()
#5 0x00007fb2cb9e4e41 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x00007fb2c5ca8110 in ?? ()
#8 0x00007fb2ca5f2180 in ?? ()
#9 0x0000004aab5cd104 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 127 (LWP 5613):
#0 0x00007fb31be314fc in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000158c95544 in ?? ()
#3 0x0000000000000d26 in ?? ()
#4 0x000061a00036afc8 in ?? ()
#5 0x0000000045e0360e in ?? ()
#6 0x00007fb2c64aab20 in ?? ()
#7 0x00007fb2c64aab90 in ?? ()
#8 0x000061a00036afa0 in ?? ()
#9 0x00000c340006d5ea in ?? ()
#10 0x00007fb2c64aaa60 in ?? ()
#11 0x00000ff658c9554c in ?? ()
#12 0x00007fb3187ac43a in ?? ()
#13 0x0000000041b58ab3 in ?? ()
#14 0x00007fb31838cff7 in ?? ()
#15 0x00007fb3187ac380 in ?? ()
#16 0x00007fb323c73f59 in ?? ()
#17 0x0000000045e0360e in ?? ()
#18 0x00007fb323b9bd08 in ?? ()
#19 0x00007fb323c73e90 in ?? ()
#20 0x00007fb2c64aaa40 in ?? ()
#21 0x00006150000a9e00 in ?? ()
#22 0x00006150000a9e00 in ?? ()
#23 0x00006150000aa000 in ?? ()
#24 0x000060600156e7f8 in ?? ()
#25 0x00007fb2c64aaad0 in ?? ()
#26 0x00007fb2c64ab070 in ?? ()
#27 0x00000c340006d60a in ?? ()
#28 0x00000c340006d5ea in ?? ()
#29 0x00006150000a9e00 in ?? ()
#30 0x000060d00276ae90 in ?? ()
#31 0x0000000000000000 in ?? ()
Thread 126 (LWP 5612):
#0 0x00007fb317a7fa2b in ?? ()
#1 0x0000000041b58ab3 in ?? ()
#2 0x00007fb317954254 in ?? ()
#3 0x00007fb317a7f920 in ?? ()
#4 0x00007fb2c6cabe80 in ?? ()
#5 0x00007fb323bae5e0 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x00007fb2c6cabbd0 in ?? ()
#8 0x00007fb2c6cabb60 in ?? ()
#9 0x000000000000001d in ?? ()
#10 0x00000ff658d95770 in ?? ()
#11 0x000061d0023beb4b in ?? ()
#12 0x00007fb2c6cabba0 in ?? ()
#13 0x00007fb2c6cabb80 in ?? ()
#14 0x0000000000000000 in ?? ()
Thread 125 (LWP 5611):
#0 0x00007fb3254e16b0 in ?? ()
#1 0x00007fb32549ecc7 in ?? ()
#2 0x0000000041b58ab3 in ?? ()
#3 0x00007fb324fbc949 in ?? ()
#4 0x00007fb32549e7d0 in ?? ()
#5 0x00007fb317a7fbb9 in ?? ()
#6 0x00007fb324f573a0 in ?? ()
#7 0x00007fb30000001a in ?? ()
#8 0x00007fb317a7f920 in ?? ()
#9 0x00007fb3179c6720 in ?? ()
#10 0x000060400062e290 in ?? ()
#11 0x0000000000000020 in ?? ()
#12 0x0000000000000020 in ?? ()
#13 0x00007fb2c74acca0 in ?? ()
#14 0x000000000000000f in ?? ()
#15 0x00000ff658e95998 in ?? ()
#16 0x0000000000000001 in ?? ()
#17 0x00007fb2c74acce0 in ?? ()
#18 0x0000000000000000 in ?? ()
Thread 124 (LWP 5610):
#0 0x00007fb31be30fb9 in ?? ()
#1 0x0000602000029018 in ?? ()
#2 0x0000000000000d28 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061a00036afb0 in ?? ()
#5 0x00007fb2d466c090 in ?? ()
#6 0x0000000000001a50 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 123 (LWP 5609):
#0 0x00007fb31a9553f0 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 122 (LWP 5071):
#0 0x00007fb31be30fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000125 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007fb2d0572550 in ?? ()
#6 0x000000000000024a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 5070):
#0 0x00007fb31be30fb9 in ?? ()
#1 0xabd77187b7907576 in ?? ()
#2 0x000000000000004a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007fb2d0d8a5b0 in ?? ()
#6 0x0000000000000094 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 5069):
#0 0x00007fb314b9c0c7 in ?? ()
#1 0x0000614000021aa8 in ?? ()
#2 0x00007fb2d15b1d50 in ?? ()
#3 0x00007fb2d15b1d80 in ?? ()
#4 0x00007fb2d15b1e80 in ?? ()
#5 0x00007fb2d15b1d70 in ?? ()
#6 0x00007fb2d15b1de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 5068):
#0 0x00007fb31be30fb9 in ?? ()
#1 0x00007fb2d1dcaf00 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffea35d4090 in ?? ()
#5 0x00007fb2d1dcaef0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 5067):
#0 0x00007fb31be30fb9 in ?? ()
#1 0x00007fb2d25e30c0 in ?? ()
#2 0x0000000000000049 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007fb2d25e30b0 in ?? ()
#6 0x0000000000000092 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 5066):
#0 0x00007fb31be30fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 5065):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x000000000000000c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000346888 in ?? ()
#4 0x00007fb2d3612e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb2d3612e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 5064):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000014 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0003367c8 in ?? ()
#4 0x00007fb2d3e2ae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb2d3e2ae70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 5062):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 5061):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 5060):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 5059):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 5058):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 5057):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 5056):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 5055):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 5054):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 5053):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 5052):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 5051):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 5050):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 5049):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 5048):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 5047):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 5045):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 5044):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 5043):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 5042):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 5039):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 5038):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 5037):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 5036):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 5034):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 5031):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 5030):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 5028):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 5027):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 5026):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 5025):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 5024):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 5023):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 5022):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 5020):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 5019):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 5018):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 5017):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 5016):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000231 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001e684c in ?? ()
#4 0x00007fb2e831ee50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb2e831ee70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001e6800 in ?? ()
#9 0x00007fb31be30770 in ?? ()
#10 0x00007fb2e831ee70 in ?? ()
#11 0x00000ff65d063dd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 5015):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x000000000000004f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001d685c in ?? ()
#4 0x00007fb2e8b36e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb2e8b36e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001d6810 in ?? ()
#9 0x00007fb31be30770 in ?? ()
#10 0x00007fb2e8b36e70 in ?? ()
#11 0x00000ff65d166dd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 74 (LWP 5014):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 5013):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 5012):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 5011):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 5010):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 5009):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 5008):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 5007):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 5006):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 5005):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 5004):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 5003):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 5002):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 5001):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 5000):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 4999):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 4998):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 4997):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 4996):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000150008 in ?? ()
#4 0x00007fb2f24ffe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb2f24ffe70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 4995):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 4994):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 4993):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 4992):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 4991):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 4990):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 4989):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 4988):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 4987):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 4986):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 4985):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 4984):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 4983):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 4982):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 4981):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 4980):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 4979):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 4978):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 4977):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 4976):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000bcfd8 in ?? ()
#4 0x00007fb30b3b2e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb30b3b2e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 4975):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 4974):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 4973):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 4972):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 4971):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 4970):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 4969):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 4968):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 4967):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 4966):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 4965):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 4964):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 4963):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 4962):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 4961):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 4960):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 4959):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 4958):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 4957):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 4956):
#0 0x00007fb31be30fb9 in ?? ()
#1 0x00007fb3047bfc80 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ba60 in ?? ()
#5 0x00007fb3047bfc70 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 4955):
#0 0x00007fb31be30fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 4953):
#0 0x00007fb31be30fb9 in ?? ()
#1 0x00007fb305fc21c0 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007fb305fc21b0 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 4952):
#0 0x00007fb31be30ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 4951):
#0 0x00007fb314b9aa47 in ?? ()
#1 0x00007fb306fc52e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007fb306fc52d0 in ?? ()
#4 0x00007fb306fc54e0 in ?? ()
#5 0x00007fb306fc5320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007fb306fc53a0 in ?? ()
#8 0x00007fb3171fa25d in ?? ()
#9 0x3fb964cea596b000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000075000000 in ?? ()
#14 0x00007fb300000000 in ?? ()
#15 0x41d9e077162c1108 in ?? ()
#16 0x00000ff6e0df0a74 in ?? ()
#17 0x00007fb306fc5380 in ?? ()
#18 0x00007fb3171feba3 in ?? ()
#19 0x00007fb306fc5350 in ?? ()
#20 0x3fb964cea596b000 in ?? ()
#21 0x0000000006fc53a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb964cea596b000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 4950):
#0 0x00007fb314b9aa47 in ?? ()
#1 0x00007fb3077c62e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007fb3077c62d0 in ?? ()
#4 0x00007fb3077c64e0 in ?? ()
#5 0x00007fb3077c6320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007fb3077c63a0 in ?? ()
#8 0x00007fb3171fa25d in ?? ()
#9 0x3fb41fc390745000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x0000000037000000 in ?? ()
#14 0x00007fb300000000 in ?? ()
#15 0x41d9e077162c110a in ?? ()
#16 0x00000ff6e0ef0c74 in ?? ()
#17 0x00007fb3077c6380 in ?? ()
#18 0x00007fb3171feba3 in ?? ()
#19 0x00007fb3077c6350 in ?? ()
#20 0x3fb41fc390745000 in ?? ()
#21 0x00000000077c63a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb41fc390745000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 4949):
#0 0x00007fb314b9aa47 in ?? ()
#1 0x00007fb307fc72e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007fb307fc72d0 in ?? ()
#4 0x00007fb307fc74e0 in ?? ()
#5 0x00007fb307fc7320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007fb307fc73a0 in ?? ()
#8 0x00007fb3171fa25d in ?? ()
#9 0x3fb955baa990f000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000057800000 in ?? ()
#14 0x00007fb300000000 in ?? ()
#15 0x41d9e077162c1109 in ?? ()
#16 0x00000ff6e0ff0e74 in ?? ()
#17 0x00007fb307fc7380 in ?? ()
#18 0x00007fb3171feba3 in ?? ()
#19 0x00007fb307fc7350 in ?? ()
#20 0x3fb955baa990f000 in ?? ()
#21 0x0000000007fc73a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb955baa990f000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 4948):
#0 0x00007fb314b9aa47 in ?? ()
#1 0x00007fb309bb02e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007fb309bb02d0 in ?? ()
#4 0x00007fb309bb04e0 in ?? ()
#5 0x00007fb309bb0320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fb309bb03a0 in ?? ()
#8 0x00007fb3171fa25d in ?? ()
#9 0x3fb987106a994000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x00000000a1000000 in ?? ()
#14 0x00007fb300000000 in ?? ()
#15 0x41d9e077162c110c in ?? ()
#16 0x00000ff6e136e074 in ?? ()
#17 0x00007fb309bb0380 in ?? ()
#18 0x00007fb3171feba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 4942):
#0 0x00007fb314b8dcb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fb30d3b7788 in ?? ()
#3 0x0000602000017cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 4941):
#0 0x00007fb31be30fb9 in ?? ()
#1 0x00007fb30cbb6080 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007fb30cbb6070 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 4940):
#0 0x00007fb31be349e2 in ?? ()
#1 0x00007fb30bbb2b60 in ?? ()
#2 0x00007fb30bbb2c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fb30bbb2bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007fb30bbb2c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007fb30bbb2c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007fb321758f0c in ?? ()
#12 0x00007fb3217487b9 in ?? ()
#13 0x00007fb32174cda6 in ?? ()
#14 0x00007fb3217517e0 in ?? ()
#15 0x00007fb321751609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007fb318a470be in ?? ()
#18 0x00007fb31be2a6db in ?? ()
#19 0x00000ff6e176e47c in ?? ()
#20 0x00007fb30bbb2400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 4934):
#0 0x00007fb31be30fb9 in ?? ()
#1 0x00007fb30e3b8c40 in ?? ()
#2 0x0000000000000179 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007fb30e3b8c30 in ?? ()
#6 0x00000000000002f2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 4933):
#0 0x00007fb31be30fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 4932):
#0 0x00007fb31be30fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 4931):
#0 0x00007fb31be30fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000004 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fb30fbbc410 in ?? ()
#6 0x0000000000000008 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 4929):
#0 0x00007fb31be34d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250111 02:58:19.049113 1102 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 4 with UUID 0eb58b1fa7c746f8abc51f33c199e832 and pid 2825
************************ BEGIN STACKS **************************
[New LWP 2831]
[New LWP 2832]
[New LWP 2833]
[New LWP 2835]
[New LWP 2841]
[New LWP 2842]
[New LWP 2843]
[New LWP 2846]
[New LWP 2847]
[New LWP 2848]
[New LWP 2849]
[New LWP 2850]
[New LWP 2851]
[New LWP 2852]
[New LWP 2853]
[New LWP 2854]
[New LWP 2855]
[New LWP 2856]
[New LWP 2857]
[New LWP 2858]
[New LWP 2859]
[New LWP 2860]
[New LWP 2861]
[New LWP 2862]
[New LWP 2863]
[New LWP 2864]
[New LWP 2865]
[New LWP 2866]
[New LWP 2867]
[New LWP 2868]
[New LWP 2869]
[New LWP 2870]
[New LWP 2871]
[New LWP 2872]
[New LWP 2873]
[New LWP 2874]
[New LWP 2875]
[New LWP 2876]
[New LWP 2877]
[New LWP 2878]
[New LWP 2879]
[New LWP 2880]
[New LWP 2881]
[New LWP 2882]
[New LWP 2883]
[New LWP 2884]
[New LWP 2885]
[New LWP 2886]
[New LWP 2887]
[New LWP 2888]
[New LWP 2889]
[New LWP 2890]
[New LWP 2891]
[New LWP 2892]
[New LWP 2893]
[New LWP 2894]
[New LWP 2895]
[New LWP 2896]
[New LWP 2897]
[New LWP 2898]
[New LWP 2899]
[New LWP 2900]
[New LWP 2901]
[New LWP 2902]
[New LWP 2903]
[New LWP 2904]
[New LWP 2905]
[New LWP 2906]
[New LWP 2907]
[New LWP 2908]
[New LWP 2909]
[New LWP 2910]
[New LWP 2911]
[New LWP 2912]
[New LWP 2913]
[New LWP 2914]
[New LWP 2915]
[New LWP 2916]
[New LWP 2917]
[New LWP 2918]
[New LWP 2919]
[New LWP 2920]
[New LWP 2921]
[New LWP 2922]
[New LWP 2923]
[New LWP 2924]
[New LWP 2925]
[New LWP 2926]
[New LWP 2927]
[New LWP 2928]
[New LWP 2929]
[New LWP 2930]
[New LWP 2931]
[New LWP 2932]
[New LWP 2933]
[New LWP 2934]
[New LWP 2935]
[New LWP 2936]
[New LWP 2937]
[New LWP 2938]
[New LWP 2939]
[New LWP 2940]
[New LWP 2941]
[New LWP 2942]
[New LWP 2943]
[New LWP 2944]
[New LWP 2945]
[New LWP 2946]
[New LWP 2947]
[New LWP 2948]
[New LWP 2949]
[New LWP 2950]
[New LWP 2951]
[New LWP 2952]
[New LWP 2953]
[New LWP 2954]
[New LWP 2955]
[New LWP 2956]
[New LWP 2957]
[New LWP 2958]
[New LWP 2959]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007fb6aac72d50 in ?? ()
Id Target Id Frame
* 1 LWP 2825 "kudu" 0x00007fb6aac72d50 in ?? ()
2 LWP 2831 "kudu" 0x00007fb6aac6efb9 in ?? ()
3 LWP 2832 "kudu" 0x00007fb6aac6efb9 in ?? ()
4 LWP 2833 "kudu" 0x00007fb6aac6efb9 in ?? ()
5 LWP 2835 "kernel-watcher-" 0x00007fb6aac6efb9 in ?? ()
6 LWP 2841 "ntp client-2841" 0x00007fb6aac729e2 in ?? ()
7 LWP 2842 "file cache-evic" 0x00007fb6aac6efb9 in ?? ()
8 LWP 2843 "sq_acceptor" 0x00007fb6a39cbcb9 in ?? ()
9 LWP 2846 "rpc reactor-284" 0x00007fb6a39d8a47 in ?? ()
10 LWP 2847 "rpc reactor-284" 0x00007fb6a39d8a47 in ?? ()
11 LWP 2848 "rpc reactor-284" 0x00007fb6a39d8a47 in ?? ()
12 LWP 2849 "rpc reactor-284" 0x00007fb6a39d8a47 in ?? ()
13 LWP 2850 "MaintenanceMgr " 0x00007fb6aac6ead3 in ?? ()
14 LWP 2851 "txn-status-mana" 0x00007fb6aac6efb9 in ?? ()
15 LWP 2852 "collect_and_rem" 0x00007fb6aac6efb9 in ?? ()
16 LWP 2853 "tc-session-exp-" 0x00007fb6aac6efb9 in ?? ()
17 LWP 2854 "rpc worker-2854" 0x00007fb6aac6ead3 in ?? ()
18 LWP 2855 "rpc worker-2855" 0x00007fb6aac6ead3 in ?? ()
19 LWP 2856 "rpc worker-2856" 0x00007fb6aac6ead3 in ?? ()
20 LWP 2857 "rpc worker-2857" 0x00007fb6aac6ead3 in ?? ()
21 LWP 2858 "rpc worker-2858" 0x00007fb6aac6ead3 in ?? ()
22 LWP 2859 "rpc worker-2859" 0x00007fb6aac6ead3 in ?? ()
23 LWP 2860 "rpc worker-2860" 0x00007fb6aac6ead3 in ?? ()
24 LWP 2861 "rpc worker-2861" 0x00007fb6aac6ead3 in ?? ()
25 LWP 2862 "rpc worker-2862" 0x00007fb6aac6ead3 in ?? ()
26 LWP 2863 "rpc worker-2863" 0x00007fb6aac6ead3 in ?? ()
27 LWP 2864 "rpc worker-2864" 0x00007fb6aac6ead3 in ?? ()
28 LWP 2865 "rpc worker-2865" 0x00007fb6aac6ead3 in ?? ()
29 LWP 2866 "rpc worker-2866" 0x00007fb6aac6ead3 in ?? ()
30 LWP 2867 "rpc worker-2867" 0x00007fb6aac6ead3 in ?? ()
31 LWP 2868 "rpc worker-2868" 0x00007fb6aac6ead3 in ?? ()
32 LWP 2869 "rpc worker-2869" 0x00007fb6aac6ead3 in ?? ()
33 LWP 2870 "rpc worker-2870" 0x00007fb6aac6ead3 in ?? ()
34 LWP 2871 "rpc worker-2871" 0x00007fb6aac6ead3 in ?? ()
35 LWP 2872 "rpc worker-2872" 0x00007fb6aac6ead3 in ?? ()
36 LWP 2873 "rpc worker-2873" 0x00007fb6aac6ead3 in ?? ()
37 LWP 2874 "rpc worker-2874" 0x00007fb6aac6ead3 in ?? ()
38 LWP 2875 "rpc worker-2875" 0x00007fb6aac6ead3 in ?? ()
39 LWP 2876 "rpc worker-2876" 0x00007fb6aac6ead3 in ?? ()
40 LWP 2877 "rpc worker-2877" 0x00007fb6aac6ead3 in ?? ()
41 LWP 2878 "rpc worker-2878" 0x00007fb6aac6ead3 in ?? ()
42 LWP 2879 "rpc worker-2879" 0x00007fb6aac6ead3 in ?? ()
43 LWP 2880 "rpc worker-2880" 0x00007fb6aac6ead3 in ?? ()
44 LWP 2881 "rpc worker-2881" 0x00007fb6aac6ead3 in ?? ()
45 LWP 2882 "rpc worker-2882" 0x00007fb6aac6ead3 in ?? ()
46 LWP 2883 "rpc worker-2883" 0x00007fb6aac6ead3 in ?? ()
47 LWP 2884 "rpc worker-2884" 0x00007fb6aac6ead3 in ?? ()
48 LWP 2885 "rpc worker-2885" 0x00007fb6aac6ead3 in ?? ()
49 LWP 2886 "rpc worker-2886" 0x00007fb6aac6ead3 in ?? ()
50 LWP 2887 "rpc worker-2887" 0x00007fb6aac6ead3 in ?? ()
51 LWP 2888 "rpc worker-2888" 0x00007fb6aac6ead3 in ?? ()
52 LWP 2889 "rpc worker-2889" 0x00007fb6aac6ead3 in ?? ()
53 LWP 2890 "rpc worker-2890" 0x00007fb6aac6ead3 in ?? ()
54 LWP 2891 "rpc worker-2891" 0x00007fb6aac6ead3 in ?? ()
55 LWP 2892 "rpc worker-2892" 0x00007fb6aac6ead3 in ?? ()
56 LWP 2893 "rpc worker-2893" 0x00007fb6aac6ead3 in ?? ()
57 LWP 2894 "rpc worker-2894" 0x00007fb6aac6ead3 in ?? ()
58 LWP 2895 "rpc worker-2895" 0x00007fb6aac6ead3 in ?? ()
59 LWP 2896 "rpc worker-2896" 0x00007fb6aac6ead3 in ?? ()
60 LWP 2897 "rpc worker-2897" 0x00007fb6aac6ead3 in ?? ()
61 LWP 2898 "rpc worker-2898" 0x00007fb6aac6ead3 in ?? ()
62 LWP 2899 "rpc worker-2899" 0x00007fb6aac6ead3 in ?? ()
63 LWP 2900 "rpc worker-2900" 0x00007fb6aac6ead3 in ?? ()
64 LWP 2901 "rpc worker-2901" 0x00007fb6aac6ead3 in ?? ()
65 LWP 2902 "rpc worker-2902" 0x00007fb6aac6ead3 in ?? ()
66 LWP 2903 "rpc worker-2903" 0x00007fb6aac6ead3 in ?? ()
67 LWP 2904 "rpc worker-2904" 0x00007fb6aac6ead3 in ?? ()
68 LWP 2905 "rpc worker-2905" 0x00007fb6aac6ead3 in ?? ()
69 LWP 2906 "rpc worker-2906" 0x00007fb6aac6ead3 in ?? ()
70 LWP 2907 "rpc worker-2907" 0x00007fb6aac6ead3 in ?? ()
71 LWP 2908 "rpc worker-2908" 0x00007fb6aac6ead3 in ?? ()
72 LWP 2909 "rpc worker-2909" 0x00007fb6aac6ead3 in ?? ()
73 LWP 2910 "rpc worker-2910" 0x00007fb6aac6ead3 in ?? ()
74 LWP 2911 "rpc worker-2911" 0x00007fb6aac6ead3 in ?? ()
75 LWP 2912 "rpc worker-2912" 0x00007fb6aac6ead3 in ?? ()
76 LWP 2913 "rpc worker-2913" 0x00007fb6aac6ead3 in ?? ()
77 LWP 2914 "rpc worker-2914" 0x00007fb6aac6ead3 in ?? ()
78 LWP 2915 "rpc worker-2915" 0x00007fb6aac6ead3 in ?? ()
79 LWP 2916 "rpc worker-2916" 0x00007fb6aac6ead3 in ?? ()
80 LWP 2917 "rpc worker-2917" 0x00007fb6aac6ead3 in ?? ()
81 LWP 2918 "rpc worker-2918" 0x00007fb6aac6ead3 in ?? ()
82 LWP 2919 "rpc worker-2919" 0x00007fb6aac6ead3 in ?? ()
83 LWP 2920 "rpc worker-2920" 0x00007fb6aac6ead3 in ?? ()
84 LWP 2921 "rpc worker-2921" 0x00007fb6aac6ead3 in ?? ()
85 LWP 2922 "rpc worker-2922" 0x00007fb6aac6ead3 in ?? ()
86 LWP 2923 "rpc worker-2923" 0x00007fb6aac6ead3 in ?? ()
87 LWP 2924 "rpc worker-2924" 0x00007fb6aac6ead3 in ?? ()
88 LWP 2925 "rpc worker-2925" 0x00007fb6aac6ead3 in ?? ()
89 LWP 2926 "rpc worker-2926" 0x00007fb6aac6ead3 in ?? ()
90 LWP 2927 "rpc worker-2927" 0x00007fb6aac6ead3 in ?? ()
91 LWP 2928 "rpc worker-2928" 0x00007fb6aac6ead3 in ?? ()
92 LWP 2929 "rpc worker-2929" 0x00007fb6aac6ead3 in ?? ()
93 LWP 2930 "rpc worker-2930" 0x00007fb6aac6ead3 in ?? ()
94 LWP 2931 "rpc worker-2931" 0x00007fb6aac6ead3 in ?? ()
95 LWP 2932 "rpc worker-2932" 0x00007fb6aac6ead3 in ?? ()
96 LWP 2933 "rpc worker-2933" 0x00007fb6aac6ead3 in ?? ()
97 LWP 2934 "rpc worker-2934" 0x00007fb6aac6ead3 in ?? ()
98 LWP 2935 "rpc worker-2935" 0x00007fb6aac6ead3 in ?? ()
99 LWP 2936 "rpc worker-2936" 0x00007fb6aac6ead3 in ?? ()
100 LWP 2937 "rpc worker-2937" 0x00007fb6aac6ead3 in ?? ()
101 LWP 2938 "rpc worker-2938" 0x00007fb6aac6ead3 in ?? ()
102 LWP 2939 "rpc worker-2939" 0x00007fb6aac6ead3 in ?? ()
103 LWP 2940 "rpc worker-2940" 0x00007fb6aac6ead3 in ?? ()
104 LWP 2941 "rpc worker-2941" 0x00007fb6aac6ead3 in ?? ()
105 LWP 2942 "rpc worker-2942" 0x00007fb6aac6ead3 in ?? ()
106 LWP 2943 "rpc worker-2943" 0x00007fb6aac6ead3 in ?? ()
107 LWP 2944 "rpc worker-2944" 0x00007fb6aac6ead3 in ?? ()
108 LWP 2945 "rpc worker-2945" 0x00007fb6aac6ead3 in ?? ()
109 LWP 2946 "rpc worker-2946" 0x00007fb6aac6ead3 in ?? ()
110 LWP 2947 "rpc worker-2947" 0x00007fb6aac6ead3 in ?? ()
111 LWP 2948 "rpc worker-2948" 0x00007fb6aac6ead3 in ?? ()
112 LWP 2949 "rpc worker-2949" 0x00007fb6aac6ead3 in ?? ()
113 LWP 2950 "rpc worker-2950" 0x00007fb6aac6ead3 in ?? ()
114 LWP 2951 "rpc worker-2951" 0x00007fb6aac6ead3 in ?? ()
115 LWP 2952 "rpc worker-2952" 0x00007fb6aac6ead3 in ?? ()
116 LWP 2953 "rpc worker-2953" 0x00007fb6aac6ead3 in ?? ()
117 LWP 2954 "diag-logger-295" 0x00007fb6aac6efb9 in ?? ()
118 LWP 2955 "result-tracker-" 0x00007fb6aac6efb9 in ?? ()
119 LWP 2956 "excess-log-dele" 0x00007fb6aac6efb9 in ?? ()
120 LWP 2957 "acceptor-2957" 0x00007fb6a39da0c7 in ?? ()
121 LWP 2958 "heartbeat-2958" 0x00007fb6aac6efb9 in ?? ()
122 LWP 2959 "maintenance_sch" 0x00007fb6aac6efb9 in ?? ()
Thread 122 (LWP 2959):
#0 0x00007fb6aac6efb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000351 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007fb660552550 in ?? ()
#6 0x00000000000006a2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 2958):
#0 0x00007fb6aac6efb9 in ?? ()
#1 0xabd77187b7907576 in ?? ()
#2 0x00000000000000d3 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007fb660d6a5b0 in ?? ()
#6 0x00000000000001a6 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 2957):
#0 0x00007fb6a39da0c7 in ?? ()
#1 0x0000614000021aa8 in ?? ()
#2 0x00007fb661591d50 in ?? ()
#3 0x00007fb661591d80 in ?? ()
#4 0x00007fb661591e80 in ?? ()
#5 0x00007fb661591d70 in ?? ()
#6 0x00007fb661591de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007fb6a7864620 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 2956):
#0 0x00007fb6aac6efb9 in ?? ()
#1 0x00007fb661dabf00 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffffbbaed10 in ?? ()
#5 0x00007fb661dabef0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 2955):
#0 0x00007fb6aac6efb9 in ?? ()
#1 0x00007fb6625c40c0 in ?? ()
#2 0x00000000000000d4 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007fb6625c40b0 in ?? ()
#6 0x00000000000001a8 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 2954):
#0 0x00007fb6aac6efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 2953):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x000000000000002b in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007fb6635f3e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb6635f3e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007fb6aac6e770 in ?? ()
#10 0x00007fb6635f3e70 in ?? ()
#11 0x00000ff6cc6be7d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 2952):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x000000000000001d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007fb663e0be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb663e0be70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007fb6aac6e770 in ?? ()
#10 0x00007fb663e0be70 in ?? ()
#11 0x00000ff6cc7c17d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 2951):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 2950):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 2949):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 2948):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 2947):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 2946):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 2945):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 2944):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 2943):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 2942):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 2941):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 2940):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 2939):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 2938):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 2937):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 2936):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 2935):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 2934):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 2933):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 2932):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 2931):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 2930):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 2929):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 2928):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 2927):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 2926):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 2925):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 2924):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 2923):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 2922):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 2921):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 2920):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 2919):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 2918):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 2917):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 2916):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 2915):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 2914):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 2913):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007fb6771ace50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb6771ace70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 2912):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 2911):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 2910):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 2909):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 2908):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 2907):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 2906):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 2905):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 2904):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 2903):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 2902):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 2901):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 2900):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 2899):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 2898):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 2897):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 2896):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 2895):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 2894):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 2893):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007fb68138de50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb68138de70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 2892):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 2891):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 2890):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 2889):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 2888):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 2887):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 2886):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 2885):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 2884):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 2883):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 2882):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 2881):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 2880):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 2879):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 2878):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 2877):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 2876):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 2875):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 2874):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 2873):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007fb68b56ee50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb68b56ee70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 2872):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 2871):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 2870):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 2869):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 2868):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 2867):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 2866):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 2865):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 2864):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 2863):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 2862):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 2861):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 2860):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 2859):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 2858):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 2857):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 2856):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 2855):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 2854):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 2853):
#0 0x00007fb6aac6efb9 in ?? ()
#1 0x00007fb695771c80 in ?? ()
#2 0x0000000000000015 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001b360 in ?? ()
#5 0x00007fb695771c70 in ?? ()
#6 0x000000000000002a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 2852):
#0 0x00007fb6aac6efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 2851):
#0 0x00007fb6aac6efb9 in ?? ()
#1 0x00007fb6967af1c0 in ?? ()
#2 0x0000000000000015 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007fb6967af1b0 in ?? ()
#6 0x000000000000002a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 2850):
#0 0x00007fb6aac6ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 2849):
#0 0x00007fb6a39d8a47 in ?? ()
#1 0x00007fb6977b22e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007fb6977b22d0 in ?? ()
#4 0x00007fb6977b24e0 in ?? ()
#5 0x00007fb6977b2320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007fb6977b23a0 in ?? ()
#8 0x00007fb6a603825d in ?? ()
#9 0x3fb9734866198000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000054000000 in ?? ()
#14 0x00007fb600000000 in ?? ()
#15 0x41d9e077162c1109 in ?? ()
#16 0x00000ff752eee474 in ?? ()
#17 0x00007fb6977b2380 in ?? ()
#18 0x00007fb6a603cba3 in ?? ()
#19 0x00007fb6977b2350 in ?? ()
#20 0x3fb9734866198000 in ?? ()
#21 0x00000000977b23a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb9734866198000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 2848):
#0 0x00007fb6a39d8a47 in ?? ()
#1 0x00007fb697fb32e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007fb697fb32d0 in ?? ()
#4 0x00007fb697fb34e0 in ?? ()
#5 0x00007fb697fb3320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007fb697fb33a0 in ?? ()
#8 0x00007fb6a603825d in ?? ()
#9 0x3fb96e9bf718c000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x000000007f800000 in ?? ()
#14 0x00007fb600000000 in ?? ()
#15 0x41d9e077162c110a in ?? ()
#16 0x00000ff752fee674 in ?? ()
#17 0x00007fb697fb3380 in ?? ()
#18 0x00007fb6a603cba3 in ?? ()
#19 0x00007fb697fb3350 in ?? ()
#20 0x3fb96e9bf718c000 in ?? ()
#21 0x0000000097fb33a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb96e9bf718c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 2847):
#0 0x00007fb6a39d8a47 in ?? ()
#1 0x00007fb6987b42e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007fb6987b42d0 in ?? ()
#4 0x00007fb6987b44e0 in ?? ()
#5 0x00007fb6987b4320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007fb6987b43a0 in ?? ()
#8 0x00007fb6a603825d in ?? ()
#9 0x3fb96d12e44ed000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x00000000ea000000 in ?? ()
#14 0x00007fb600000000 in ?? ()
#15 0x41d9e077162c1109 in ?? ()
#16 0x00000ff7530ee874 in ?? ()
#17 0x00007fb6987b4380 in ?? ()
#18 0x00007fb6a603cba3 in ?? ()
#19 0x00007fb6987b4350 in ?? ()
#20 0x3fb96d12e44ed000 in ?? ()
#21 0x00000000987b43a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb96d12e44ed000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 2846):
#0 0x00007fb6a39d8a47 in ?? ()
#1 0x00007fb69a7ad2e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007fb69a7ad2d0 in ?? ()
#4 0x00007fb69a7ad4e0 in ?? ()
#5 0x00007fb69a7ad320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fb69a7ad3a0 in ?? ()
#8 0x00007fb6a603825d in ?? ()
#9 0x3fb955d8cfc4d000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x0000000092800000 in ?? ()
#14 0x00007fb600000000 in ?? ()
#15 0x41d9e077162c110c in ?? ()
#16 0x00000ff7534eda74 in ?? ()
#17 0x00007fb69a7ad380 in ?? ()
#18 0x00007fb6a603cba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 2843):
#0 0x00007fb6a39cbcb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fb69c1b7788 in ?? ()
#3 0x0000602000015cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 2842):
#0 0x00007fb6aac6efb9 in ?? ()
#1 0x00007fb69b9b6080 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007fb69b9b6070 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 2841):
#0 0x00007fb6aac729e2 in ?? ()
#1 0x00007fb69b1b3b60 in ?? ()
#2 0x00007fb69b1b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fb69b1b3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007fb69b1b3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007fb69b1b3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007fb6b0596f0c in ?? ()
#12 0x00007fb6b05867b9 in ?? ()
#13 0x00007fb6b058ada6 in ?? ()
#14 0x00007fb6b058f7e0 in ?? ()
#15 0x00007fb6b058f609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007fb6a78850be in ?? ()
#18 0x00007fb6aac686db in ?? ()
#19 0x00000ff75362e67c in ?? ()
#20 0x00007fb69b1b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 2835):
#0 0x00007fb6aac6efb9 in ?? ()
#1 0x00007fb69d1b8c40 in ?? ()
#2 0x0000000000000427 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007fb69d1b8c30 in ?? ()
#6 0x000000000000084e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 2833):
#0 0x00007fb6aac6efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 2832):
#0 0x00007fb6aac6efb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 2831):
#0 0x00007fb6aac6efb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000009 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fb69e9bc410 in ?? ()
#6 0x0000000000000012 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 2825):
#0 0x00007fb6aac72d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250111 02:58:20.100863 1102 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 5 with UUID 51a314e6744a4a11ab63fa6e76d4f780 and pid 2547
************************ BEGIN STACKS **************************
[New LWP 2551]
[New LWP 2552]
[New LWP 2553]
[New LWP 2554]
[New LWP 2562]
[New LWP 2563]
[New LWP 2564]
[New LWP 2567]
[New LWP 2568]
[New LWP 2569]
[New LWP 2570]
[New LWP 2571]
[New LWP 2572]
[New LWP 2573]
[New LWP 2574]
[New LWP 2575]
[New LWP 2576]
[New LWP 2577]
[New LWP 2578]
[New LWP 2579]
[New LWP 2580]
[New LWP 2581]
[New LWP 2582]
[New LWP 2583]
[New LWP 2584]
[New LWP 2585]
[New LWP 2586]
[New LWP 2587]
[New LWP 2588]
[New LWP 2589]
[New LWP 2590]
[New LWP 2591]
[New LWP 2592]
[New LWP 2593]
[New LWP 2594]
[New LWP 2595]
[New LWP 2596]
[New LWP 2597]
[New LWP 2598]
[New LWP 2599]
[New LWP 2600]
[New LWP 2601]
[New LWP 2602]
[New LWP 2603]
[New LWP 2604]
[New LWP 2605]
[New LWP 2606]
[New LWP 2607]
[New LWP 2608]
[New LWP 2609]
[New LWP 2610]
[New LWP 2611]
[New LWP 2612]
[New LWP 2613]
[New LWP 2614]
[New LWP 2615]
[New LWP 2616]
[New LWP 2617]
[New LWP 2618]
[New LWP 2619]
[New LWP 2620]
[New LWP 2621]
[New LWP 2622]
[New LWP 2623]
[New LWP 2624]
[New LWP 2625]
[New LWP 2626]
[New LWP 2627]
[New LWP 2628]
[New LWP 2629]
[New LWP 2630]
[New LWP 2631]
[New LWP 2632]
[New LWP 2633]
[New LWP 2634]
[New LWP 2635]
[New LWP 2636]
[New LWP 2637]
[New LWP 2638]
[New LWP 2639]
[New LWP 2640]
[New LWP 2641]
[New LWP 2642]
[New LWP 2643]
[New LWP 2644]
[New LWP 2645]
[New LWP 2646]
[New LWP 2647]
[New LWP 2648]
[New LWP 2649]
[New LWP 2650]
[New LWP 2651]
[New LWP 2652]
[New LWP 2653]
[New LWP 2654]
[New LWP 2655]
[New LWP 2656]
[New LWP 2657]
[New LWP 2658]
[New LWP 2659]
[New LWP 2660]
[New LWP 2661]
[New LWP 2662]
[New LWP 2663]
[New LWP 2664]
[New LWP 2665]
[New LWP 2666]
[New LWP 2667]
[New LWP 2668]
[New LWP 2669]
[New LWP 2670]
[New LWP 2671]
[New LWP 2672]
[New LWP 2673]
[New LWP 2674]
[New LWP 2675]
[New LWP 2676]
[New LWP 2677]
[New LWP 2678]
[New LWP 2679]
[New LWP 2684]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007ff27f373d50 in ?? ()
Id Target Id Frame
* 1 LWP 2547 "kudu" 0x00007ff27f373d50 in ?? ()
2 LWP 2551 "kudu" 0x00007ff27f36ffb9 in ?? ()
3 LWP 2552 "kudu" 0x00007ff27f36ffb9 in ?? ()
4 LWP 2553 "kudu" 0x00007ff27f36ffb9 in ?? ()
5 LWP 2554 "kernel-watcher-" 0x00007ff27f36ffb9 in ?? ()
6 LWP 2562 "ntp client-2562" 0x00007ff27f3739e2 in ?? ()
7 LWP 2563 "file cache-evic" 0x00007ff27f36ffb9 in ?? ()
8 LWP 2564 "sq_acceptor" 0x00007ff2780cccb9 in ?? ()
9 LWP 2567 "rpc reactor-256" 0x00007ff2780d9a47 in ?? ()
10 LWP 2568 "rpc reactor-256" 0x00007ff2780d9a47 in ?? ()
11 LWP 2569 "rpc reactor-256" 0x00007ff2780d9a47 in ?? ()
12 LWP 2570 "rpc reactor-257" 0x00007ff2780d9a47 in ?? ()
13 LWP 2571 "MaintenanceMgr " 0x00007ff27f36fad3 in ?? ()
14 LWP 2572 "txn-status-mana" 0x00007ff27f36ffb9 in ?? ()
15 LWP 2573 "collect_and_rem" 0x00007ff27f36ffb9 in ?? ()
16 LWP 2574 "tc-session-exp-" 0x00007ff27f36ffb9 in ?? ()
17 LWP 2575 "rpc worker-2575" 0x00007ff27f36fad3 in ?? ()
18 LWP 2576 "rpc worker-2576" 0x00007ff27f36fad3 in ?? ()
19 LWP 2577 "rpc worker-2577" 0x00007ff27f36fad3 in ?? ()
20 LWP 2578 "rpc worker-2578" 0x00007ff27f36fad3 in ?? ()
21 LWP 2579 "rpc worker-2579" 0x00007ff27f36fad3 in ?? ()
22 LWP 2580 "rpc worker-2580" 0x00007ff27f36fad3 in ?? ()
23 LWP 2581 "rpc worker-2581" 0x00007ff27f36fad3 in ?? ()
24 LWP 2582 "rpc worker-2582" 0x00007ff27f36fad3 in ?? ()
25 LWP 2583 "rpc worker-2583" 0x00007ff27f36fad3 in ?? ()
26 LWP 2584 "rpc worker-2584" 0x00007ff27f36fad3 in ?? ()
27 LWP 2585 "rpc worker-2585" 0x00007ff27f36fad3 in ?? ()
28 LWP 2586 "rpc worker-2586" 0x00007ff27f36fad3 in ?? ()
29 LWP 2587 "rpc worker-2587" 0x00007ff27f36fad3 in ?? ()
30 LWP 2588 "rpc worker-2588" 0x00007ff27f36fad3 in ?? ()
31 LWP 2589 "rpc worker-2589" 0x00007ff27f36fad3 in ?? ()
32 LWP 2590 "rpc worker-2590" 0x00007ff27f36fad3 in ?? ()
33 LWP 2591 "rpc worker-2591" 0x00007ff27f36fad3 in ?? ()
34 LWP 2592 "rpc worker-2592" 0x00007ff27f36fad3 in ?? ()
35 LWP 2593 "rpc worker-2593" 0x00007ff27f36fad3 in ?? ()
36 LWP 2594 "rpc worker-2594" 0x00007ff27f36fad3 in ?? ()
37 LWP 2595 "rpc worker-2595" 0x00007ff27f36fad3 in ?? ()
38 LWP 2596 "rpc worker-2596" 0x00007ff27f36fad3 in ?? ()
39 LWP 2597 "rpc worker-2597" 0x00007ff27f36fad3 in ?? ()
40 LWP 2598 "rpc worker-2598" 0x00007ff27f36fad3 in ?? ()
41 LWP 2599 "rpc worker-2599" 0x00007ff27f36fad3 in ?? ()
42 LWP 2600 "rpc worker-2600" 0x00007ff27f36fad3 in ?? ()
43 LWP 2601 "rpc worker-2601" 0x00007ff27f36fad3 in ?? ()
44 LWP 2602 "rpc worker-2602" 0x00007ff27f36fad3 in ?? ()
45 LWP 2603 "rpc worker-2603" 0x00007ff27f36fad3 in ?? ()
46 LWP 2604 "rpc worker-2604" 0x00007ff27f36fad3 in ?? ()
47 LWP 2605 "rpc worker-2605" 0x00007ff27f36fad3 in ?? ()
48 LWP 2606 "rpc worker-2606" 0x00007ff27f36fad3 in ?? ()
49 LWP 2607 "rpc worker-2607" 0x00007ff27f36fad3 in ?? ()
50 LWP 2608 "rpc worker-2608" 0x00007ff27f36fad3 in ?? ()
51 LWP 2609 "rpc worker-2609" 0x00007ff27f36fad3 in ?? ()
52 LWP 2610 "rpc worker-2610" 0x00007ff27f36fad3 in ?? ()
53 LWP 2611 "rpc worker-2611" 0x00007ff27f36fad3 in ?? ()
54 LWP 2612 "rpc worker-2612" 0x00007ff27f36fad3 in ?? ()
55 LWP 2613 "rpc worker-2613" 0x00007ff27f36fad3 in ?? ()
56 LWP 2614 "rpc worker-2614" 0x00007ff27f36fad3 in ?? ()
57 LWP 2615 "rpc worker-2615" 0x00007ff27f36fad3 in ?? ()
58 LWP 2616 "rpc worker-2616" 0x00007ff27f36fad3 in ?? ()
59 LWP 2617 "rpc worker-2617" 0x00007ff27f36fad3 in ?? ()
60 LWP 2618 "rpc worker-2618" 0x00007ff27f36fad3 in ?? ()
61 LWP 2619 "rpc worker-2619" 0x00007ff27f36fad3 in ?? ()
62 LWP 2620 "rpc worker-2620" 0x00007ff27f36fad3 in ?? ()
63 LWP 2621 "rpc worker-2621" 0x00007ff27f36fad3 in ?? ()
64 LWP 2622 "rpc worker-2622" 0x00007ff27f36fad3 in ?? ()
65 LWP 2623 "rpc worker-2623" 0x00007ff27f36fad3 in ?? ()
66 LWP 2624 "rpc worker-2624" 0x00007ff27f36fad3 in ?? ()
67 LWP 2625 "rpc worker-2625" 0x00007ff27f36fad3 in ?? ()
68 LWP 2626 "rpc worker-2626" 0x00007ff27f36fad3 in ?? ()
69 LWP 2627 "rpc worker-2627" 0x00007ff27f36fad3 in ?? ()
70 LWP 2628 "rpc worker-2628" 0x00007ff27f36fad3 in ?? ()
71 LWP 2629 "rpc worker-2629" 0x00007ff27f36fad3 in ?? ()
72 LWP 2630 "rpc worker-2630" 0x00007ff27f36fad3 in ?? ()
73 LWP 2631 "rpc worker-2631" 0x00007ff27f36fad3 in ?? ()
74 LWP 2632 "rpc worker-2632" 0x00007ff27f36fad3 in ?? ()
75 LWP 2633 "rpc worker-2633" 0x00007ff27f36fad3 in ?? ()
76 LWP 2634 "rpc worker-2634" 0x00007ff27f36fad3 in ?? ()
77 LWP 2635 "rpc worker-2635" 0x00007ff27f36fad3 in ?? ()
78 LWP 2636 "rpc worker-2636" 0x00007ff27f36fad3 in ?? ()
79 LWP 2637 "rpc worker-2637" 0x00007ff27f36fad3 in ?? ()
80 LWP 2638 "rpc worker-2638" 0x00007ff27f36fad3 in ?? ()
81 LWP 2639 "rpc worker-2639" 0x00007ff27f36fad3 in ?? ()
82 LWP 2640 "rpc worker-2640" 0x00007ff27f36fad3 in ?? ()
83 LWP 2641 "rpc worker-2641" 0x00007ff27f36fad3 in ?? ()
84 LWP 2642 "rpc worker-2642" 0x00007ff27f36fad3 in ?? ()
85 LWP 2643 "rpc worker-2643" 0x00007ff27f36fad3 in ?? ()
86 LWP 2644 "rpc worker-2644" 0x00007ff27f36fad3 in ?? ()
87 LWP 2645 "rpc worker-2645" 0x00007ff27f36fad3 in ?? ()
88 LWP 2646 "rpc worker-2646" 0x00007ff27f36fad3 in ?? ()
89 LWP 2647 "rpc worker-2647" 0x00007ff27f36fad3 in ?? ()
90 LWP 2648 "rpc worker-2648" 0x00007ff27f36fad3 in ?? ()
91 LWP 2649 "rpc worker-2649" 0x00007ff27f36fad3 in ?? ()
92 LWP 2650 "rpc worker-2650" 0x00007ff27f36fad3 in ?? ()
93 LWP 2651 "rpc worker-2651" 0x00007ff27f36fad3 in ?? ()
94 LWP 2652 "rpc worker-2652" 0x00007ff27f36fad3 in ?? ()
95 LWP 2653 "rpc worker-2653" 0x00007ff27f36fad3 in ?? ()
96 LWP 2654 "rpc worker-2654" 0x00007ff27f36fad3 in ?? ()
97 LWP 2655 "rpc worker-2655" 0x00007ff27f36fad3 in ?? ()
98 LWP 2656 "rpc worker-2656" 0x00007ff27f36fad3 in ?? ()
99 LWP 2657 "rpc worker-2657" 0x00007ff27f36fad3 in ?? ()
100 LWP 2658 "rpc worker-2658" 0x00007ff27f36fad3 in ?? ()
101 LWP 2659 "rpc worker-2659" 0x00007ff27f36fad3 in ?? ()
102 LWP 2660 "rpc worker-2660" 0x00007ff27f36fad3 in ?? ()
103 LWP 2661 "rpc worker-2661" 0x00007ff27f36fad3 in ?? ()
104 LWP 2662 "rpc worker-2662" 0x00007ff27f36fad3 in ?? ()
105 LWP 2663 "rpc worker-2663" 0x00007ff27f36fad3 in ?? ()
106 LWP 2664 "rpc worker-2664" 0x00007ff27f36fad3 in ?? ()
107 LWP 2665 "rpc worker-2665" 0x00007ff27f36fad3 in ?? ()
108 LWP 2666 "rpc worker-2666" 0x00007ff27f36fad3 in ?? ()
109 LWP 2667 "rpc worker-2667" 0x00007ff27f36fad3 in ?? ()
110 LWP 2668 "rpc worker-2668" 0x00007ff27f36fad3 in ?? ()
111 LWP 2669 "rpc worker-2669" 0x00007ff27f36fad3 in ?? ()
112 LWP 2670 "rpc worker-2670" 0x00007ff27f36fad3 in ?? ()
113 LWP 2671 "rpc worker-2671" 0x00007ff27f36fad3 in ?? ()
114 LWP 2672 "rpc worker-2672" 0x00007ff27f36fad3 in ?? ()
115 LWP 2673 "rpc worker-2673" 0x00007ff27f36fad3 in ?? ()
116 LWP 2674 "rpc worker-2674" 0x00007ff27f36fad3 in ?? ()
117 LWP 2675 "diag-logger-267" 0x00007ff27f36ffb9 in ?? ()
118 LWP 2676 "result-tracker-" 0x00007ff27f36ffb9 in ?? ()
119 LWP 2677 "excess-log-dele" 0x00007ff27f36ffb9 in ?? ()
120 LWP 2678 "acceptor-2678" 0x00007ff2780db0c7 in ?? ()
121 LWP 2679 "heartbeat-2679" 0x00007ff27f36ffb9 in ?? ()
122 LWP 2684 "maintenance_sch" 0x00007ff27f36ffb9 in ?? ()
Thread 122 (LWP 2684):
#0 0x00007ff27f36ffb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000365 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007ff233c2a550 in ?? ()
#6 0x00000000000006ca in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 2679):
#0 0x00007ff27f36ffb9 in ?? ()
#1 0xabd77187b7907576 in ?? ()
#2 0x00000000000000d9 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007ff234c515b0 in ?? ()
#6 0x00000000000001b2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 2678):
#0 0x00007ff2780db0c7 in ?? ()
#1 0x00006140000218a8 in ?? ()
#2 0x00007ff235478d50 in ?? ()
#3 0x00007ff235478d80 in ?? ()
#4 0x00007ff235478e80 in ?? ()
#5 0x00007ff235478d70 in ?? ()
#6 0x00007ff235478de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007ff27bf65620 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 2677):
#0 0x00007ff27f36ffb9 in ?? ()
#1 0x00007ff235c92f00 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffccc1fd7f0 in ?? ()
#5 0x00007ff235c92ef0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 2676):
#0 0x00007ff27f36ffb9 in ?? ()
#1 0x00007ff2364ab0c0 in ?? ()
#2 0x00000000000000d9 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007ff2364ab0b0 in ?? ()
#6 0x00000000000001b2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 2675):
#0 0x00007ff27f36ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 2674):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000023 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007ff2374dae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff2374dae70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007ff27f36f770 in ?? ()
#10 0x00007ff2374dae70 in ?? ()
#11 0x00000ffe46e9b5d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 2673):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000025 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007ff237cf2e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff237cf2e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007ff27f36f770 in ?? ()
#10 0x00007ff237cf2e70 in ?? ()
#11 0x00000ffe46f9e5d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 2672):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 2671):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 2670):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 2669):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 2668):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 2667):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 2666):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 2665):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 2664):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 2663):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 2662):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 2661):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 2660):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 2659):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 2658):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 2657):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 2656):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 2655):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 2654):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 2653):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 2652):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 2651):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 2650):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 2649):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 2648):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 2647):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 2646):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 2645):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 2644):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 2643):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 2642):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 2641):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 2640):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 2639):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 2638):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 2637):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 2636):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 2635):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 2634):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007ff24b8abe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff24b8abe70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 2633):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 2632):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 2631):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 2630):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 2629):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 2628):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 2627):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 2626):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 2625):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 2624):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 2623):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 2622):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 2621):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 2620):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 2619):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 2618):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 2617):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 2616):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 2615):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 2614):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007ff255a8ce50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff255a8ce70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 2613):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 2612):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 2611):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 2610):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 2609):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 2608):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 2607):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 2606):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 2605):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 2604):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 2603):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 2602):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 2601):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 2600):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 2599):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 2598):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 2597):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 2596):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 2595):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 2594):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007ff25fc6de50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff25fc6de70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 2593):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 2592):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 2591):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 2590):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 2589):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 2588):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 2587):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 2586):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 2585):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 2584):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 2583):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 2582):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 2581):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 2580):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 2579):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 2578):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 2577):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 2576):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 2575):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 2574):
#0 0x00007ff27f36ffb9 in ?? ()
#1 0x00007ff269cc3c80 in ?? ()
#2 0x0000000000000015 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001b360 in ?? ()
#5 0x00007ff269cc3c70 in ?? ()
#6 0x000000000000002a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 2573):
#0 0x00007ff27f36ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 2572):
#0 0x00007ff27f36ffb9 in ?? ()
#1 0x00007ff26acc51c0 in ?? ()
#2 0x0000000000000015 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007ff26acc51b0 in ?? ()
#6 0x000000000000002a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 2571):
#0 0x00007ff27f36fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 2570):
#0 0x00007ff2780d9a47 in ?? ()
#1 0x00007ff26bcc82e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007ff26bcc82d0 in ?? ()
#4 0x00007ff26bcc84e0 in ?? ()
#5 0x00007ff26bcc8320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007ff26bcc83a0 in ?? ()
#8 0x00007ff27a73925d in ?? ()
#9 0x3fb701bfda11f000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x00000000a3a00000 in ?? ()
#14 0x00007ff200000000 in ?? ()
#15 0x41d9e077162c1108 in ?? ()
#16 0x00000ffecd791074 in ?? ()
#17 0x00007ff26bcc8380 in ?? ()
#18 0x00007ff27a73dba3 in ?? ()
#19 0x00007ff26bcc8350 in ?? ()
#20 0x3fb701bfda11f000 in ?? ()
#21 0x000000006bcc83a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb701bfda11f000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 2569):
#0 0x00007ff2780d9a47 in ?? ()
#1 0x00007ff26c4c92e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007ff26c4c92d0 in ?? ()
#4 0x00007ff26c4c94e0 in ?? ()
#5 0x00007ff26c4c9320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007ff26c4c93a0 in ?? ()
#8 0x00007ff27a73925d in ?? ()
#9 0x3fb96d7fd7d4a000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x000000008f000000 in ?? ()
#14 0x00007ff200000000 in ?? ()
#15 0x41d9e077162c110c in ?? ()
#16 0x00000ffecd891274 in ?? ()
#17 0x00007ff26c4c9380 in ?? ()
#18 0x00007ff27a73dba3 in ?? ()
#19 0x00007ff26c4c9350 in ?? ()
#20 0x3fb96d7fd7d4a000 in ?? ()
#21 0x000000006c4c93a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb96d7fd7d4a000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 2568):
#0 0x00007ff2780d9a47 in ?? ()
#1 0x00007ff26ccca2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007ff26ccca2d0 in ?? ()
#4 0x00007ff26ccca4e0 in ?? ()
#5 0x00007ff26ccca320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007ff26ccca3a0 in ?? ()
#8 0x00007ff27a73925d in ?? ()
#9 0x3fb98cd8f4a10000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x000000009a800000 in ?? ()
#14 0x00007ff200000000 in ?? ()
#15 0x41d9e077162c110b in ?? ()
#16 0x00000ffecd991474 in ?? ()
#17 0x00007ff26ccca380 in ?? ()
#18 0x00007ff27a73dba3 in ?? ()
#19 0x00007ff26ccca350 in ?? ()
#20 0x3fb98cd8f4a10000 in ?? ()
#21 0x000000006ccca3a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb98cd8f4a10000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 2567):
#0 0x00007ff2780d9a47 in ?? ()
#1 0x00007ff26e8b32e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007ff26e8b32d0 in ?? ()
#4 0x00007ff26e8b34e0 in ?? ()
#5 0x00007ff26e8b3320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007ff26e8b33a0 in ?? ()
#8 0x00007ff27a73925d in ?? ()
#9 0x3fb98b7224406000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x0000000013000000 in ?? ()
#14 0x00007ff200000000 in ?? ()
#15 0x41d9e077162c110b in ?? ()
#16 0x00000ffecdd0e674 in ?? ()
#17 0x00007ff26e8b3380 in ?? ()
#18 0x00007ff27a73dba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 2564):
#0 0x00007ff2780cccb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007ff2708b7788 in ?? ()
#3 0x0000602000017c90 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 2563):
#0 0x00007ff27f36ffb9 in ?? ()
#1 0x00007ff2700b6080 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007ff2700b6070 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 2562):
#0 0x00007ff27f3739e2 in ?? ()
#1 0x00007ff26f0b2b60 in ?? ()
#2 0x00007ff26f0b2c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007ff26f0b2bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007ff26f0b2c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007ff26f0b2c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007ff284c97f0c in ?? ()
#12 0x00007ff284c877b9 in ?? ()
#13 0x00007ff284c8bda6 in ?? ()
#14 0x00007ff284c907e0 in ?? ()
#15 0x00007ff284c90609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007ff27bf860be in ?? ()
#18 0x00007ff27f3696db in ?? ()
#19 0x00000ffecde0e47c in ?? ()
#20 0x00007ff26f0b2400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 2554):
#0 0x00007ff27f36ffb9 in ?? ()
#1 0x00007ff2718b8c40 in ?? ()
#2 0x0000000000000440 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007ff2718b8c30 in ?? ()
#6 0x0000000000000880 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 2553):
#0 0x00007ff27f36ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 2552):
#0 0x00007ff27f36ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 2551):
#0 0x00007ff27f36ffb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000009 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007ff2730bc410 in ?? ()
#6 0x0000000000000012 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 2547):
#0 0x00007ff27f373d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250111 02:58:20.899144 1102 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 6 with UUID 34d6de1ad631433e9b596bf231a8c9d5 and pid 2687
************************ BEGIN STACKS **************************
[New LWP 2690]
[New LWP 2691]
[New LWP 2692]
[New LWP 2693]
[New LWP 2699]
[New LWP 2700]
[New LWP 2701]
[New LWP 2705]
[New LWP 2706]
[New LWP 2707]
[New LWP 2708]
[New LWP 2709]
[New LWP 2710]
[New LWP 2711]
[New LWP 2712]
[New LWP 2713]
[New LWP 2714]
[New LWP 2715]
[New LWP 2716]
[New LWP 2718]
[New LWP 2719]
[New LWP 2720]
[New LWP 2721]
[New LWP 2722]
[New LWP 2723]
[New LWP 2724]
[New LWP 2725]
[New LWP 2726]
[New LWP 2727]
[New LWP 2728]
[New LWP 2729]
[New LWP 2730]
[New LWP 2731]
[New LWP 2732]
[New LWP 2733]
[New LWP 2734]
[New LWP 2735]
[New LWP 2736]
[New LWP 2737]
[New LWP 2738]
[New LWP 2739]
[New LWP 2740]
[New LWP 2741]
[New LWP 2742]
[New LWP 2743]
[New LWP 2744]
[New LWP 2745]
[New LWP 2746]
[New LWP 2747]
[New LWP 2748]
[New LWP 2749]
[New LWP 2750]
[New LWP 2751]
[New LWP 2752]
[New LWP 2753]
[New LWP 2754]
[New LWP 2755]
[New LWP 2756]
[New LWP 2757]
[New LWP 2758]
[New LWP 2759]
[New LWP 2760]
[New LWP 2761]
[New LWP 2762]
[New LWP 2763]
[New LWP 2764]
[New LWP 2765]
[New LWP 2766]
[New LWP 2767]
[New LWP 2768]
[New LWP 2769]
[New LWP 2770]
[New LWP 2771]
[New LWP 2772]
[New LWP 2773]
[New LWP 2774]
[New LWP 2775]
[New LWP 2776]
[New LWP 2777]
[New LWP 2778]
[New LWP 2779]
[New LWP 2780]
[New LWP 2781]
[New LWP 2782]
[New LWP 2783]
[New LWP 2784]
[New LWP 2785]
[New LWP 2786]
[New LWP 2787]
[New LWP 2788]
[New LWP 2789]
[New LWP 2790]
[New LWP 2791]
[New LWP 2792]
[New LWP 2793]
[New LWP 2794]
[New LWP 2795]
[New LWP 2796]
[New LWP 2797]
[New LWP 2798]
[New LWP 2799]
[New LWP 2800]
[New LWP 2801]
[New LWP 2802]
[New LWP 2803]
[New LWP 2804]
[New LWP 2805]
[New LWP 2806]
[New LWP 2807]
[New LWP 2808]
[New LWP 2809]
[New LWP 2810]
[New LWP 2811]
[New LWP 2812]
[New LWP 2813]
[New LWP 2814]
[New LWP 2815]
[New LWP 2816]
[New LWP 2817]
[New LWP 2818]
[New LWP 2821]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007fad5d46bd50 in ?? ()
Id Target Id Frame
* 1 LWP 2687 "kudu" 0x00007fad5d46bd50 in ?? ()
2 LWP 2690 "kudu" 0x00007fad5d467fb9 in ?? ()
3 LWP 2691 "kudu" 0x00007fad5d467fb9 in ?? ()
4 LWP 2692 "kudu" 0x00007fad5d467fb9 in ?? ()
5 LWP 2693 "kernel-watcher-" 0x00007fad5d467fb9 in ?? ()
6 LWP 2699 "ntp client-2699" 0x00007fad5d46b9e2 in ?? ()
7 LWP 2700 "file cache-evic" 0x00007fad5d467fb9 in ?? ()
8 LWP 2701 "sq_acceptor" 0x00007fad561c4cb9 in ?? ()
9 LWP 2705 "rpc reactor-270" 0x00007fad561d1a47 in ?? ()
10 LWP 2706 "rpc reactor-270" 0x00007fad561d1a47 in ?? ()
11 LWP 2707 "rpc reactor-270" 0x00007fad561d1a47 in ?? ()
12 LWP 2708 "rpc reactor-270" 0x00007fad561d1a47 in ?? ()
13 LWP 2709 "MaintenanceMgr " 0x00007fad5d467ad3 in ?? ()
14 LWP 2710 "txn-status-mana" 0x00007fad5d467fb9 in ?? ()
15 LWP 2711 "collect_and_rem" 0x00007fad5d467fb9 in ?? ()
16 LWP 2712 "tc-session-exp-" 0x00007fad5d467fb9 in ?? ()
17 LWP 2713 "rpc worker-2713" 0x00007fad5d467ad3 in ?? ()
18 LWP 2714 "rpc worker-2714" 0x00007fad5d467ad3 in ?? ()
19 LWP 2715 "rpc worker-2715" 0x00007fad5d467ad3 in ?? ()
20 LWP 2716 "rpc worker-2716" 0x00007fad5d467ad3 in ?? ()
21 LWP 2718 "rpc worker-2718" 0x00007fad5d467ad3 in ?? ()
22 LWP 2719 "rpc worker-2719" 0x00007fad5d467ad3 in ?? ()
23 LWP 2720 "rpc worker-2720" 0x00007fad5d467ad3 in ?? ()
24 LWP 2721 "rpc worker-2721" 0x00007fad5d467ad3 in ?? ()
25 LWP 2722 "rpc worker-2722" 0x00007fad5d467ad3 in ?? ()
26 LWP 2723 "rpc worker-2723" 0x00007fad5d467ad3 in ?? ()
27 LWP 2724 "rpc worker-2724" 0x00007fad5d467ad3 in ?? ()
28 LWP 2725 "rpc worker-2725" 0x00007fad5d467ad3 in ?? ()
29 LWP 2726 "rpc worker-2726" 0x00007fad5d467ad3 in ?? ()
30 LWP 2727 "rpc worker-2727" 0x00007fad5d467ad3 in ?? ()
31 LWP 2728 "rpc worker-2728" 0x00007fad5d467ad3 in ?? ()
32 LWP 2729 "rpc worker-2729" 0x00007fad5d467ad3 in ?? ()
33 LWP 2730 "rpc worker-2730" 0x00007fad5d467ad3 in ?? ()
34 LWP 2731 "rpc worker-2731" 0x00007fad5d467ad3 in ?? ()
35 LWP 2732 "rpc worker-2732" 0x00007fad5d467ad3 in ?? ()
36 LWP 2733 "rpc worker-2733" 0x00007fad5d467ad3 in ?? ()
37 LWP 2734 "rpc worker-2734" 0x00007fad5d467ad3 in ?? ()
38 LWP 2735 "rpc worker-2735" 0x00007fad5d467ad3 in ?? ()
39 LWP 2736 "rpc worker-2736" 0x00007fad5d467ad3 in ?? ()
40 LWP 2737 "rpc worker-2737" 0x00007fad5d467ad3 in ?? ()
41 LWP 2738 "rpc worker-2738" 0x00007fad5d467ad3 in ?? ()
42 LWP 2739 "rpc worker-2739" 0x00007fad5d467ad3 in ?? ()
43 LWP 2740 "rpc worker-2740" 0x00007fad5d467ad3 in ?? ()
44 LWP 2741 "rpc worker-2741" 0x00007fad5d467ad3 in ?? ()
45 LWP 2742 "rpc worker-2742" 0x00007fad5d467ad3 in ?? ()
46 LWP 2743 "rpc worker-2743" 0x00007fad5d467ad3 in ?? ()
47 LWP 2744 "rpc worker-2744" 0x00007fad5d467ad3 in ?? ()
48 LWP 2745 "rpc worker-2745" 0x00007fad5d467ad3 in ?? ()
49 LWP 2746 "rpc worker-2746" 0x00007fad5d467ad3 in ?? ()
50 LWP 2747 "rpc worker-2747" 0x00007fad5d467ad3 in ?? ()
51 LWP 2748 "rpc worker-2748" 0x00007fad5d467ad3 in ?? ()
52 LWP 2749 "rpc worker-2749" 0x00007fad5d467ad3 in ?? ()
53 LWP 2750 "rpc worker-2750" 0x00007fad5d467ad3 in ?? ()
54 LWP 2751 "rpc worker-2751" 0x00007fad5d467ad3 in ?? ()
55 LWP 2752 "rpc worker-2752" 0x00007fad5d467ad3 in ?? ()
56 LWP 2753 "rpc worker-2753" 0x00007fad5d467ad3 in ?? ()
57 LWP 2754 "rpc worker-2754" 0x00007fad5d467ad3 in ?? ()
58 LWP 2755 "rpc worker-2755" 0x00007fad5d467ad3 in ?? ()
59 LWP 2756 "rpc worker-2756" 0x00007fad5d467ad3 in ?? ()
60 LWP 2757 "rpc worker-2757" 0x00007fad5d467ad3 in ?? ()
61 LWP 2758 "rpc worker-2758" 0x00007fad5d467ad3 in ?? ()
62 LWP 2759 "rpc worker-2759" 0x00007fad5d467ad3 in ?? ()
63 LWP 2760 "rpc worker-2760" 0x00007fad5d467ad3 in ?? ()
64 LWP 2761 "rpc worker-2761" 0x00007fad5d467ad3 in ?? ()
65 LWP 2762 "rpc worker-2762" 0x00007fad5d467ad3 in ?? ()
66 LWP 2763 "rpc worker-2763" 0x00007fad5d467ad3 in ?? ()
67 LWP 2764 "rpc worker-2764" 0x00007fad5d467ad3 in ?? ()
68 LWP 2765 "rpc worker-2765" 0x00007fad5d467ad3 in ?? ()
69 LWP 2766 "rpc worker-2766" 0x00007fad5d467ad3 in ?? ()
70 LWP 2767 "rpc worker-2767" 0x00007fad5d467ad3 in ?? ()
71 LWP 2768 "rpc worker-2768" 0x00007fad5d467ad3 in ?? ()
72 LWP 2769 "rpc worker-2769" 0x00007fad5d467ad3 in ?? ()
73 LWP 2770 "rpc worker-2770" 0x00007fad5d467ad3 in ?? ()
74 LWP 2771 "rpc worker-2771" 0x00007fad5d467ad3 in ?? ()
75 LWP 2772 "rpc worker-2772" 0x00007fad5d467ad3 in ?? ()
76 LWP 2773 "rpc worker-2773" 0x00007fad5d467ad3 in ?? ()
77 LWP 2774 "rpc worker-2774" 0x00007fad5d467ad3 in ?? ()
78 LWP 2775 "rpc worker-2775" 0x00007fad5d467ad3 in ?? ()
79 LWP 2776 "rpc worker-2776" 0x00007fad5d467ad3 in ?? ()
80 LWP 2777 "rpc worker-2777" 0x00007fad5d467ad3 in ?? ()
81 LWP 2778 "rpc worker-2778" 0x00007fad5d467ad3 in ?? ()
82 LWP 2779 "rpc worker-2779" 0x00007fad5d467ad3 in ?? ()
83 LWP 2780 "rpc worker-2780" 0x00007fad5d467ad3 in ?? ()
84 LWP 2781 "rpc worker-2781" 0x00007fad5d467ad3 in ?? ()
85 LWP 2782 "rpc worker-2782" 0x00007fad5d467ad3 in ?? ()
86 LWP 2783 "rpc worker-2783" 0x00007fad5d467ad3 in ?? ()
87 LWP 2784 "rpc worker-2784" 0x00007fad5d467ad3 in ?? ()
88 LWP 2785 "rpc worker-2785" 0x00007fad5d467ad3 in ?? ()
89 LWP 2786 "rpc worker-2786" 0x00007fad5d467ad3 in ?? ()
90 LWP 2787 "rpc worker-2787" 0x00007fad5d467ad3 in ?? ()
91 LWP 2788 "rpc worker-2788" 0x00007fad5d467ad3 in ?? ()
92 LWP 2789 "rpc worker-2789" 0x00007fad5d467ad3 in ?? ()
93 LWP 2790 "rpc worker-2790" 0x00007fad5d467ad3 in ?? ()
94 LWP 2791 "rpc worker-2791" 0x00007fad5d467ad3 in ?? ()
95 LWP 2792 "rpc worker-2792" 0x00007fad5d467ad3 in ?? ()
96 LWP 2793 "rpc worker-2793" 0x00007fad5d467ad3 in ?? ()
97 LWP 2794 "rpc worker-2794" 0x00007fad5d467ad3 in ?? ()
98 LWP 2795 "rpc worker-2795" 0x00007fad5d467ad3 in ?? ()
99 LWP 2796 "rpc worker-2796" 0x00007fad5d467ad3 in ?? ()
100 LWP 2797 "rpc worker-2797" 0x00007fad5d467ad3 in ?? ()
101 LWP 2798 "rpc worker-2798" 0x00007fad5d467ad3 in ?? ()
102 LWP 2799 "rpc worker-2799" 0x00007fad5d467ad3 in ?? ()
103 LWP 2800 "rpc worker-2800" 0x00007fad5d467ad3 in ?? ()
104 LWP 2801 "rpc worker-2801" 0x00007fad5d467ad3 in ?? ()
105 LWP 2802 "rpc worker-2802" 0x00007fad5d467ad3 in ?? ()
106 LWP 2803 "rpc worker-2803" 0x00007fad5d467ad3 in ?? ()
107 LWP 2804 "rpc worker-2804" 0x00007fad5d467ad3 in ?? ()
108 LWP 2805 "rpc worker-2805" 0x00007fad5d467ad3 in ?? ()
109 LWP 2806 "rpc worker-2806" 0x00007fad5d467ad3 in ?? ()
110 LWP 2807 "rpc worker-2807" 0x00007fad5d467ad3 in ?? ()
111 LWP 2808 "rpc worker-2808" 0x00007fad5d467ad3 in ?? ()
112 LWP 2809 "rpc worker-2809" 0x00007fad5d467ad3 in ?? ()
113 LWP 2810 "rpc worker-2810" 0x00007fad5d467ad3 in ?? ()
114 LWP 2811 "rpc worker-2811" 0x00007fad5d467ad3 in ?? ()
115 LWP 2812 "rpc worker-2812" 0x00007fad5d467ad3 in ?? ()
116 LWP 2813 "rpc worker-2813" 0x00007fad5d467ad3 in ?? ()
117 LWP 2814 "diag-logger-281" 0x00007fad5d467fb9 in ?? ()
118 LWP 2815 "result-tracker-" 0x00007fad5d467fb9 in ?? ()
119 LWP 2816 "excess-log-dele" 0x00007fad5d467fb9 in ?? ()
120 LWP 2817 "acceptor-2817" 0x00007fad561d30c7 in ?? ()
121 LWP 2818 "heartbeat-2818" 0x00007fad5d467fb9 in ?? ()
122 LWP 2821 "maintenance_sch" 0x00007fad5d467fb9 in ?? ()
Thread 122 (LWP 2821):
#0 0x00007fad5d467fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000035f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007fad12d4c550 in ?? ()
#6 0x00000000000006be in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 2818):
#0 0x00007fad5d467fb9 in ?? ()
#1 0xabd77187b7907576 in ?? ()
#2 0x00000000000000d7 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007fad135635b0 in ?? ()
#6 0x00000000000001ae in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 2817):
#0 0x00007fad561d30c7 in ?? ()
#1 0x0000614000021aa8 in ?? ()
#2 0x00007fad13d8ad50 in ?? ()
#3 0x00007fad13d8ad80 in ?? ()
#4 0x00007fad13d8ae80 in ?? ()
#5 0x00007fad13d8ad70 in ?? ()
#6 0x00007fad13d8ade0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007fad5a05d620 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 2816):
#0 0x00007fad5d467fb9 in ?? ()
#1 0x00007fad145a4f00 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff40bd3c50 in ?? ()
#5 0x00007fad145a4ef0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 2815):
#0 0x00007fad5d467fb9 in ?? ()
#1 0x00007fad14dbd0c0 in ?? ()
#2 0x00000000000000d8 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007fad14dbd0b0 in ?? ()
#6 0x00000000000001b0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 2814):
#0 0x00007fad5d467fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 2813):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x000000000000002f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007fad15dece50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fad15dece70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007fad5d467770 in ?? ()
#10 0x00007fad15dece70 in ?? ()
#11 0x00000ff5a2bbd9d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 2812):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000019 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007fad16604e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fad16604e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007fad5d467770 in ?? ()
#10 0x00007fad16604e70 in ?? ()
#11 0x00000ff5a2cc09d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 2811):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 2810):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 2809):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 2808):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 2807):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 2806):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 2805):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 2804):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 2803):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 2802):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 2801):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 2800):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 2799):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 2798):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 2797):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 2796):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 2795):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 2794):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 2793):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 2792):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 2791):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 2790):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 2789):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 2788):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 2787):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 2786):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 2785):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 2784):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 2783):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 2782):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 2781):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 2780):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 2779):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 2778):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 2777):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 2776):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 2775):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 2774):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 2773):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007fad2a1bde50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fad2a1bde70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 2772):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 2771):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 2770):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 2769):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 2768):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 2767):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 2766):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 2765):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 2764):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 2763):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 2762):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 2761):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 2760):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 2759):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 2758):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 2757):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 2756):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 2755):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 2754):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 2753):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007fad3439ee50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fad3439ee70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 2752):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 2751):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 2750):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 2749):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 2748):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 2747):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 2746):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 2745):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 2744):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 2743):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 2742):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 2741):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 2740):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 2739):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 2738):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 2737):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 2736):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 2735):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 2734):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 2733):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000012 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007fad3dd67e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fad3dd67e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 2732):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 2731):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 2730):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 2729):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 2728):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 2727):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 2726):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 2725):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 2724):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 2723):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 2722):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 2721):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 2720):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 2719):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 2718):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 2716):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 2715):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 2714):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 2713):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 2712):
#0 0x00007fad5d467fb9 in ?? ()
#1 0x00007fad47f69c80 in ?? ()
#2 0x0000000000000015 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001b360 in ?? ()
#5 0x00007fad47f69c70 in ?? ()
#6 0x000000000000002a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 2711):
#0 0x00007fad5d467fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 2710):
#0 0x00007fad5d467fb9 in ?? ()
#1 0x00007fad48f9f1c0 in ?? ()
#2 0x0000000000000015 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007fad48f9f1b0 in ?? ()
#6 0x000000000000002a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 2709):
#0 0x00007fad5d467ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 2708):
#0 0x00007fad561d1a47 in ?? ()
#1 0x00007fad49fb22e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007fad49fb22d0 in ?? ()
#4 0x00007fad49fb24e0 in ?? ()
#5 0x00007fad49fb2320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007fad49fb23a0 in ?? ()
#8 0x00007fad5883125d in ?? ()
#9 0x3fb972c300d18000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000080800000 in ?? ()
#14 0x00007fad00000000 in ?? ()
#15 0x41d9e077162c110a in ?? ()
#16 0x00000ff6293ee474 in ?? ()
#17 0x00007fad49fb2380 in ?? ()
#18 0x00007fad58835ba3 in ?? ()
#19 0x00007fad49fb2350 in ?? ()
#20 0x3fb972c300d18000 in ?? ()
#21 0x0000000049fb23a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb972c300d18000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 2707):
#0 0x00007fad561d1a47 in ?? ()
#1 0x00007fad4a7b32e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007fad4a7b32d0 in ?? ()
#4 0x00007fad4a7b34e0 in ?? ()
#5 0x00007fad4a7b3320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007fad4a7b33a0 in ?? ()
#8 0x00007fad5883125d in ?? ()
#9 0x3fb969af465f8000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x000000009e000000 in ?? ()
#14 0x00007fad00000000 in ?? ()
#15 0x41d9e077162c110c in ?? ()
#16 0x00000ff6294ee674 in ?? ()
#17 0x00007fad4a7b3380 in ?? ()
#18 0x00007fad58835ba3 in ?? ()
#19 0x00007fad4a7b3350 in ?? ()
#20 0x3fb969af465f8000 in ?? ()
#21 0x000000004a7b33a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb969af465f8000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 2706):
#0 0x00007fad561d1a47 in ?? ()
#1 0x00007fad4afb42e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007fad4afb42d0 in ?? ()
#4 0x00007fad4afb44e0 in ?? ()
#5 0x00007fad4afb4320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007fad4afb43a0 in ?? ()
#8 0x00007fad5883125d in ?? ()
#9 0x3fb96f0d9d358000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x00000000f2000000 in ?? ()
#14 0x00007fad00000000 in ?? ()
#15 0x41d9e077162c110a in ?? ()
#16 0x00000ff6295ee874 in ?? ()
#17 0x00007fad4afb4380 in ?? ()
#18 0x00007fad58835ba3 in ?? ()
#19 0x00007fad4afb4350 in ?? ()
#20 0x3fb96f0d9d358000 in ?? ()
#21 0x000000004afb43a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb96f0d9d358000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 2705):
#0 0x00007fad561d1a47 in ?? ()
#1 0x00007fad4cfad2e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007fad4cfad2d0 in ?? ()
#4 0x00007fad4cfad4e0 in ?? ()
#5 0x00007fad4cfad320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fad4cfad3a0 in ?? ()
#8 0x00007fad5883125d in ?? ()
#9 0x3fb96f63d9574000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x0000000093000000 in ?? ()
#14 0x00007fad00000000 in ?? ()
#15 0x41d9e077162c110d in ?? ()
#16 0x00000ff6299eda74 in ?? ()
#17 0x00007fad4cfad380 in ?? ()
#18 0x00007fad58835ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 2701):
#0 0x00007fad561c4cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fad4e9b7788 in ?? ()
#3 0x0000602000017750 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 2700):
#0 0x00007fad5d467fb9 in ?? ()
#1 0x00007fad4e1b6080 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007fad4e1b6070 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 2699):
#0 0x00007fad5d46b9e2 in ?? ()
#1 0x00007fad4d9b3b60 in ?? ()
#2 0x00007fad4d9b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fad4d9b3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007fad4d9b3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007fad4d9b3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007fad62d8ff0c in ?? ()
#12 0x00007fad62d7f7b9 in ?? ()
#13 0x00007fad62d83da6 in ?? ()
#14 0x00007fad62d887e0 in ?? ()
#15 0x00007fad62d88609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007fad5a07e0be in ?? ()
#18 0x00007fad5d4616db in ?? ()
#19 0x00000ff629b2e67c in ?? ()
#20 0x00007fad4d9b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 2693):
#0 0x00007fad5d467fb9 in ?? ()
#1 0x00007fad4f9b8c40 in ?? ()
#2 0x000000000000043b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007fad4f9b8c30 in ?? ()
#6 0x0000000000000876 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 2692):
#0 0x00007fad5d467fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 2691):
#0 0x00007fad5d467fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 2690):
#0 0x00007fad5d467fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000009 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fad511bc410 in ?? ()
#6 0x0000000000000012 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 2687):
#0 0x00007fad5d46bd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250111 02:58:21.663877 1102 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 7 with UUID 5d154c4e3a614e269ad20079b76f38d8 and pid 4117
************************ BEGIN STACKS **************************
[New LWP 4118]
[New LWP 4119]
[New LWP 4120]
[New LWP 4121]
[New LWP 4127]
[New LWP 4129]
[New LWP 4130]
[New LWP 4133]
[New LWP 4134]
[New LWP 4135]
[New LWP 4136]
[New LWP 4137]
[New LWP 4138]
[New LWP 4139]
[New LWP 4140]
[New LWP 4141]
[New LWP 4142]
[New LWP 4143]
[New LWP 4144]
[New LWP 4145]
[New LWP 4146]
[New LWP 4147]
[New LWP 4148]
[New LWP 4149]
[New LWP 4150]
[New LWP 4151]
[New LWP 4152]
[New LWP 4153]
[New LWP 4154]
[New LWP 4155]
[New LWP 4156]
[New LWP 4157]
[New LWP 4158]
[New LWP 4159]
[New LWP 4160]
[New LWP 4161]
[New LWP 4162]
[New LWP 4163]
[New LWP 4164]
[New LWP 4165]
[New LWP 4166]
[New LWP 4167]
[New LWP 4168]
[New LWP 4169]
[New LWP 4170]
[New LWP 4172]
[New LWP 4173]
[New LWP 4174]
[New LWP 4175]
[New LWP 4176]
[New LWP 4177]
[New LWP 4178]
[New LWP 4179]
[New LWP 4180]
[New LWP 4181]
[New LWP 4182]
[New LWP 4183]
[New LWP 4184]
[New LWP 4185]
[New LWP 4186]
[New LWP 4187]
[New LWP 4188]
[New LWP 4189]
[New LWP 4190]
[New LWP 4191]
[New LWP 4192]
[New LWP 4193]
[New LWP 4194]
[New LWP 4195]
[New LWP 4196]
[New LWP 4197]
[New LWP 4198]
[New LWP 4199]
[New LWP 4200]
[New LWP 4201]
[New LWP 4202]
[New LWP 4203]
[New LWP 4204]
[New LWP 4205]
[New LWP 4206]
[New LWP 4207]
[New LWP 4208]
[New LWP 4209]
[New LWP 4210]
[New LWP 4211]
[New LWP 4212]
[New LWP 4213]
[New LWP 4214]
[New LWP 4215]
[New LWP 4216]
[New LWP 4217]
[New LWP 4218]
[New LWP 4219]
[New LWP 4220]
[New LWP 4221]
[New LWP 4222]
[New LWP 4223]
[New LWP 4224]
[New LWP 4225]
[New LWP 4226]
[New LWP 4227]
[New LWP 4228]
[New LWP 4229]
[New LWP 4230]
[New LWP 4231]
[New LWP 4232]
[New LWP 4233]
[New LWP 4234]
[New LWP 4235]
[New LWP 4236]
[New LWP 4237]
[New LWP 4238]
[New LWP 4239]
[New LWP 4241]
[New LWP 4242]
[New LWP 4243]
[New LWP 4244]
[New LWP 4245]
[New LWP 4246]
[New LWP 4247]
[New LWP 4250]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007fa25e798d50 in ?? ()
Id Target Id Frame
* 1 LWP 4117 "kudu" 0x00007fa25e798d50 in ?? ()
2 LWP 4118 "kudu" 0x00007fa25e794fb9 in ?? ()
3 LWP 4119 "kudu" 0x00007fa25e794fb9 in ?? ()
4 LWP 4120 "kudu" 0x00007fa25e794fb9 in ?? ()
5 LWP 4121 "kernel-watcher-" 0x00007fa25e794fb9 in ?? ()
6 LWP 4127 "ntp client-4127" 0x00007fa25e7989e2 in ?? ()
7 LWP 4129 "file cache-evic" 0x00007fa25e794fb9 in ?? ()
8 LWP 4130 "sq_acceptor" 0x00007fa2574f1cb9 in ?? ()
9 LWP 4133 "rpc reactor-413" 0x00007fa2574fea47 in ?? ()
10 LWP 4134 "rpc reactor-413" 0x00007fa2574fea47 in ?? ()
11 LWP 4135 "rpc reactor-413" 0x00007fa2574fea47 in ?? ()
12 LWP 4136 "rpc reactor-413" 0x00007fa2574fea47 in ?? ()
13 LWP 4137 "MaintenanceMgr " 0x00007fa25e794ad3 in ?? ()
14 LWP 4138 "txn-status-mana" 0x00007fa25e794fb9 in ?? ()
15 LWP 4139 "collect_and_rem" 0x00007fa25e794fb9 in ?? ()
16 LWP 4140 "tc-session-exp-" 0x00007fa25e794fb9 in ?? ()
17 LWP 4141 "rpc worker-4141" 0x00007fa25e794ad3 in ?? ()
18 LWP 4142 "rpc worker-4142" 0x00007fa25e794ad3 in ?? ()
19 LWP 4143 "rpc worker-4143" 0x00007fa25e794ad3 in ?? ()
20 LWP 4144 "rpc worker-4144" 0x00007fa25e794ad3 in ?? ()
21 LWP 4145 "rpc worker-4145" 0x00007fa25e794ad3 in ?? ()
22 LWP 4146 "rpc worker-4146" 0x00007fa25e794ad3 in ?? ()
23 LWP 4147 "rpc worker-4147" 0x00007fa25e794ad3 in ?? ()
24 LWP 4148 "rpc worker-4148" 0x00007fa25e794ad3 in ?? ()
25 LWP 4149 "rpc worker-4149" 0x00007fa25e794ad3 in ?? ()
26 LWP 4150 "rpc worker-4150" 0x00007fa25e794ad3 in ?? ()
27 LWP 4151 "rpc worker-4151" 0x00007fa25e794ad3 in ?? ()
28 LWP 4152 "rpc worker-4152" 0x00007fa25e794ad3 in ?? ()
29 LWP 4153 "rpc worker-4153" 0x00007fa25e794ad3 in ?? ()
30 LWP 4154 "rpc worker-4154" 0x00007fa25e794ad3 in ?? ()
31 LWP 4155 "rpc worker-4155" 0x00007fa25e794ad3 in ?? ()
32 LWP 4156 "rpc worker-4156" 0x00007fa25e794ad3 in ?? ()
33 LWP 4157 "rpc worker-4157" 0x00007fa25e794ad3 in ?? ()
34 LWP 4158 "rpc worker-4158" 0x00007fa25e794ad3 in ?? ()
35 LWP 4159 "rpc worker-4159" 0x00007fa25e794ad3 in ?? ()
36 LWP 4160 "rpc worker-4160" 0x00007fa25e794ad3 in ?? ()
37 LWP 4161 "rpc worker-4161" 0x00007fa25e794ad3 in ?? ()
38 LWP 4162 "rpc worker-4162" 0x00007fa25e794ad3 in ?? ()
39 LWP 4163 "rpc worker-4163" 0x00007fa25e794ad3 in ?? ()
40 LWP 4164 "rpc worker-4164" 0x00007fa25e794ad3 in ?? ()
41 LWP 4165 "rpc worker-4165" 0x00007fa25e794ad3 in ?? ()
42 LWP 4166 "rpc worker-4166" 0x00007fa25e794ad3 in ?? ()
43 LWP 4167 "rpc worker-4167" 0x00007fa25e794ad3 in ?? ()
44 LWP 4168 "rpc worker-4168" 0x00007fa25e794ad3 in ?? ()
45 LWP 4169 "rpc worker-4169" 0x00007fa25e794ad3 in ?? ()
46 LWP 4170 "rpc worker-4170" 0x00007fa25e794ad3 in ?? ()
47 LWP 4172 "rpc worker-4172" 0x00007fa25e794ad3 in ?? ()
48 LWP 4173 "rpc worker-4173" 0x00007fa25e794ad3 in ?? ()
49 LWP 4174 "rpc worker-4174" 0x00007fa25e794ad3 in ?? ()
50 LWP 4175 "rpc worker-4175" 0x00007fa25e794ad3 in ?? ()
51 LWP 4176 "rpc worker-4176" 0x00007fa25e794ad3 in ?? ()
52 LWP 4177 "rpc worker-4177" 0x00007fa25e794ad3 in ?? ()
53 LWP 4178 "rpc worker-4178" 0x00007fa25e794ad3 in ?? ()
54 LWP 4179 "rpc worker-4179" 0x00007fa25e794ad3 in ?? ()
55 LWP 4180 "rpc worker-4180" 0x00007fa25e794ad3 in ?? ()
56 LWP 4181 "rpc worker-4181" 0x00007fa25e794ad3 in ?? ()
57 LWP 4182 "rpc worker-4182" 0x00007fa25e794ad3 in ?? ()
58 LWP 4183 "rpc worker-4183" 0x00007fa25e794ad3 in ?? ()
59 LWP 4184 "rpc worker-4184" 0x00007fa25e794ad3 in ?? ()
60 LWP 4185 "rpc worker-4185" 0x00007fa25e794ad3 in ?? ()
61 LWP 4186 "rpc worker-4186" 0x00007fa25e794ad3 in ?? ()
62 LWP 4187 "rpc worker-4187" 0x00007fa25e794ad3 in ?? ()
63 LWP 4188 "rpc worker-4188" 0x00007fa25e794ad3 in ?? ()
64 LWP 4189 "rpc worker-4189" 0x00007fa25e794ad3 in ?? ()
65 LWP 4190 "rpc worker-4190" 0x00007fa25e794ad3 in ?? ()
66 LWP 4191 "rpc worker-4191" 0x00007fa25e794ad3 in ?? ()
67 LWP 4192 "rpc worker-4192" 0x00007fa25e794ad3 in ?? ()
68 LWP 4193 "rpc worker-4193" 0x00007fa25e794ad3 in ?? ()
69 LWP 4194 "rpc worker-4194" 0x00007fa25e794ad3 in ?? ()
70 LWP 4195 "rpc worker-4195" 0x00007fa25e794ad3 in ?? ()
71 LWP 4196 "rpc worker-4196" 0x00007fa25e794ad3 in ?? ()
72 LWP 4197 "rpc worker-4197" 0x00007fa25e794ad3 in ?? ()
73 LWP 4198 "rpc worker-4198" 0x00007fa25e794ad3 in ?? ()
74 LWP 4199 "rpc worker-4199" 0x00007fa25e794ad3 in ?? ()
75 LWP 4200 "rpc worker-4200" 0x00007fa25e794ad3 in ?? ()
76 LWP 4201 "rpc worker-4201" 0x00007fa25e794ad3 in ?? ()
77 LWP 4202 "rpc worker-4202" 0x00007fa25e794ad3 in ?? ()
78 LWP 4203 "rpc worker-4203" 0x00007fa25e794ad3 in ?? ()
79 LWP 4204 "rpc worker-4204" 0x00007fa25e794ad3 in ?? ()
80 LWP 4205 "rpc worker-4205" 0x00007fa25e794ad3 in ?? ()
81 LWP 4206 "rpc worker-4206" 0x00007fa25e794ad3 in ?? ()
82 LWP 4207 "rpc worker-4207" 0x00007fa25e794ad3 in ?? ()
83 LWP 4208 "rpc worker-4208" 0x00007fa25e794ad3 in ?? ()
84 LWP 4209 "rpc worker-4209" 0x00007fa25e794ad3 in ?? ()
85 LWP 4210 "rpc worker-4210" 0x00007fa25e794ad3 in ?? ()
86 LWP 4211 "rpc worker-4211" 0x00007fa25e794ad3 in ?? ()
87 LWP 4212 "rpc worker-4212" 0x00007fa25e794ad3 in ?? ()
88 LWP 4213 "rpc worker-4213" 0x00007fa25e794ad3 in ?? ()
89 LWP 4214 "rpc worker-4214" 0x00007fa25e794ad3 in ?? ()
90 LWP 4215 "rpc worker-4215" 0x00007fa25e794ad3 in ?? ()
91 LWP 4216 "rpc worker-4216" 0x00007fa25e794ad3 in ?? ()
92 LWP 4217 "rpc worker-4217" 0x00007fa25e794ad3 in ?? ()
93 LWP 4218 "rpc worker-4218" 0x00007fa25e794ad3 in ?? ()
94 LWP 4219 "rpc worker-4219" 0x00007fa25e794ad3 in ?? ()
95 LWP 4220 "rpc worker-4220" 0x00007fa25e794ad3 in ?? ()
96 LWP 4221 "rpc worker-4221" 0x00007fa25e794ad3 in ?? ()
97 LWP 4222 "rpc worker-4222" 0x00007fa25e794ad3 in ?? ()
98 LWP 4223 "rpc worker-4223" 0x00007fa25e794ad3 in ?? ()
99 LWP 4224 "rpc worker-4224" 0x00007fa25e794ad3 in ?? ()
100 LWP 4225 "rpc worker-4225" 0x00007fa25e794ad3 in ?? ()
101 LWP 4226 "rpc worker-4226" 0x00007fa25e794ad3 in ?? ()
102 LWP 4227 "rpc worker-4227" 0x00007fa25e794ad3 in ?? ()
103 LWP 4228 "rpc worker-4228" 0x00007fa25e794ad3 in ?? ()
104 LWP 4229 "rpc worker-4229" 0x00007fa25e794ad3 in ?? ()
105 LWP 4230 "rpc worker-4230" 0x00007fa25e794ad3 in ?? ()
106 LWP 4231 "rpc worker-4231" 0x00007fa25e794ad3 in ?? ()
107 LWP 4232 "rpc worker-4232" 0x00007fa25e794ad3 in ?? ()
108 LWP 4233 "rpc worker-4233" 0x00007fa25e794ad3 in ?? ()
109 LWP 4234 "rpc worker-4234" 0x00007fa25e794ad3 in ?? ()
110 LWP 4235 "rpc worker-4235" 0x00007fa25e794ad3 in ?? ()
111 LWP 4236 "rpc worker-4236" 0x00007fa25e794ad3 in ?? ()
112 LWP 4237 "rpc worker-4237" 0x00007fa25e794ad3 in ?? ()
113 LWP 4238 "rpc worker-4238" 0x00007fa25e794ad3 in ?? ()
114 LWP 4239 "rpc worker-4239" 0x00007fa25e794ad3 in ?? ()
115 LWP 4241 "rpc worker-4241" 0x00007fa25e794ad3 in ?? ()
116 LWP 4242 "rpc worker-4242" 0x00007fa25e794ad3 in ?? ()
117 LWP 4243 "diag-logger-424" 0x00007fa25e794fb9 in ?? ()
118 LWP 4244 "result-tracker-" 0x00007fa25e794fb9 in ?? ()
119 LWP 4245 "excess-log-dele" 0x00007fa25e794fb9 in ?? ()
120 LWP 4246 "acceptor-4246" 0x00007fa2575000c7 in ?? ()
121 LWP 4247 "heartbeat-4247" 0x00007fa25e794fb9 in ?? ()
122 LWP 4250 "maintenance_sch" 0x00007fa25e794fb9 in ?? ()
Thread 122 (LWP 4250):
#0 0x00007fa25e794fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000233 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007fa214078550 in ?? ()
#6 0x0000000000000466 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 4247):
#0 0x00007fa25e794fb9 in ?? ()
#1 0xabd77187b7907576 in ?? ()
#2 0x000000000000008c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007fa2148905b0 in ?? ()
#6 0x0000000000000118 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 4246):
#0 0x00007fa2575000c7 in ?? ()
#1 0x0000614000021aa8 in ?? ()
#2 0x00007fa2150b7d50 in ?? ()
#3 0x00007fa2150b7d80 in ?? ()
#4 0x00007fa2150b7e80 in ?? ()
#5 0x00007fa2150b7d70 in ?? ()
#6 0x00007fa2150b7de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 4245):
#0 0x00007fa25e794fb9 in ?? ()
#1 0x00007fa2158d1f00 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fffe6d7e850 in ?? ()
#5 0x00007fa2158d1ef0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 4244):
#0 0x00007fa25e794fb9 in ?? ()
#1 0x00007fa2160ea0c0 in ?? ()
#2 0x000000000000008d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007fa2160ea0b0 in ?? ()
#6 0x000000000000011a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 4243):
#0 0x00007fa25e794fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 4242):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x000000000000001c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007fa217119e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa217119e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 4241):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x000000000000001c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002a6858 in ?? ()
#4 0x00007fa217931e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa217931e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 4239):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 4238):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 4237):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 4236):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 4235):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 4234):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 4233):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 4232):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 4231):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 4230):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 4229):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 4228):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 4227):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 4226):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 4225):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 4224):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 4223):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 4222):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 4221):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 4220):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 4219):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 4218):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 4217):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 4216):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 4215):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 4214):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 4213):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 4212):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 4211):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 4210):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 4209):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 4208):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 4207):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 4206):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 4205):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 4204):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 4203):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 4202):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 4201):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x000000000000000e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007fa22acd2e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa22acd2e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 4200):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 4199):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 4198):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 4197):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 4196):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 4195):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 4194):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 4193):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 4192):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 4191):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 4190):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 4189):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 4188):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 4187):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 4186):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 4185):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 4184):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 4183):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 4182):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 4181):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x000000000000000e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007fa234eb3e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa234eb3e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 4180):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 4179):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 4178):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 4177):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 4176):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 4175):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 4174):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 4173):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 4172):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 4170):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 4169):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 4168):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 4167):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 4166):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 4165):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 4164):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 4163):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 4162):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 4161):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 4160):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x000000000000000e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007fa23f091e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa23f091e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 4159):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 4158):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 4157):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 4156):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 4155):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 4154):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 4153):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 4152):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 4151):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 4150):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 4149):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 4148):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 4147):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 4146):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 4145):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 4144):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 4143):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 4142):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 4141):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 4140):
#0 0x00007fa25e794fb9 in ?? ()
#1 0x00007fa2490c3c80 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001b360 in ?? ()
#5 0x00007fa2490c3c70 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 4139):
#0 0x00007fa25e794fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 4138):
#0 0x00007fa25e794fb9 in ?? ()
#1 0x00007fa24a0c51c0 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007fa24a0c51b0 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 4137):
#0 0x00007fa25e794ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 4136):
#0 0x00007fa2574fea47 in ?? ()
#1 0x00007fa24b0c82e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007fa24b0c82d0 in ?? ()
#4 0x00007fa24b0c84e0 in ?? ()
#5 0x00007fa24b0c8320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007fa24b0c83a0 in ?? ()
#8 0x00007fa259b5e25d in ?? ()
#9 0x3fb95f15ff023000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x000000005e000000 in ?? ()
#14 0x00007fa200000000 in ?? ()
#15 0x41d9e077162c1109 in ?? ()
#16 0x00000ff4c9611074 in ?? ()
#17 0x00007fa24b0c8380 in ?? ()
#18 0x00007fa259b62ba3 in ?? ()
#19 0x00007fa24b0c8350 in ?? ()
#20 0x3fb95f15ff023000 in ?? ()
#21 0x000000004b0c83a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb95f15ff023000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 4135):
#0 0x00007fa2574fea47 in ?? ()
#1 0x00007fa24b8c92e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007fa24b8c92d0 in ?? ()
#4 0x00007fa24b8c94e0 in ?? ()
#5 0x00007fa24b8c9320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007fa24b8c93a0 in ?? ()
#8 0x00007fa259b5e25d in ?? ()
#9 0x3fb97a59dad7b000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x0000000018000000 in ?? ()
#14 0x00007fa200000000 in ?? ()
#15 0x41d9e077162c110c in ?? ()
#16 0x00000ff4c9711274 in ?? ()
#17 0x00007fa24b8c9380 in ?? ()
#18 0x00007fa259b62ba3 in ?? ()
#19 0x00007fa24b8c9350 in ?? ()
#20 0x3fb97a59dad7b000 in ?? ()
#21 0x000000004b8c93a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb97a59dad7b000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 4134):
#0 0x00007fa2574fea47 in ?? ()
#1 0x00007fa24c0ca2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007fa24c0ca2d0 in ?? ()
#4 0x00007fa24c0ca4e0 in ?? ()
#5 0x00007fa24c0ca320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007fa24c0ca3a0 in ?? ()
#8 0x00007fa259b5e25d in ?? ()
#9 0x3fb97773eecc9000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000014800000 in ?? ()
#14 0x00007fa200000000 in ?? ()
#15 0x41d9e077162c110a in ?? ()
#16 0x00000ff4c9811474 in ?? ()
#17 0x00007fa24c0ca380 in ?? ()
#18 0x00007fa259b62ba3 in ?? ()
#19 0x00007fa24c0ca350 in ?? ()
#20 0x3fb97773eecc9000 in ?? ()
#21 0x000000004c0ca3a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb97773eecc9000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 4133):
#0 0x00007fa2574fea47 in ?? ()
#1 0x00007fa24dcb32e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007fa24dcb32d0 in ?? ()
#4 0x00007fa24dcb34e0 in ?? ()
#5 0x00007fa24dcb3320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fa24dcb33a0 in ?? ()
#8 0x00007fa259b5e25d in ?? ()
#9 0x3fb95bac23579000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x00000000c5800000 in ?? ()
#14 0x00007fa200000000 in ?? ()
#15 0x41d9e077162c110a in ?? ()
#16 0x00000ff4c9b8e674 in ?? ()
#17 0x00007fa24dcb3380 in ?? ()
#18 0x00007fa259b62ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 4130):
#0 0x00007fa2574f1cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fa24fcb7788 in ?? ()
#3 0x0000602000017cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 4129):
#0 0x00007fa25e794fb9 in ?? ()
#1 0x00007fa24f4b6080 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007fa24f4b6070 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 4127):
#0 0x00007fa25e7989e2 in ?? ()
#1 0x00007fa24e4b2b60 in ?? ()
#2 0x00007fa24e4b2c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fa24e4b2bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007fa24e4b2c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007fa24e4b2c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007fa2640bcf0c in ?? ()
#12 0x00007fa2640ac7b9 in ?? ()
#13 0x00007fa2640b0da6 in ?? ()
#14 0x00007fa2640b57e0 in ?? ()
#15 0x00007fa2640b5609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007fa25b3ab0be in ?? ()
#18 0x00007fa25e78e6db in ?? ()
#19 0x00000ff4c9c8e47c in ?? ()
#20 0x00007fa24e4b2400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 4121):
#0 0x00007fa25e794fb9 in ?? ()
#1 0x00007fa250cb8c40 in ?? ()
#2 0x00000000000002c4 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007fa250cb8c30 in ?? ()
#6 0x0000000000000588 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 4120):
#0 0x00007fa25e794fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 4119):
#0 0x00007fa25e794fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 4118):
#0 0x00007fa25e794fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fa2524bc410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 4117):
#0 0x00007fa25e798d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250111 02:58:22.415737 1102 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 8 with UUID 560d9bddebd54e0da0bf8d2da0a7a593 and pid 4258
************************ BEGIN STACKS **************************
[New LWP 4262]
[New LWP 4263]
[New LWP 4264]
[New LWP 4265]
[New LWP 4271]
[New LWP 4272]
[New LWP 4273]
[New LWP 4276]
[New LWP 4277]
[New LWP 4278]
[New LWP 4279]
[New LWP 4280]
[New LWP 4282]
[New LWP 4285]
[New LWP 4286]
[New LWP 4287]
[New LWP 4288]
[New LWP 4289]
[New LWP 4290]
[New LWP 4291]
[New LWP 4292]
[New LWP 4293]
[New LWP 4294]
[New LWP 4295]
[New LWP 4296]
[New LWP 4297]
[New LWP 4298]
[New LWP 4299]
[New LWP 4300]
[New LWP 4301]
[New LWP 4302]
[New LWP 4303]
[New LWP 4304]
[New LWP 4305]
[New LWP 4306]
[New LWP 4307]
[New LWP 4308]
[New LWP 4309]
[New LWP 4310]
[New LWP 4312]
[New LWP 4313]
[New LWP 4314]
[New LWP 4316]
[New LWP 4317]
[New LWP 4318]
[New LWP 4319]
[New LWP 4320]
[New LWP 4322]
[New LWP 4323]
[New LWP 4324]
[New LWP 4325]
[New LWP 4326]
[New LWP 4327]
[New LWP 4328]
[New LWP 4329]
[New LWP 4330]
[New LWP 4331]
[New LWP 4332]
[New LWP 4333]
[New LWP 4334]
[New LWP 4335]
[New LWP 4336]
[New LWP 4337]
[New LWP 4338]
[New LWP 4339]
[New LWP 4340]
[New LWP 4341]
[New LWP 4342]
[New LWP 4343]
[New LWP 4344]
[New LWP 4345]
[New LWP 4346]
[New LWP 4347]
[New LWP 4348]
[New LWP 4349]
[New LWP 4350]
[New LWP 4351]
[New LWP 4352]
[New LWP 4353]
[New LWP 4354]
[New LWP 4355]
[New LWP 4356]
[New LWP 4357]
[New LWP 4358]
[New LWP 4359]
[New LWP 4360]
[New LWP 4361]
[New LWP 4362]
[New LWP 4363]
[New LWP 4364]
[New LWP 4365]
[New LWP 4366]
[New LWP 4367]
[New LWP 4368]
[New LWP 4369]
[New LWP 4370]
[New LWP 4371]
[New LWP 4372]
[New LWP 4373]
[New LWP 4374]
[New LWP 4375]
[New LWP 4376]
[New LWP 4377]
[New LWP 4378]
[New LWP 4379]
[New LWP 4380]
[New LWP 4381]
[New LWP 4382]
[New LWP 4383]
[New LWP 4384]
[New LWP 4385]
[New LWP 4386]
[New LWP 4387]
[New LWP 4388]
[New LWP 4389]
[New LWP 4390]
[New LWP 4391]
[New LWP 4392]
[New LWP 4393]
[New LWP 4394]
[New LWP 4398]
[New LWP 5688]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007f4dc956fd50 in ?? ()
Id Target Id Frame
* 1 LWP 4258 "kudu" 0x00007f4dc956fd50 in ?? ()
2 LWP 4262 "kudu" 0x00007f4dc956bfb9 in ?? ()
3 LWP 4263 "kudu" 0x00007f4dc956bfb9 in ?? ()
4 LWP 4264 "kudu" 0x00007f4dc956bfb9 in ?? ()
5 LWP 4265 "kernel-watcher-" 0x00007f4dc956bfb9 in ?? ()
6 LWP 4271 "ntp client-4271" 0x00007f4dc956f9e2 in ?? ()
7 LWP 4272 "file cache-evic" 0x00007f4dc956bfb9 in ?? ()
8 LWP 4273 "sq_acceptor" 0x00007f4dc22c8cb9 in ?? ()
9 LWP 4276 "rpc reactor-427" 0x00007f4dc22d5a47 in ?? ()
10 LWP 4277 "rpc reactor-427" 0x00007f4dc22d5a47 in ?? ()
11 LWP 4278 "rpc reactor-427" 0x00007f4dc22d5a47 in ?? ()
12 LWP 4279 "rpc reactor-427" 0x00007f4dc22d5a47 in ?? ()
13 LWP 4280 "MaintenanceMgr " 0x00007f4dc956bad3 in ?? ()
14 LWP 4282 "txn-status-mana" 0x00007f4dc956bfb9 in ?? ()
15 LWP 4285 "collect_and_rem" 0x00007f4dc956bfb9 in ?? ()
16 LWP 4286 "tc-session-exp-" 0x00007f4dc956bfb9 in ?? ()
17 LWP 4287 "rpc worker-4287" 0x00007f4dc956bad3 in ?? ()
18 LWP 4288 "rpc worker-4288" 0x00007f4dc956bad3 in ?? ()
19 LWP 4289 "rpc worker-4289" 0x00007f4dc956bad3 in ?? ()
20 LWP 4290 "rpc worker-4290" 0x00007f4dc956bad3 in ?? ()
21 LWP 4291 "rpc worker-4291" 0x00007f4dc956bad3 in ?? ()
22 LWP 4292 "rpc worker-4292" 0x00007f4dc956bad3 in ?? ()
23 LWP 4293 "rpc worker-4293" 0x00007f4dc956bad3 in ?? ()
24 LWP 4294 "rpc worker-4294" 0x00007f4dc956bad3 in ?? ()
25 LWP 4295 "rpc worker-4295" 0x00007f4dc956bad3 in ?? ()
26 LWP 4296 "rpc worker-4296" 0x00007f4dc956bad3 in ?? ()
27 LWP 4297 "rpc worker-4297" 0x00007f4dc956bad3 in ?? ()
28 LWP 4298 "rpc worker-4298" 0x00007f4dc956bad3 in ?? ()
29 LWP 4299 "rpc worker-4299" 0x00007f4dc956bad3 in ?? ()
30 LWP 4300 "rpc worker-4300" 0x00007f4dc956bad3 in ?? ()
31 LWP 4301 "rpc worker-4301" 0x00007f4dc956bad3 in ?? ()
32 LWP 4302 "rpc worker-4302" 0x00007f4dc956bad3 in ?? ()
33 LWP 4303 "rpc worker-4303" 0x00007f4dc956bad3 in ?? ()
34 LWP 4304 "rpc worker-4304" 0x00007f4dc956bad3 in ?? ()
35 LWP 4305 "rpc worker-4305" 0x00007f4dc956bad3 in ?? ()
36 LWP 4306 "rpc worker-4306" 0x00007f4dc956bad3 in ?? ()
37 LWP 4307 "rpc worker-4307" 0x00007f4dc956bad3 in ?? ()
38 LWP 4308 "rpc worker-4308" 0x00007f4dc956bad3 in ?? ()
39 LWP 4309 "rpc worker-4309" 0x00007f4dc956bad3 in ?? ()
40 LWP 4310 "rpc worker-4310" 0x00007f4dc956bad3 in ?? ()
41 LWP 4312 "rpc worker-4312" 0x00007f4dc956bad3 in ?? ()
42 LWP 4313 "rpc worker-4313" 0x00007f4dc956bad3 in ?? ()
43 LWP 4314 "rpc worker-4314" 0x00007f4dc956bad3 in ?? ()
44 LWP 4316 "rpc worker-4316" 0x00007f4dc956bad3 in ?? ()
45 LWP 4317 "rpc worker-4317" 0x00007f4dc956bad3 in ?? ()
46 LWP 4318 "rpc worker-4318" 0x00007f4dc956bad3 in ?? ()
47 LWP 4319 "rpc worker-4319" 0x00007f4dc956bad3 in ?? ()
48 LWP 4320 "rpc worker-4320" 0x00007f4dc956bad3 in ?? ()
49 LWP 4322 "rpc worker-4322" 0x00007f4dc956bad3 in ?? ()
50 LWP 4323 "rpc worker-4323" 0x00007f4dc956bad3 in ?? ()
51 LWP 4324 "rpc worker-4324" 0x00007f4dc956bad3 in ?? ()
52 LWP 4325 "rpc worker-4325" 0x00007f4dc956bad3 in ?? ()
53 LWP 4326 "rpc worker-4326" 0x00007f4dc956bad3 in ?? ()
54 LWP 4327 "rpc worker-4327" 0x00007f4dc956bad3 in ?? ()
55 LWP 4328 "rpc worker-4328" 0x00007f4dc956bad3 in ?? ()
56 LWP 4329 "rpc worker-4329" 0x00007f4dc956bad3 in ?? ()
57 LWP 4330 "rpc worker-4330" 0x00007f4dc956bad3 in ?? ()
58 LWP 4331 "rpc worker-4331" 0x00007f4dc956bad3 in ?? ()
59 LWP 4332 "rpc worker-4332" 0x00007f4dc956bad3 in ?? ()
60 LWP 4333 "rpc worker-4333" 0x00007f4dc956bad3 in ?? ()
61 LWP 4334 "rpc worker-4334" 0x00007f4dc956bad3 in ?? ()
62 LWP 4335 "rpc worker-4335" 0x00007f4dc956bad3 in ?? ()
63 LWP 4336 "rpc worker-4336" 0x00007f4dc956bad3 in ?? ()
64 LWP 4337 "rpc worker-4337" 0x00007f4dc956bad3 in ?? ()
65 LWP 4338 "rpc worker-4338" 0x00007f4dc956bad3 in ?? ()
66 LWP 4339 "rpc worker-4339" 0x00007f4dc956bad3 in ?? ()
67 LWP 4340 "rpc worker-4340" 0x00007f4dc956bad3 in ?? ()
68 LWP 4341 "rpc worker-4341" 0x00007f4dc956bad3 in ?? ()
69 LWP 4342 "rpc worker-4342" 0x00007f4dc956bad3 in ?? ()
70 LWP 4343 "rpc worker-4343" 0x00007f4dc956bad3 in ?? ()
71 LWP 4344 "rpc worker-4344" 0x00007f4dc956bad3 in ?? ()
72 LWP 4345 "rpc worker-4345" 0x00007f4dc956bad3 in ?? ()
73 LWP 4346 "rpc worker-4346" 0x00007f4dc956bad3 in ?? ()
74 LWP 4347 "rpc worker-4347" 0x00007f4dc956bad3 in ?? ()
75 LWP 4348 "rpc worker-4348" 0x00007f4dc956bad3 in ?? ()
76 LWP 4349 "rpc worker-4349" 0x00007f4dc956bad3 in ?? ()
77 LWP 4350 "rpc worker-4350" 0x00007f4dc956bad3 in ?? ()
78 LWP 4351 "rpc worker-4351" 0x00007f4dc956bad3 in ?? ()
79 LWP 4352 "rpc worker-4352" 0x00007f4dc956bad3 in ?? ()
80 LWP 4353 "rpc worker-4353" 0x00007f4dc956bad3 in ?? ()
81 LWP 4354 "rpc worker-4354" 0x00007f4dc956bad3 in ?? ()
82 LWP 4355 "rpc worker-4355" 0x00007f4dc956bad3 in ?? ()
83 LWP 4356 "rpc worker-4356" 0x00007f4dc956bad3 in ?? ()
84 LWP 4357 "rpc worker-4357" 0x00007f4dc956bad3 in ?? ()
85 LWP 4358 "rpc worker-4358" 0x00007f4dc956bad3 in ?? ()
86 LWP 4359 "rpc worker-4359" 0x00007f4dc956bad3 in ?? ()
87 LWP 4360 "rpc worker-4360" 0x00007f4dc956bad3 in ?? ()
88 LWP 4361 "rpc worker-4361" 0x00007f4dc956bad3 in ?? ()
89 LWP 4362 "rpc worker-4362" 0x00007f4dc956bad3 in ?? ()
90 LWP 4363 "rpc worker-4363" 0x00007f4dc956bad3 in ?? ()
91 LWP 4364 "rpc worker-4364" 0x00007f4dc956bad3 in ?? ()
92 LWP 4365 "rpc worker-4365" 0x00007f4dc956bad3 in ?? ()
93 LWP 4366 "rpc worker-4366" 0x00007f4dc956bad3 in ?? ()
94 LWP 4367 "rpc worker-4367" 0x00007f4dc956bad3 in ?? ()
95 LWP 4368 "rpc worker-4368" 0x00007f4dc956bad3 in ?? ()
96 LWP 4369 "rpc worker-4369" 0x00007f4dc956bad3 in ?? ()
97 LWP 4370 "rpc worker-4370" 0x00007f4dc956bad3 in ?? ()
98 LWP 4371 "rpc worker-4371" 0x00007f4dc956bad3 in ?? ()
99 LWP 4372 "rpc worker-4372" 0x00007f4dc956bad3 in ?? ()
100 LWP 4373 "rpc worker-4373" 0x00007f4dc956bad3 in ?? ()
101 LWP 4374 "rpc worker-4374" 0x00007f4dc956bad3 in ?? ()
102 LWP 4375 "rpc worker-4375" 0x00007f4dc956bad3 in ?? ()
103 LWP 4376 "rpc worker-4376" 0x00007f4dc956bad3 in ?? ()
104 LWP 4377 "rpc worker-4377" 0x00007f4dc956bad3 in ?? ()
105 LWP 4378 "rpc worker-4378" 0x00007f4dc956bad3 in ?? ()
106 LWP 4379 "rpc worker-4379" 0x00007f4dc956bad3 in ?? ()
107 LWP 4380 "rpc worker-4380" 0x00007f4dc956bad3 in ?? ()
108 LWP 4381 "rpc worker-4381" 0x00007f4dc956bad3 in ?? ()
109 LWP 4382 "rpc worker-4382" 0x00007f4dc956bad3 in ?? ()
110 LWP 4383 "rpc worker-4383" 0x00007f4dc956bad3 in ?? ()
111 LWP 4384 "rpc worker-4384" 0x00007f4dc956bad3 in ?? ()
112 LWP 4385 "rpc worker-4385" 0x00007f4dc956bad3 in ?? ()
113 LWP 4386 "rpc worker-4386" 0x00007f4dc956bad3 in ?? ()
114 LWP 4387 "rpc worker-4387" 0x00007f4dc956bad3 in ?? ()
115 LWP 4388 "rpc worker-4388" 0x00007f4dc956bad3 in ?? ()
116 LWP 4389 "rpc worker-4389" 0x00007f4dc956bad3 in ?? ()
117 LWP 4390 "diag-logger-439" 0x00007f4dc956bfb9 in ?? ()
118 LWP 4391 "result-tracker-" 0x00007f4dc956bfb9 in ?? ()
119 LWP 4392 "excess-log-dele" 0x00007f4dc956bfb9 in ?? ()
120 LWP 4393 "acceptor-4393" 0x00007f4dc22d70c7 in ?? ()
121 LWP 4394 "heartbeat-4394" 0x00007f4dc956bfb9 in ?? ()
122 LWP 4398 "maintenance_sch" 0x00007f4dc956bfb9 in ?? ()
123 LWP 5688 "raft [worker]-5" 0x00007f4dc956bfb9 in ?? ()
Thread 123 (LWP 5688):
#0 0x00007f4dc956bfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000002d in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00007f4d75c49b24 in ?? ()
#5 0x00007f4d75c497b0 in ?? ()
#6 0x000000000000005b in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f4d75c497d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 122 (LWP 4398):
#0 0x00007f4dc956bfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000022a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007f4d7d5d9550 in ?? ()
#6 0x0000000000000454 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 4394):
#0 0x00007f4dc956bfb9 in ?? ()
#1 0xabd77187b7907576 in ?? ()
#2 0x000000000000008e in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x000061300001aac4 in ?? ()
#5 0x00007f4d7ee0f5b0 in ?? ()
#6 0x000000000000011d in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f4d7ee0f5d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007f4d7ee0f650 in ?? ()
#12 0x00000fe9afdc1ecc in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 120 (LWP 4393):
#0 0x00007f4dc22d70c7 in ?? ()
#1 0x00006140000218a8 in ?? ()
#2 0x00007f4d7f636d50 in ?? ()
#3 0x00007f4d7f636d80 in ?? ()
#4 0x00007f4d7f636e80 in ?? ()
#5 0x00007f4d7f636d70 in ?? ()
#6 0x00007f4d7f636de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007f4dc6161620 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 4392):
#0 0x00007f4dc956bfb9 in ?? ()
#1 0x00007f4d7fe4ff00 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffe3b195010 in ?? ()
#5 0x00007f4d7fe4fef0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 4391):
#0 0x00007f4dc956bfb9 in ?? ()
#1 0x00007f4d806690c0 in ?? ()
#2 0x000000000000008b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007f4d806690b0 in ?? ()
#6 0x0000000000000116 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 4390):
#0 0x00007f4dc956bfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 4389):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x000000000000001c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002e0018 in ?? ()
#4 0x00007f4d81698e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4d81698e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 4388):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x000000000000001c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002dd028 in ?? ()
#4 0x00007f4d81eb0e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4d81eb0e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 4387):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 4386):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 4385):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 4384):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 4383):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 4382):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 4381):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 4380):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 4379):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 4378):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 4377):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 4376):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 4375):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 4374):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 4373):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 4372):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 4371):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 4370):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 4369):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 4368):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 4367):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 4366):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 4365):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 4364):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 4363):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 4362):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 4361):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 4360):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 4359):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 4358):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 4357):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 4356):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 4355):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 4354):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 4353):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 4352):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 4351):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 4350):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 4349):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x00000000000002a3 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001d005c in ?? ()
#4 0x00007f4d95a69e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4d95a69e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001d0010 in ?? ()
#9 0x00007f4dc956b770 in ?? ()
#10 0x00007f4d95a69e70 in ?? ()
#11 0x00000fe9b2b4d3d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 4348):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x00000000000000f9 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001cd06c in ?? ()
#4 0x00007f4d96281e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4d96281e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001cd020 in ?? ()
#9 0x00007f4dc956b770 in ?? ()
#10 0x00007f4d96281e70 in ?? ()
#11 0x00000fe9b2c503d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 74 (LWP 4347):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 4346):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 4345):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 4344):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 4343):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 4342):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 4341):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 4340):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 4339):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 4338):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 4337):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 4336):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 4335):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 4334):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 4333):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 4332):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 4331):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 4330):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 4329):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x000000000000000e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000140018 in ?? ()
#4 0x00007f4d9fc4ae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4d9fc4ae70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 4328):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 4327):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 4326):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 4325):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 4324):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 4323):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 4322):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 4320):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 4319):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 4318):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 4317):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 4316):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 4314):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 4313):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 4312):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 4310):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 4309):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 4308):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 4307):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 4306):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x00000000000002d4 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000affd8 in ?? ()
#4 0x00007f4da9613e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4da9613e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 4305):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x00000000000006ae in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000acfe8 in ?? ()
#4 0x00007f4da9e2be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4da9e2be70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 34 (LWP 4304):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000a96 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007f4daa643e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4daa643e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 33 (LWP 4303):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000ed8 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00009ffe8 in ?? ()
#4 0x00007f4daae5be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4daae5be70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 32 (LWP 4302):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000b45 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000967fc in ?? ()
#4 0x00007f4dab673e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4dab673e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000967b0 in ?? ()
#9 0x00007f4dc956b770 in ?? ()
#10 0x00007f4dab673e70 in ?? ()
#11 0x00000fe9b56ce7d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 31 (LWP 4301):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x000000000000019b in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00008fffc in ?? ()
#4 0x00007f4dabe8be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4dabe8be70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00008ffb0 in ?? ()
#9 0x00007f4dc956b770 in ?? ()
#10 0x00007f4dabe8be70 in ?? ()
#11 0x00000fe9b57d17d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 30 (LWP 4300):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 4299):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 4298):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 4297):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 4296):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 4295):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 4294):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 4293):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 4292):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 4291):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 4290):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 4289):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 4288):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 4287):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 4286):
#0 0x00007f4dc956bfb9 in ?? ()
#1 0x00007f4db3818c80 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ba60 in ?? ()
#5 0x00007f4db3818c70 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 4285):
#0 0x00007f4dc956bfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 4282):
#0 0x00007f4dc956bfb9 in ?? ()
#1 0x00007f4db50af1c0 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000fda8 in ?? ()
#5 0x00007f4db50af1b0 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 4280):
#0 0x00007f4dc956bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 4279):
#0 0x00007f4dc22d5a47 in ?? ()
#1 0x00007f4db60b22e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007f4db60b22d0 in ?? ()
#4 0x00007f4db60b24e0 in ?? ()
#5 0x00007f4db60b2320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007f4db60b23a0 in ?? ()
#8 0x00007f4dc493525d in ?? ()
#9 0x3fb3326727ced000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000056000000 in ?? ()
#14 0x00007f4d00000000 in ?? ()
#15 0x41d9e077162c110a in ?? ()
#16 0x00000fea36c0e474 in ?? ()
#17 0x00007f4db60b2380 in ?? ()
#18 0x00007f4dc4939ba3 in ?? ()
#19 0x00007f4db60b2350 in ?? ()
#20 0x3fb3326727ced000 in ?? ()
#21 0x00000000b60b23a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb3326727ced000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 4278):
#0 0x00007f4dc22d5a47 in ?? ()
#1 0x00007f4db68b32e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007f4db68b32d0 in ?? ()
#4 0x00007f4db68b34e0 in ?? ()
#5 0x00007f4db68b3320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007f4db68b33a0 in ?? ()
#8 0x00007f4dc493525d in ?? ()
#9 0x3fad8e9175708000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x0000000036b00000 in ?? ()
#14 0x00007f4d00000000 in ?? ()
#15 0x41d9e077162c110a in ?? ()
#16 0x00000fea36d0e674 in ?? ()
#17 0x00007f4db68b3380 in ?? ()
#18 0x00007f4dc4939ba3 in ?? ()
#19 0x00007f4db68b3350 in ?? ()
#20 0x3fad8e9175708000 in ?? ()
#21 0x00000000b68b33a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fad8e9175708000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 4277):
#0 0x00007f4dc22d5a47 in ?? ()
#1 0x00007f4db70b42e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007f4db70b42d0 in ?? ()
#4 0x00007f4db70b44e0 in ?? ()
#5 0x00007f4db70b4320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007f4db70b43a0 in ?? ()
#8 0x00007f4dc493525d in ?? ()
#9 0x3fb959441cbb4000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000072000000 in ?? ()
#14 0x00007f4d00000000 in ?? ()
#15 0x41d9e077162c110c in ?? ()
#16 0x00000fea36e0e874 in ?? ()
#17 0x00007f4db70b4380 in ?? ()
#18 0x00007f4dc4939ba3 in ?? ()
#19 0x00007f4db70b4350 in ?? ()
#20 0x3fb959441cbb4000 in ?? ()
#21 0x00000000b70b43a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb959441cbb4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 4276):
#0 0x00007f4dc22d5a47 in ?? ()
#1 0x00007f4db90ad2e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007f4db90ad2d0 in ?? ()
#4 0x00007f4db90ad4e0 in ?? ()
#5 0x00007f4db90ad320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f4db90ad3a0 in ?? ()
#8 0x00007f4dc493525d in ?? ()
#9 0x3fb979498b3a2000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x0000000097000000 in ?? ()
#14 0x00007f4d00000000 in ?? ()
#15 0x41d9e077162c110d in ?? ()
#16 0x00000fea3720da74 in ?? ()
#17 0x00007f4db90ad380 in ?? ()
#18 0x00007f4dc4939ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 4273):
#0 0x00007f4dc22c8cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f4dbaab7788 in ?? ()
#3 0x0000602000018a10 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 4272):
#0 0x00007f4dc956bfb9 in ?? ()
#1 0x00007f4dba2b6080 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007f4dba2b6070 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 4271):
#0 0x00007f4dc956f9e2 in ?? ()
#1 0x00007f4db9ab3b60 in ?? ()
#2 0x00007f4db9ab3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f4db9ab3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007f4db9ab3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007f4db9ab3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007f4dcee93f0c in ?? ()
#12 0x00007f4dcee837b9 in ?? ()
#13 0x00007f4dcee87da6 in ?? ()
#14 0x00007f4dcee8c7e0 in ?? ()
#15 0x00007f4dcee8c609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007f4dc61820be in ?? ()
#18 0x00007f4dc95656db in ?? ()
#19 0x00000fea3734e67c in ?? ()
#20 0x00007f4db9ab3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 4265):
#0 0x00007f4dc956bfb9 in ?? ()
#1 0x00007f4dbbab8c40 in ?? ()
#2 0x00000000000002bc in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007f4dbbab8c30 in ?? ()
#6 0x0000000000000578 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 4264):
#0 0x00007f4dc956bfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 4263):
#0 0x00007f4dc956bfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 4262):
#0 0x00007f4dc956bfb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f4dbd2bc410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 4258):
#0 0x00007f4dc956fd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250111 02:58:23.192440 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 3254
I20250111 02:58:23.199990 3380 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:58:23.330448 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 3254
I20250111 02:58:23.357686 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 3393
I20250111 02:58:23.364336 3520 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:58:23.501438 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 3393
I20250111 02:58:23.527963 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 3535
I20250111 02:58:23.534884 3667 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:58:23.949705 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 3535
W20250111 02:58:24.015167 1125 connection.cc:537] server connection from 127.1.19.131:38647 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250111 02:58:24.015470 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 4929
W20250111 02:58:24.016695 4278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.19.131:37075: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20250111 02:58:24.017908 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.131:37075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:58:24.022114 5065 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:58:24.442212 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 4929
I20250111 02:58:24.496452 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 2825
W20250111 02:58:24.497259 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.132:41691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:58:24.503052 2952 generic_service.cc:196] Checking for leaks (request via RPC)
W20250111 02:58:24.541415 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.131:37075: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250111 02:58:24.639515 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 2825
I20250111 02:58:24.666011 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 2547
I20250111 02:58:24.672535 2673 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:58:24.807690 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 2547
I20250111 02:58:24.834244 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 2687
I20250111 02:58:24.840708 2812 generic_service.cc:196] Checking for leaks (request via RPC)
W20250111 02:58:24.881186 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 185ad035c2754007b1b3d08bb65546c2 (127.1.19.132:41691): Couldn't send request to peer 185ad035c2754007b1b3d08bb65546c2. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.132:41691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250111 02:58:24.974857 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 2687
W20250111 02:58:24.983891 4278 consensus_peers.cc:487] T a851ac2cebaa49d5b317fd02bc2766fa P 560d9bddebd54e0da0bf8d2da0a7a593 -> Peer 486f7fe875934cebb51c6717b5e4312d (127.1.19.131:37075): Couldn't send request to peer 486f7fe875934cebb51c6717b5e4312d. Status: Network error: Client connection negotiation failed: client connection to 127.1.19.131:37075: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250111 02:58:25.001335 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 4117
I20250111 02:58:25.007634 4241 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:58:25.126855 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 4117
I20250111 02:58:25.151304 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 4258
I20250111 02:58:25.157601 4389 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:58:25.582923 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 4258
W20250111 02:58:25.669231 1125 connection.cc:537] server connection from 127.1.19.137:46543 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250111 02:58:25.669811 1102 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu pid 1110
I20250111 02:58:25.678603 1170 generic_service.cc:196] Checking for leaks (request via RPC)
I20250111 02:58:25.885674 1102 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskaChIIQ/build/asan/bin/kudu with pid 1110
2025-01-11T02:58:25Z chronyd exiting
I20250111 02:58:25.943838 1102 test_util.cc:181] -----------------------------------------------
I20250111 02:58:25.943933 1102 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskaChIIQ/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1736564068598300-1102-0
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3, where GetParam() = 800-byte object <01-00 00-00 01-61 00-00 09-00 00-00 BE-BE BE-BE A0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-1F 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 03-00 00-00 03-00 00-00 00-BE BE-BE BE-BE BE-BE> (237273 ms)
[----------] 1 test from RollingRestartArgs/RollingRestartITest (237273 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (237273 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3, where GetParam() = 800-byte object <01-00 00-00 01-61 00-00 09-00 00-00 BE-BE BE-BE A0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-1F 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 03-00 00-00 03-00 00-00 00-BE BE-BE BE-BE BE-BE>
1 FAILED TEST