Note: This is test shard 8 of 8.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RollingRestartArgs/RollingRestartITest
[ RUN ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3
2024-05-06T16:15:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-06T16:15:43Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240506 16:15:43.245054 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.173.190:43641
--webserver_interface=127.12.173.190
--webserver_port=0
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.173.190:43641
--location_mapping_cmd=/tmp/dist-test-taskGIku5t/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/location-assignment.state --map /L0:3 --map /L1:4 --map /L2:4
--master_client_location_assignment_enabled=false with env {}
W20240506 16:15:43.593156 12990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:15:43.593569 12990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:15:43.593665 12990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:15:43.601393 12990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240506 16:15:43.601547 12990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:15:43.601593 12990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240506 16:15:43.601651 12990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240506 16:15:43.611119 12990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/wal
--location_mapping_cmd=/tmp/dist-test-taskGIku5t/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/location-assignment.state --map /L0:3 --map /L1:4 --map /L2:4
--ipki_ca_key_size=768
--master_addresses=127.12.173.190:43641
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.173.190:43641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.173.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:15:43.612529 12990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:15:43.614399 12990 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:15:43.625872 12995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:43.626215 12996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:43.626585 12998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:15:43.627045 12990 server_base.cc:1034] running on GCE node
I20240506 16:15:43.627851 12990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:15:43.629351 12990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:15:43.630586 12990 hybrid_clock.cc:648] HybridClock initialized: now 1715012143630516 us; error 47 us; skew 500 ppm
I20240506 16:15:43.630930 12990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:15:43.633409 12990 webserver.cc:460] Webserver started at http://127.12.173.190:41535/ using document root <none> and password file <none>
I20240506 16:15:43.634145 12990 fs_manager.cc:359] Metadata directory not provided
I20240506 16:15:43.634274 12990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:15:43.634616 12990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 16:15:43.637470 12990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/data/instance:
uuid: "e31cda4593704d9c9f2ef36e57bde84a"
format_stamp: "Formatted at 2024-05-06 16:15:43 on dist-test-slave-g936"
I20240506 16:15:43.638265 12990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/wal/instance:
uuid: "e31cda4593704d9c9f2ef36e57bde84a"
format_stamp: "Formatted at 2024-05-06 16:15:43 on dist-test-slave-g936"
I20240506 16:15:43.643630 12990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240506 16:15:43.647127 13004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:43.648550 12990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240506 16:15:43.648722 12990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/wal
uuid: "e31cda4593704d9c9f2ef36e57bde84a"
format_stamp: "Formatted at 2024-05-06 16:15:43 on dist-test-slave-g936"
I20240506 16:15:43.648902 12990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:15:43.671061 12990 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:15:43.672004 12990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:15:43.672294 12990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:15:43.691196 12990 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.190:43641
I20240506 16:15:43.691215 13056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.190:43641 every 8 connection(s)
I20240506 16:15:43.692741 12990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/data/info.pb
I20240506 16:15:43.695335 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 12990
I20240506 16:15:43.695647 12982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/wal/instance
I20240506 16:15:43.697566 13057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 16:15:43.708639 13057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a: Bootstrap starting.
I20240506 16:15:43.711827 13057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a: Neither blocks nor log segments found. Creating new log.
I20240506 16:15:43.713348 13057 log.cc:824] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a: Log is configured to *not* fsync() on all Append() calls
I20240506 16:15:43.716251 13057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a: No bootstrap required, opened a new log
I20240506 16:15:43.724480 13057 raft_consensus.cc:348] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31cda4593704d9c9f2ef36e57bde84a" member_type: VOTER last_known_addr { host: "127.12.173.190" port: 43641 } }
I20240506 16:15:43.724850 13057 raft_consensus.cc:374] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 16:15:43.724972 13057 raft_consensus.cc:729] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31cda4593704d9c9f2ef36e57bde84a, State: Initialized, Role: FOLLOWER
I20240506 16:15:43.725629 13057 consensus_queue.cc:260] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31cda4593704d9c9f2ef36e57bde84a" member_type: VOTER last_known_addr { host: "127.12.173.190" port: 43641 } }
I20240506 16:15:43.725832 13057 raft_consensus.cc:388] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240506 16:15:43.725965 13057 raft_consensus.cc:482] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240506 16:15:43.726135 13057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [term 0 FOLLOWER]: Advancing to term 1
I20240506 16:15:43.728241 13057 raft_consensus.cc:504] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31cda4593704d9c9f2ef36e57bde84a" member_type: VOTER last_known_addr { host: "127.12.173.190" port: 43641 } }
I20240506 16:15:43.728711 13057 leader_election.cc:302] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e31cda4593704d9c9f2ef36e57bde84a; no voters:
I20240506 16:15:43.729342 13057 leader_election.cc:288] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240506 16:15:43.729439 13062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [term 1 FOLLOWER]: Leader election won for term 1
I20240506 16:15:43.730221 13062 raft_consensus.cc:686] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [term 1 LEADER]: Becoming Leader. State: Replica: e31cda4593704d9c9f2ef36e57bde84a, State: Running, Role: LEADER
I20240506 16:15:43.730772 13062 consensus_queue.cc:237] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31cda4593704d9c9f2ef36e57bde84a" member_type: VOTER last_known_addr { host: "127.12.173.190" port: 43641 } }
I20240506 16:15:43.731449 13057 sys_catalog.cc:564] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [sys.catalog]: configured and running, proceeding with master startup.
I20240506 16:15:43.736868 13063 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e31cda4593704d9c9f2ef36e57bde84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31cda4593704d9c9f2ef36e57bde84a" member_type: VOTER last_known_addr { host: "127.12.173.190" port: 43641 } } }
I20240506 16:15:43.737488 13063 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [sys.catalog]: This master's current role is: LEADER
I20240506 16:15:43.736888 13064 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e31cda4593704d9c9f2ef36e57bde84a. Latest consensus state: current_term: 1 leader_uuid: "e31cda4593704d9c9f2ef36e57bde84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31cda4593704d9c9f2ef36e57bde84a" member_type: VOTER last_known_addr { host: "127.12.173.190" port: 43641 } } }
I20240506 16:15:43.738058 13064 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a [sys.catalog]: This master's current role is: LEADER
I20240506 16:15:43.743062 13069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240506 16:15:43.749250 13069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240506 16:15:43.759830 13069 catalog_manager.cc:1347] Generated new cluster ID: 2d414fdaac5b4bb19c40c39510dc2d50
I20240506 16:15:43.759966 13069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240506 16:15:43.780006 13069 catalog_manager.cc:1370] Generated new certificate authority record
I20240506 16:15:43.781399 13069 catalog_manager.cc:1504] Loading token signing keys...
I20240506 16:15:43.792977 13069 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e31cda4593704d9c9f2ef36e57bde84a: Generated new TSK 0
I20240506 16:15:43.793949 13069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240506 16:15:43.809362 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.129:0
--local_ip_for_outbound_sockets=127.12.173.129
--webserver_interface=127.12.173.129
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20240506 16:15:44.179020 13081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:15:44.179330 13081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:15:44.179466 13081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:15:44.187141 13081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:15:44.187371 13081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.129
I20240506 16:15:44.197757 13081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.173.129
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.129
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:15:44.199249 13081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:15:44.201072 13081 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:15:44.214732 13086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:44.215112 13089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:44.215430 13087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:15:44.215978 13081 server_base.cc:1034] running on GCE node
I20240506 16:15:44.216632 13081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:15:44.217792 13081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:15:44.219048 13081 hybrid_clock.cc:648] HybridClock initialized: now 1715012144218983 us; error 39 us; skew 500 ppm
I20240506 16:15:44.219360 13081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:15:44.226912 13081 webserver.cc:460] Webserver started at http://127.12.173.129:39161/ using document root <none> and password file <none>
I20240506 16:15:44.227715 13081 fs_manager.cc:359] Metadata directory not provided
I20240506 16:15:44.227834 13081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:15:44.228104 13081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 16:15:44.230962 13081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data/instance:
uuid: "15538fc3b6084a96a18216a2ec74cc3a"
format_stamp: "Formatted at 2024-05-06 16:15:44 on dist-test-slave-g936"
I20240506 16:15:44.231838 13081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/wal/instance:
uuid: "15538fc3b6084a96a18216a2ec74cc3a"
format_stamp: "Formatted at 2024-05-06 16:15:44 on dist-test-slave-g936"
I20240506 16:15:44.237416 13081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240506 16:15:44.240985 13095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:44.242537 13081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240506 16:15:44.242741 13081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/wal
uuid: "15538fc3b6084a96a18216a2ec74cc3a"
format_stamp: "Formatted at 2024-05-06 16:15:44 on dist-test-slave-g936"
I20240506 16:15:44.242925 13081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:15:44.261979 13081 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:15:44.262851 13081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:15:44.263110 13081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:15:44.264147 13081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:15:44.265821 13081 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:15:44.265901 13081 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:44.266002 13081 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:15:44.266047 13081 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:44.300128 13081 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.129:43195
I20240506 16:15:44.300165 13207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.129:43195 every 8 connection(s)
I20240506 16:15:44.301635 13081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data/info.pb
I20240506 16:15:44.311622 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 13081
I20240506 16:15:44.311861 12982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/wal/instance
I20240506 16:15:44.315706 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.130:0
--local_ip_for_outbound_sockets=127.12.173.130
--webserver_interface=127.12.173.130
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20240506 16:15:44.316045 13208 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:15:44.316428 13208 heartbeater.cc:464] Registering TS with master...
I20240506 16:15:44.317741 13208 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:15:44.380697 13022 ts_manager.cc:194] Registered new tserver with Master: 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195)
I20240506 16:15:44.382599 13022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.129:37127
W20240506 16:15:44.682001 13212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:15:44.682345 13212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:15:44.682519 13212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:15:44.690519 13212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:15:44.690788 13212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.130
I20240506 16:15:44.700352 13212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.173.130
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.130
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:15:44.701802 13212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:15:44.703635 13212 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:15:44.716490 13218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:44.716893 13219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:44.717372 13221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:15:44.717916 13212 server_base.cc:1034] running on GCE node
I20240506 16:15:44.718593 13212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:15:44.719655 13212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:15:44.720882 13212 hybrid_clock.cc:648] HybridClock initialized: now 1715012144720822 us; error 39 us; skew 500 ppm
I20240506 16:15:44.721197 13212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:15:44.723620 13212 webserver.cc:460] Webserver started at http://127.12.173.130:34765/ using document root <none> and password file <none>
I20240506 16:15:44.724473 13212 fs_manager.cc:359] Metadata directory not provided
I20240506 16:15:44.724622 13212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:15:44.724956 13212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 16:15:44.727836 13212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data/instance:
uuid: "cea103e071de48bbbad5e030eca24b4a"
format_stamp: "Formatted at 2024-05-06 16:15:44 on dist-test-slave-g936"
I20240506 16:15:44.728614 13212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/wal/instance:
uuid: "cea103e071de48bbbad5e030eca24b4a"
format_stamp: "Formatted at 2024-05-06 16:15:44 on dist-test-slave-g936"
I20240506 16:15:44.734527 13212 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240506 16:15:44.738225 13227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:44.739704 13212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240506 16:15:44.739899 13212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/wal
uuid: "cea103e071de48bbbad5e030eca24b4a"
format_stamp: "Formatted at 2024-05-06 16:15:44 on dist-test-slave-g936"
I20240506 16:15:44.740101 13212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:15:44.774778 13212 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:15:44.775700 13212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:15:44.775961 13212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:15:44.776886 13212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:15:44.778391 13212 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:15:44.778466 13212 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:44.778563 13212 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:15:44.778606 13212 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:44.813170 13212 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.130:42665
I20240506 16:15:44.813274 13339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.130:42665 every 8 connection(s)
I20240506 16:15:44.814601 13212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data/info.pb
I20240506 16:15:44.816256 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 13212
I20240506 16:15:44.816426 12982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/wal/instance
I20240506 16:15:44.819909 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.131:0
--local_ip_for_outbound_sockets=127.12.173.131
--webserver_interface=127.12.173.131
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20240506 16:15:44.829483 13340 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:15:44.829799 13340 heartbeater.cc:464] Registering TS with master...
I20240506 16:15:44.830710 13340 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:15:44.874979 13022 ts_manager.cc:194] Registered new tserver with Master: cea103e071de48bbbad5e030eca24b4a (127.12.173.130:42665)
I20240506 16:15:44.876302 13022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.130:36441
W20240506 16:15:45.163504 13344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:15:45.163841 13344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:15:45.163945 13344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:15:45.171559 13344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:15:45.171772 13344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.131
I20240506 16:15:45.181190 13344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.173.131
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.131
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:15:45.182623 13344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:15:45.184406 13344 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:15:45.198107 13350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:45.198465 13353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:45.198979 13351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:15:45.199607 13344 server_base.cc:1034] running on GCE node
I20240506 16:15:45.200242 13344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:15:45.201270 13344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:15:45.202497 13344 hybrid_clock.cc:648] HybridClock initialized: now 1715012145202434 us; error 46 us; skew 500 ppm
I20240506 16:15:45.202805 13344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:15:45.205137 13344 webserver.cc:460] Webserver started at http://127.12.173.131:38629/ using document root <none> and password file <none>
I20240506 16:15:45.205929 13344 fs_manager.cc:359] Metadata directory not provided
I20240506 16:15:45.206063 13344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:15:45.206360 13344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 16:15:45.209045 13344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data/instance:
uuid: "405f2b51c2244be3b1a66f2c908bcdd9"
format_stamp: "Formatted at 2024-05-06 16:15:45 on dist-test-slave-g936"
I20240506 16:15:45.209820 13344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/wal/instance:
uuid: "405f2b51c2244be3b1a66f2c908bcdd9"
format_stamp: "Formatted at 2024-05-06 16:15:45 on dist-test-slave-g936"
I20240506 16:15:45.215296 13344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240506 16:15:45.219074 13359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:45.220547 13344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240506 16:15:45.220731 13344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/wal
uuid: "405f2b51c2244be3b1a66f2c908bcdd9"
format_stamp: "Formatted at 2024-05-06 16:15:45 on dist-test-slave-g936"
I20240506 16:15:45.220925 13344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:15:45.278004 13344 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:15:45.278906 13344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:15:45.279196 13344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:15:45.280175 13344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:15:45.281736 13344 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:15:45.281818 13344 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:45.281950 13344 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:15:45.282006 13344 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:45.314510 13344 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.131:33797
I20240506 16:15:45.314544 13471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.131:33797 every 8 connection(s)
I20240506 16:15:45.315806 13344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data/info.pb
I20240506 16:15:45.320518 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 13344
I20240506 16:15:45.320750 12982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/wal/instance
I20240506 16:15:45.324363 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.132:0
--local_ip_for_outbound_sockets=127.12.173.132
--webserver_interface=127.12.173.132
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20240506 16:15:45.329805 13472 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:15:45.330173 13472 heartbeater.cc:464] Registering TS with master...
I20240506 16:15:45.331210 13472 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:15:45.376966 13022 ts_manager.cc:194] Registered new tserver with Master: 405f2b51c2244be3b1a66f2c908bcdd9 (127.12.173.131:33797)
I20240506 16:15:45.378285 13022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.131:58357
I20240506 16:15:45.385882 13208 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
W20240506 16:15:45.665436 13475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:15:45.665750 13475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:15:45.665853 13475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:15:45.673524 13475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:15:45.673719 13475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.132
I20240506 16:15:45.683017 13475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.12.173.132
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.132
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:15:45.684347 13475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:15:45.686216 13475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:15:45.698596 13481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:45.699399 13482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:45.700312 13484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:15:45.701371 13475 server_base.cc:1034] running on GCE node
I20240506 16:15:45.702164 13475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:15:45.703356 13475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:15:45.704617 13475 hybrid_clock.cc:648] HybridClock initialized: now 1715012145704541 us; error 47 us; skew 500 ppm
I20240506 16:15:45.704916 13475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:15:45.707451 13475 webserver.cc:460] Webserver started at http://127.12.173.132:44537/ using document root <none> and password file <none>
I20240506 16:15:45.708212 13475 fs_manager.cc:359] Metadata directory not provided
I20240506 16:15:45.708331 13475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:15:45.708621 13475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 16:15:45.711403 13475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data/instance:
uuid: "e6d1cddba8c94532b40c0ae9c06740ef"
format_stamp: "Formatted at 2024-05-06 16:15:45 on dist-test-slave-g936"
I20240506 16:15:45.712137 13475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/wal/instance:
uuid: "e6d1cddba8c94532b40c0ae9c06740ef"
format_stamp: "Formatted at 2024-05-06 16:15:45 on dist-test-slave-g936"
I20240506 16:15:45.717679 13475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240506 16:15:45.721158 13490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:45.722700 13475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240506 16:15:45.722868 13475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/wal
uuid: "e6d1cddba8c94532b40c0ae9c06740ef"
format_stamp: "Formatted at 2024-05-06 16:15:45 on dist-test-slave-g936"
I20240506 16:15:45.723058 13475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:15:45.745292 13475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:15:45.746160 13475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:15:45.746454 13475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:15:45.747439 13475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:15:45.749118 13475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:15:45.749195 13475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:45.749367 13475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:15:45.749491 13475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:45.781810 13475 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.132:39055
I20240506 16:15:45.781850 13602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.132:39055 every 8 connection(s)
I20240506 16:15:45.783149 13475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data/info.pb
I20240506 16:15:45.784667 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 13475
I20240506 16:15:45.784837 12982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/wal/instance
I20240506 16:15:45.788812 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.133:0
--local_ip_for_outbound_sockets=127.12.173.133
--webserver_interface=127.12.173.133
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20240506 16:15:45.798089 13603 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:15:45.798416 13603 heartbeater.cc:464] Registering TS with master...
I20240506 16:15:45.799260 13603 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:15:45.859038 13022 ts_manager.cc:194] Registered new tserver with Master: e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055)
I20240506 16:15:45.860013 13022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.132:59627
I20240506 16:15:45.880262 13340 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
W20240506 16:15:46.149298 13606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:15:46.149643 13606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:15:46.149798 13606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:15:46.157511 13606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:15:46.157759 13606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.133
I20240506 16:15:46.166961 13606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data/info.pb
--webserver_interface=127.12.173.133
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.133
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:15:46.168388 13606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:15:46.170311 13606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:15:46.183224 13615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:46.183648 13613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:46.184127 13612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:15:46.184707 13606 server_base.cc:1034] running on GCE node
I20240506 16:15:46.185384 13606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:15:46.186371 13606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:15:46.187583 13606 hybrid_clock.cc:648] HybridClock initialized: now 1715012146187522 us; error 39 us; skew 500 ppm
I20240506 16:15:46.187902 13606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:15:46.190341 13606 webserver.cc:460] Webserver started at http://127.12.173.133:33855/ using document root <none> and password file <none>
I20240506 16:15:46.191097 13606 fs_manager.cc:359] Metadata directory not provided
I20240506 16:15:46.191226 13606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:15:46.191529 13606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 16:15:46.194392 13606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data/instance:
uuid: "fd8906dd0f884e22911dce7df910b6d4"
format_stamp: "Formatted at 2024-05-06 16:15:46 on dist-test-slave-g936"
I20240506 16:15:46.195190 13606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/wal/instance:
uuid: "fd8906dd0f884e22911dce7df910b6d4"
format_stamp: "Formatted at 2024-05-06 16:15:46 on dist-test-slave-g936"
I20240506 16:15:46.200865 13606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240506 16:15:46.204445 13621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:46.205857 13606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240506 16:15:46.206023 13606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/wal
uuid: "fd8906dd0f884e22911dce7df910b6d4"
format_stamp: "Formatted at 2024-05-06 16:15:46 on dist-test-slave-g936"
I20240506 16:15:46.206208 13606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:15:46.226992 13606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:15:46.227869 13606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:15:46.228143 13606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:15:46.229096 13606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:15:46.230760 13606 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:15:46.230832 13606 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:46.230926 13606 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:15:46.230969 13606 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:46.262971 13606 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.133:42687
I20240506 16:15:46.263012 13733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.133:42687 every 8 connection(s)
I20240506 16:15:46.264266 13606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data/info.pb
I20240506 16:15:46.270623 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 13606
I20240506 16:15:46.270838 12982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/wal/instance
I20240506 16:15:46.274519 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.134:0
--local_ip_for_outbound_sockets=127.12.173.134
--webserver_interface=127.12.173.134
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20240506 16:15:46.278963 13734 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:15:46.279274 13734 heartbeater.cc:464] Registering TS with master...
I20240506 16:15:46.280110 13734 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:15:46.326539 13022 ts_manager.cc:194] Registered new tserver with Master: fd8906dd0f884e22911dce7df910b6d4 (127.12.173.133:42687)
I20240506 16:15:46.327679 13022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.133:48291
I20240506 16:15:46.381528 13472 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
W20240506 16:15:46.619725 13737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:15:46.620060 13737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:15:46.620209 13737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:15:46.627916 13737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:15:46.628157 13737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.134
I20240506 16:15:46.637676 13737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data/info.pb
--webserver_interface=127.12.173.134
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.134
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:15:46.639124 13737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:15:46.640941 13737 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:15:46.654032 13744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:46.654454 13743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:46.654945 13746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:15:46.655440 13737 server_base.cc:1034] running on GCE node
I20240506 16:15:46.656049 13737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:15:46.657038 13737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:15:46.658277 13737 hybrid_clock.cc:648] HybridClock initialized: now 1715012146658211 us; error 46 us; skew 500 ppm
I20240506 16:15:46.658571 13737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:15:46.660948 13737 webserver.cc:460] Webserver started at http://127.12.173.134:39891/ using document root <none> and password file <none>
I20240506 16:15:46.661728 13737 fs_manager.cc:359] Metadata directory not provided
I20240506 16:15:46.661849 13737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:15:46.662130 13737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 16:15:46.664862 13737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data/instance:
uuid: "c0471a9ec7ba4ea38e8df76c78d8b38f"
format_stamp: "Formatted at 2024-05-06 16:15:46 on dist-test-slave-g936"
I20240506 16:15:46.665640 13737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/wal/instance:
uuid: "c0471a9ec7ba4ea38e8df76c78d8b38f"
format_stamp: "Formatted at 2024-05-06 16:15:46 on dist-test-slave-g936"
I20240506 16:15:46.671139 13737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240506 16:15:46.674744 13752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:46.676187 13737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240506 16:15:46.676344 13737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/wal
uuid: "c0471a9ec7ba4ea38e8df76c78d8b38f"
format_stamp: "Formatted at 2024-05-06 16:15:46 on dist-test-slave-g936"
I20240506 16:15:46.676528 13737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:15:46.700778 13737 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:15:46.701685 13737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:15:46.701953 13737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:15:46.702888 13737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:15:46.704486 13737 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:15:46.704563 13737 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:46.704651 13737 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:15:46.704695 13737 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:46.737257 13737 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.134:36633
I20240506 16:15:46.737329 13864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.134:36633 every 8 connection(s)
I20240506 16:15:46.738612 13737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data/info.pb
I20240506 16:15:46.745558 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 13737
I20240506 16:15:46.745754 12982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/wal/instance
I20240506 16:15:46.749011 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.135:0
--local_ip_for_outbound_sockets=127.12.173.135
--webserver_interface=127.12.173.135
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20240506 16:15:46.752677 13865 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:15:46.753077 13865 heartbeater.cc:464] Registering TS with master...
I20240506 16:15:46.754217 13865 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:15:46.813859 13022 ts_manager.cc:194] Registered new tserver with Master: c0471a9ec7ba4ea38e8df76c78d8b38f (127.12.173.134:36633)
I20240506 16:15:46.815136 13022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.134:35289
I20240506 16:15:46.862953 13603 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
W20240506 16:15:47.108906 13868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:15:47.109356 13868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:15:47.109510 13868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:15:47.117393 13868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:15:47.117621 13868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.135
I20240506 16:15:47.126994 13868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data/info.pb
--webserver_interface=127.12.173.135
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.135
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:15:47.128422 13868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:15:47.130344 13868 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:15:47.143293 13877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:47.143200 13875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:47.143859 13874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:15:47.144371 13868 server_base.cc:1034] running on GCE node
I20240506 16:15:47.145013 13868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:15:47.146087 13868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:15:47.147293 13868 hybrid_clock.cc:648] HybridClock initialized: now 1715012147147220 us; error 55 us; skew 500 ppm
I20240506 16:15:47.147624 13868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:15:47.150070 13868 webserver.cc:460] Webserver started at http://127.12.173.135:45737/ using document root <none> and password file <none>
I20240506 16:15:47.150861 13868 fs_manager.cc:359] Metadata directory not provided
I20240506 16:15:47.150982 13868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:15:47.151298 13868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 16:15:47.154495 13868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data/instance:
uuid: "18c7d19aee15430b8c3a0ef90b23fee5"
format_stamp: "Formatted at 2024-05-06 16:15:47 on dist-test-slave-g936"
I20240506 16:15:47.155383 13868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/wal/instance:
uuid: "18c7d19aee15430b8c3a0ef90b23fee5"
format_stamp: "Formatted at 2024-05-06 16:15:47 on dist-test-slave-g936"
I20240506 16:15:47.161083 13868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240506 16:15:47.164829 13883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:47.166313 13868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240506 16:15:47.166491 13868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/wal
uuid: "18c7d19aee15430b8c3a0ef90b23fee5"
format_stamp: "Formatted at 2024-05-06 16:15:47 on dist-test-slave-g936"
I20240506 16:15:47.166699 13868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:15:47.194260 13868 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:15:47.195173 13868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:15:47.195462 13868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:15:47.196517 13868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:15:47.198201 13868 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:15:47.198287 13868 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:47.198410 13868 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:15:47.198467 13868 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:47.231344 13868 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.135:43823
I20240506 16:15:47.231388 13995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.135:43823 every 8 connection(s)
I20240506 16:15:47.232707 13868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data/info.pb
I20240506 16:15:47.239881 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 13868
I20240506 16:15:47.240119 12982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/wal/instance
I20240506 16:15:47.244107 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.136:0
--local_ip_for_outbound_sockets=127.12.173.136
--webserver_interface=127.12.173.136
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20240506 16:15:47.247763 13996 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:15:47.248162 13996 heartbeater.cc:464] Registering TS with master...
I20240506 16:15:47.249186 13996 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:15:47.322229 13022 ts_manager.cc:194] Registered new tserver with Master: 18c7d19aee15430b8c3a0ef90b23fee5 (127.12.173.135:43823)
I20240506 16:15:47.323232 13022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.135:38415
I20240506 16:15:47.330639 13734 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
W20240506 16:15:47.602816 13999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:15:47.603133 13999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:15:47.603235 13999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:15:47.611265 13999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:15:47.611497 13999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.136
I20240506 16:15:47.621115 13999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data/info.pb
--webserver_interface=127.12.173.136
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.136
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:15:47.622551 13999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:15:47.624302 13999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:15:47.637609 14005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:47.637988 14006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:47.638288 14008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:15:47.638855 13999 server_base.cc:1034] running on GCE node
I20240506 16:15:47.639465 13999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:15:47.640424 13999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:15:47.641670 13999 hybrid_clock.cc:648] HybridClock initialized: now 1715012147641596 us; error 43 us; skew 500 ppm
I20240506 16:15:47.642001 13999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:15:47.644412 13999 webserver.cc:460] Webserver started at http://127.12.173.136:36367/ using document root <none> and password file <none>
I20240506 16:15:47.645228 13999 fs_manager.cc:359] Metadata directory not provided
I20240506 16:15:47.645361 13999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:15:47.645675 13999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 16:15:47.648463 13999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data/instance:
uuid: "dc9fadb1e71e44c18dbade6135728983"
format_stamp: "Formatted at 2024-05-06 16:15:47 on dist-test-slave-g936"
I20240506 16:15:47.649235 13999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/wal/instance:
uuid: "dc9fadb1e71e44c18dbade6135728983"
format_stamp: "Formatted at 2024-05-06 16:15:47 on dist-test-slave-g936"
I20240506 16:15:47.654824 13999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240506 16:15:47.658349 14014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:47.659736 13999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240506 16:15:47.659911 13999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/wal
uuid: "dc9fadb1e71e44c18dbade6135728983"
format_stamp: "Formatted at 2024-05-06 16:15:47 on dist-test-slave-g936"
I20240506 16:15:47.660099 13999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:15:47.677737 13999 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:15:47.678629 13999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:15:47.678915 13999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:15:47.679895 13999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:15:47.681540 13999 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:15:47.681648 13999 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:47.681766 13999 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:15:47.681840 13999 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:47.721457 13999 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.136:34037
I20240506 16:15:47.721493 14126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.136:34037 every 8 connection(s)
I20240506 16:15:47.722811 13999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data/info.pb
I20240506 16:15:47.724637 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 13999
I20240506 16:15:47.724836 12982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/wal/instance
I20240506 16:15:47.728562 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.137:0
--local_ip_for_outbound_sockets=127.12.173.137
--webserver_interface=127.12.173.137
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20240506 16:15:47.739225 14127 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:15:47.739569 14127 heartbeater.cc:464] Registering TS with master...
I20240506 16:15:47.740701 14127 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:15:47.789186 13022 ts_manager.cc:194] Registered new tserver with Master: dc9fadb1e71e44c18dbade6135728983 (127.12.173.136:34037)
I20240506 16:15:47.790498 13022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.136:41387
I20240506 16:15:47.818071 13865 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
W20240506 16:15:48.095990 14130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:15:48.096314 14130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:15:48.096416 14130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:15:48.104007 14130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:15:48.104202 14130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.137
I20240506 16:15:48.113907 14130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.137:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data/info.pb
--webserver_interface=127.12.173.137
--webserver_port=0
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.137
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:15:48.115276 14130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:15:48.117051 14130 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:15:48.130321 14139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:48.130288 14137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:48.130861 14136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:15:48.131366 14130 server_base.cc:1034] running on GCE node
I20240506 16:15:48.131986 14130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:15:48.132990 14130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:15:48.134207 14130 hybrid_clock.cc:648] HybridClock initialized: now 1715012148134135 us; error 56 us; skew 500 ppm
I20240506 16:15:48.134500 14130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:15:48.136859 14130 webserver.cc:460] Webserver started at http://127.12.173.137:39563/ using document root <none> and password file <none>
I20240506 16:15:48.137621 14130 fs_manager.cc:359] Metadata directory not provided
I20240506 16:15:48.137760 14130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:15:48.138048 14130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240506 16:15:48.140621 14130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data/instance:
uuid: "5beda4231be140b8959d934a86e21bb1"
format_stamp: "Formatted at 2024-05-06 16:15:48 on dist-test-slave-g936"
I20240506 16:15:48.141377 14130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/wal/instance:
uuid: "5beda4231be140b8959d934a86e21bb1"
format_stamp: "Formatted at 2024-05-06 16:15:48 on dist-test-slave-g936"
I20240506 16:15:48.147274 14130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240506 16:15:48.150808 14145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:48.152156 14130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240506 16:15:48.152318 14130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/wal
uuid: "5beda4231be140b8959d934a86e21bb1"
format_stamp: "Formatted at 2024-05-06 16:15:48 on dist-test-slave-g936"
I20240506 16:15:48.152500 14130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:15:48.168593 14130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:15:48.169490 14130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:15:48.169773 14130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:15:48.170763 14130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:15:48.172281 14130 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:15:48.172362 14130 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:48.172489 14130 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:15:48.172545 14130 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:48.208305 14130 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.137:37831
I20240506 16:15:48.208360 14257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.137:37831 every 8 connection(s)
I20240506 16:15:48.209643 14130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data/info.pb
I20240506 16:15:48.219899 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 14130
I20240506 16:15:48.220106 12982 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/wal/instance
I20240506 16:15:48.223114 14258 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:15:48.223515 14258 heartbeater.cc:464] Registering TS with master...
I20240506 16:15:48.224603 14258 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:15:48.276027 13022 ts_manager.cc:194] Registered new tserver with Master: 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831)
I20240506 16:15:48.277034 13022 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.137:41437
I20240506 16:15:48.288841 12982 external_mini_cluster.cc:930] 9 TS(s) registered with all masters
I20240506 16:15:48.311463 12982 test_util.cc:268] Using random seed: 1642247550
I20240506 16:15:48.326203 13996 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
I20240506 16:15:48.335538 13022 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40380:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240506 16:15:48.367488 13143 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea58089cc7849fbabaa4bde5c75d45f (DEFAULT_TABLE table=test-workload [id=1f3fff06f6e94496bd5d3508feef68c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 16:15:48.367820 13538 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea58089cc7849fbabaa4bde5c75d45f (DEFAULT_TABLE table=test-workload [id=1f3fff06f6e94496bd5d3508feef68c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 16:15:48.368875 13143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea58089cc7849fbabaa4bde5c75d45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 16:15:48.369304 13538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea58089cc7849fbabaa4bde5c75d45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 16:15:48.369779 14193 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea58089cc7849fbabaa4bde5c75d45f (DEFAULT_TABLE table=test-workload [id=1f3fff06f6e94496bd5d3508feef68c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240506 16:15:48.371235 14193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea58089cc7849fbabaa4bde5c75d45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240506 16:15:48.380649 14282 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Bootstrap starting.
I20240506 16:15:48.384521 14282 tablet_bootstrap.cc:654] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Neither blocks nor log segments found. Creating new log.
I20240506 16:15:48.384838 14284 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap starting.
I20240506 16:15:48.387506 14282 log.cc:824] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Log is configured to *not* fsync() on all Append() calls
I20240506 16:15:48.388384 14283 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Bootstrap starting.
I20240506 16:15:48.389338 14284 tablet_bootstrap.cc:654] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Neither blocks nor log segments found. Creating new log.
I20240506 16:15:48.390709 14284 log.cc:824] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Log is configured to *not* fsync() on all Append() calls
I20240506 16:15:48.391175 14282 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: No bootstrap required, opened a new log
I20240506 16:15:48.391572 14282 ts_tablet_manager.cc:1367] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240506 16:15:48.392115 14283 tablet_bootstrap.cc:654] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Neither blocks nor log segments found. Creating new log.
I20240506 16:15:48.393468 14283 log.cc:824] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Log is configured to *not* fsync() on all Append() calls
I20240506 16:15:48.398387 14283 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: No bootstrap required, opened a new log
I20240506 16:15:48.398662 14283 ts_tablet_manager.cc:1367] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240506 16:15:48.399742 14284 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: No bootstrap required, opened a new log
I20240506 16:15:48.400058 14284 ts_tablet_manager.cc:1367] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20240506 16:15:48.401556 14282 raft_consensus.cc:348] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:15:48.402026 14282 raft_consensus.cc:374] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 16:15:48.402186 14282 raft_consensus.cc:729] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15538fc3b6084a96a18216a2ec74cc3a, State: Initialized, Role: FOLLOWER
I20240506 16:15:48.402962 14282 consensus_queue.cc:260] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:15:48.404987 14282 ts_tablet_manager.cc:1398] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20240506 16:15:48.407735 14283 raft_consensus.cc:348] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:15:48.408133 14283 raft_consensus.cc:374] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 16:15:48.408298 14283 raft_consensus.cc:729] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6d1cddba8c94532b40c0ae9c06740ef, State: Initialized, Role: FOLLOWER
I20240506 16:15:48.408622 14284 raft_consensus.cc:348] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:15:48.409065 14284 raft_consensus.cc:374] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240506 16:15:48.409176 14284 raft_consensus.cc:729] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beda4231be140b8959d934a86e21bb1, State: Initialized, Role: FOLLOWER
I20240506 16:15:48.409478 14283 consensus_queue.cc:260] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:15:48.410698 14284 consensus_queue.cc:260] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:15:48.411643 14283 ts_tablet_manager.cc:1398] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Time spent starting tablet: real 0.013s user 0.010s sys 0.000s
I20240506 16:15:48.411777 14258 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
I20240506 16:15:48.412667 14284 ts_tablet_manager.cc:1398] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20240506 16:15:48.441718 14290 raft_consensus.cc:482] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 16:15:48.442023 14290 raft_consensus.cc:504] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:15:48.446878 14290 leader_election.cc:288] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195), e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055)
I20240506 16:15:48.453402 13558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "5beda4231be140b8959d934a86e21bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" is_pre_election: true
I20240506 16:15:48.454128 13558 raft_consensus.cc:2454] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beda4231be140b8959d934a86e21bb1 in term 0.
I20240506 16:15:48.454469 13163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "5beda4231be140b8959d934a86e21bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15538fc3b6084a96a18216a2ec74cc3a" is_pre_election: true
I20240506 16:15:48.454967 13163 raft_consensus.cc:2454] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beda4231be140b8959d934a86e21bb1 in term 0.
I20240506 16:15:48.454983 14148 leader_election.cc:302] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beda4231be140b8959d934a86e21bb1, e6d1cddba8c94532b40c0ae9c06740ef; no voters:
I20240506 16:15:48.455605 14290 raft_consensus.cc:2781] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240506 16:15:48.455776 14290 raft_consensus.cc:482] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 16:15:48.455895 14290 raft_consensus.cc:3037] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 0 FOLLOWER]: Advancing to term 1
I20240506 16:15:48.459487 14290 raft_consensus.cc:504] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:15:48.460816 14290 leader_election.cc:288] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [CANDIDATE]: Term 1 election: Requested vote from peers 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195), e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055)
I20240506 16:15:48.460808 13163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "5beda4231be140b8959d934a86e21bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15538fc3b6084a96a18216a2ec74cc3a"
I20240506 16:15:48.460932 13558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "5beda4231be140b8959d934a86e21bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6d1cddba8c94532b40c0ae9c06740ef"
I20240506 16:15:48.461040 13163 raft_consensus.cc:3037] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 0 FOLLOWER]: Advancing to term 1
I20240506 16:15:48.461148 13558 raft_consensus.cc:3037] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 0 FOLLOWER]: Advancing to term 1
W20240506 16:15:48.461369 14259 tablet.cc:2369] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 16:15:48.464269 13558 raft_consensus.cc:2454] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beda4231be140b8959d934a86e21bb1 in term 1.
I20240506 16:15:48.464767 14148 leader_election.cc:302] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beda4231be140b8959d934a86e21bb1, e6d1cddba8c94532b40c0ae9c06740ef; no voters:
I20240506 16:15:48.464780 13163 raft_consensus.cc:2454] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beda4231be140b8959d934a86e21bb1 in term 1.
I20240506 16:15:48.465179 14290 raft_consensus.cc:2781] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 1 FOLLOWER]: Leader election won for term 1
I20240506 16:15:48.465720 14290 raft_consensus.cc:686] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 1 LEADER]: Becoming Leader. State: Replica: 5beda4231be140b8959d934a86e21bb1, State: Running, Role: LEADER
I20240506 16:15:48.466367 14290 consensus_queue.cc:237] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:15:48.472167 13021 catalog_manager.cc:5484] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5beda4231be140b8959d934a86e21bb1 (127.12.173.137). New cstate: current_term: 1 leader_uuid: "5beda4231be140b8959d934a86e21bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } health_report { overall_health: HEALTHY } } }
I20240506 16:15:48.487396 12982 maintenance_mode-itest.cc:745] Restarting batch of 3 tservers: c0471a9ec7ba4ea38e8df76c78d8b38f,fd8906dd0f884e22911dce7df910b6d4,18c7d19aee15430b8c3a0ef90b23fee5
W20240506 16:15:48.536885 13604 tablet.cc:2369] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240506 16:15:48.556586 13209 tablet.cc:2369] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240506 16:15:48.589713 14290 consensus_queue.cc:1035] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 16:15:48.596060 14290 consensus_queue.cc:1035] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240506 16:15:48.793882 14127 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
I20240506 16:15:49.582455 13020 ts_manager.cc:292] Set tserver state for c0471a9ec7ba4ea38e8df76c78d8b38f to MAINTENANCE_MODE
I20240506 16:15:50.075274 13020 ts_manager.cc:292] Set tserver state for 18c7d19aee15430b8c3a0ef90b23fee5 to MAINTENANCE_MODE
I20240506 16:15:50.096333 13020 ts_manager.cc:292] Set tserver state for fd8906dd0f884e22911dce7df910b6d4 to MAINTENANCE_MODE
I20240506 16:15:52.277909 13800 tablet_service.cc:1422] Tablet server c0471a9ec7ba4ea38e8df76c78d8b38f set to quiescing
I20240506 16:15:52.278211 13800 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:15:52.508246 13931 tablet_service.cc:1422] Tablet server 18c7d19aee15430b8c3a0ef90b23fee5 set to quiescing
I20240506 16:15:52.508549 13931 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:15:53.218899 13669 tablet_service.cc:1422] Tablet server fd8906dd0f884e22911dce7df910b6d4 set to quiescing
I20240506 16:15:53.219223 13669 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:15:54.809387 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 13737
I20240506 16:15:54.824587 13860 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:15:54.986296 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 13737
W20240506 16:15:55.022981 13005 connection.cc:535] server connection from 127.12.173.134:35289 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 16:15:55.024466 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.134:36633
--local_ip_for_outbound_sockets=127.12.173.134
--tserver_master_addrs=127.12.173.190:43641
--webserver_port=39891
--webserver_interface=127.12.173.134
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20240506 16:15:55.599931 14425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:15:55.600495 14425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:15:55.600745 14425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:15:55.612953 14425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:15:55.613422 14425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.134
I20240506 16:15:55.628885 14425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.134:36633
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data/info.pb
--webserver_interface=127.12.173.134
--webserver_port=39891
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.134
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:15:55.631075 14425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:15:55.633711 14425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:15:55.663672 14432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:55.666350 14434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:55.669543 14431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:15:55.679837 14425 server_base.cc:1034] running on GCE node
I20240506 16:15:55.683470 14425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:15:55.691561 14425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:15:55.707698 14425 hybrid_clock.cc:648] HybridClock initialized: now 1715012155707017 us; error 649 us; skew 500 ppm
I20240506 16:15:55.708259 14425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:15:55.726217 14425 webserver.cc:460] Webserver started at http://127.12.173.134:39891/ using document root <none> and password file <none>
I20240506 16:15:55.727509 14425 fs_manager.cc:359] Metadata directory not provided
I20240506 16:15:55.727754 14425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:15:55.742251 14425 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.011s sys 0.000s
I20240506 16:15:55.753176 14440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:55.755569 14425 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.005s
I20240506 16:15:55.755915 14425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/wal
uuid: "c0471a9ec7ba4ea38e8df76c78d8b38f"
format_stamp: "Formatted at 2024-05-06 16:15:46 on dist-test-slave-g936"
I20240506 16:15:55.757449 14425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:15:55.815415 14425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:15:55.816730 14425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:15:55.817257 14425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:15:55.890686 14425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:15:55.903028 14425 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:15:55.903390 14425 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:55.903648 14425 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:15:55.903801 14425 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:56.837505 14425 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.134:36633
I20240506 16:15:56.837580 14557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.134:36633 every 8 connection(s)
I20240506 16:15:56.839861 14425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data/info.pb
I20240506 16:15:56.843266 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 14425
I20240506 16:15:56.843537 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 13606
I20240506 16:15:56.899798 13729 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:15:57.004226 14558 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:15:57.004623 14558 heartbeater.cc:464] Registering TS with master...
I20240506 16:15:57.005738 14558 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:15:57.007880 13020 ts_manager.cc:194] Re-registered known tserver with Master: c0471a9ec7ba4ea38e8df76c78d8b38f (127.12.173.134:36633)
I20240506 16:15:57.009145 13020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.134:35363
I20240506 16:15:57.093667 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 13606
I20240506 16:15:57.135804 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.133:42687
--local_ip_for_outbound_sockets=127.12.173.133
--tserver_master_addrs=127.12.173.190:43641
--webserver_port=33855
--webserver_interface=127.12.173.133
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20240506 16:15:57.877610 14565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:15:57.878139 14565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:15:57.878336 14565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:15:57.891163 14565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:15:57.891518 14565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.133
I20240506 16:15:57.907308 14565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.133:42687
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data/info.pb
--webserver_interface=127.12.173.133
--webserver_port=33855
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.133
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:15:57.909363 14565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:15:57.911908 14565 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:15:57.943869 14571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:15:57.963852 14572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:15:57.987982 14565 server_base.cc:1034] running on GCE node
W20240506 16:15:57.996753 14574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:15:57.997989 14565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:15:58.000375 14565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:15:58.005481 14565 hybrid_clock.cc:648] HybridClock initialized: now 1715012158005377 us; error 53 us; skew 500 ppm
I20240506 16:15:58.006026 14565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:15:58.012826 14558 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
I20240506 16:15:58.023953 14565 webserver.cc:460] Webserver started at http://127.12.173.133:33855/ using document root <none> and password file <none>
I20240506 16:15:58.025348 14565 fs_manager.cc:359] Metadata directory not provided
I20240506 16:15:58.025728 14565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:15:58.050264 14565 fs_manager.cc:709] Time spent opening directory manager: real 0.022s user 0.008s sys 0.000s
I20240506 16:15:58.090816 14581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:58.094535 14565 fs_manager.cc:725] Time spent opening block manager: real 0.041s user 0.005s sys 0.006s
I20240506 16:15:58.094897 14565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/wal
uuid: "fd8906dd0f884e22911dce7df910b6d4"
format_stamp: "Formatted at 2024-05-06 16:15:46 on dist-test-slave-g936"
I20240506 16:15:58.096201 14565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:15:58.236259 14565 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:15:58.239966 14565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:15:58.240720 14565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:15:58.261487 14565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:15:58.264014 14565 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:15:58.264153 14565 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:58.264325 14565 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:15:58.264422 14565 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 16:15:59.079080 14696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.133:42687 every 8 connection(s)
I20240506 16:15:59.079025 14565 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.133:42687
I20240506 16:15:59.081612 14565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data/info.pb
I20240506 16:15:59.089658 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 14565
I20240506 16:15:59.089919 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 13868
I20240506 16:15:59.147039 13991 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:15:59.183533 14697 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:15:59.183894 14697 heartbeater.cc:464] Registering TS with master...
I20240506 16:15:59.184937 14697 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:15:59.194142 13020 ts_manager.cc:194] Re-registered known tserver with Master: fd8906dd0f884e22911dce7df910b6d4 (127.12.173.133:42687)
I20240506 16:15:59.195366 13020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.133:47947
I20240506 16:15:59.278702 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 13868
I20240506 16:15:59.351743 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.135:43823
--local_ip_for_outbound_sockets=127.12.173.135
--tserver_master_addrs=127.12.173.190:43641
--webserver_port=45737
--webserver_interface=127.12.173.135
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20240506 16:15:59.947639 14704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:15:59.948246 14704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:15:59.948459 14704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:15:59.962081 14704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:15:59.962483 14704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.135
I20240506 16:15:59.978480 14704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.135:43823
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data/info.pb
--webserver_interface=127.12.173.135
--webserver_port=45737
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.135
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:15:59.981479 14704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:15:59.984309 14704 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:16:00.042474 14712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:16:00.090409 14713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:16:00.124817 14715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:16:00.126287 14704 server_base.cc:1034] running on GCE node
I20240506 16:16:00.127389 14704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:16:00.137687 14704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:16:00.145459 14704 hybrid_clock.cc:648] HybridClock initialized: now 1715012160145344 us; error 84 us; skew 500 ppm
I20240506 16:16:00.146291 14704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:16:00.199331 14697 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
I20240506 16:16:00.203228 14704 webserver.cc:460] Webserver started at http://127.12.173.135:45737/ using document root <none> and password file <none>
I20240506 16:16:00.205581 14704 fs_manager.cc:359] Metadata directory not provided
I20240506 16:16:00.206881 14704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:16:00.263680 14704 fs_manager.cc:709] Time spent opening directory manager: real 0.031s user 0.001s sys 0.003s
I20240506 16:16:00.268879 14722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:16:00.271207 14704 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20240506 16:16:00.276281 14704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/wal
uuid: "18c7d19aee15430b8c3a0ef90b23fee5"
format_stamp: "Formatted at 2024-05-06 16:15:47 on dist-test-slave-g936"
I20240506 16:16:00.277801 14704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:16:00.399801 14704 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:16:00.406579 14704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:16:00.407014 14704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:16:00.410743 14704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:16:00.421085 14704 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:16:00.421259 14704 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20240506 16:16:00.421384 14704 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:16:00.421469 14704 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 16:16:01.339658 14704 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.135:43823
I20240506 16:16:01.341768 14704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data/info.pb
I20240506 16:16:01.344218 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 14704
I20240506 16:16:01.339623 14836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.135:43823 every 8 connection(s)
I20240506 16:16:01.454301 14837 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:16:01.454694 14837 heartbeater.cc:464] Registering TS with master...
I20240506 16:16:01.455796 14837 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:01.457903 13020 ts_manager.cc:194] Re-registered known tserver with Master: 18c7d19aee15430b8c3a0ef90b23fee5 (127.12.173.135:43823)
I20240506 16:16:01.459081 13020 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.135:50231
I20240506 16:16:02.476487 14837 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
I20240506 16:16:02.492910 13143 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:02.510725 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:02.554836 14193 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:16:02.620079 13407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:02.627735 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:02.660658 13275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:02.661857 13538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:02.679975 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:02.697768 14062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:04.323468 13014 ts_manager.cc:281] Unset tserver state for c0471a9ec7ba4ea38e8df76c78d8b38f from MAINTENANCE_MODE
I20240506 16:16:04.416672 13014 ts_manager.cc:281] Unset tserver state for fd8906dd0f884e22911dce7df910b6d4 from MAINTENANCE_MODE
I20240506 16:16:04.470692 13472 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:04.482522 14837 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:04.530900 13014 ts_manager.cc:281] Unset tserver state for 18c7d19aee15430b8c3a0ef90b23fee5 from MAINTENANCE_MODE
I20240506 16:16:04.685125 13208 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:04.700649 14258 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:04.707878 13603 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:04.835821 14127 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:04.947154 13340 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:05.037108 14558 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:05.226153 14697 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:05.475754 13472 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:05.487349 14837 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:06.623306 12982 maintenance_mode-itest.cc:745] Restarting batch of 3 tservers: cea103e071de48bbbad5e030eca24b4a,15538fc3b6084a96a18216a2ec74cc3a,405f2b51c2244be3b1a66f2c908bcdd9
I20240506 16:16:07.905728 13014 ts_manager.cc:292] Set tserver state for 15538fc3b6084a96a18216a2ec74cc3a to MAINTENANCE_MODE
I20240506 16:16:07.912963 13013 ts_manager.cc:292] Set tserver state for cea103e071de48bbbad5e030eca24b4a to MAINTENANCE_MODE
I20240506 16:16:08.196381 13013 ts_manager.cc:292] Set tserver state for 405f2b51c2244be3b1a66f2c908bcdd9 to MAINTENANCE_MODE
I20240506 16:16:10.921639 13275 tablet_service.cc:1422] Tablet server cea103e071de48bbbad5e030eca24b4a set to quiescing
I20240506 16:16:10.921947 13275 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:11.125648 13143 tablet_service.cc:1422] Tablet server 15538fc3b6084a96a18216a2ec74cc3a set to quiescing
I20240506 16:16:11.125998 13143 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:12.770021 13407 tablet_service.cc:1422] Tablet server 405f2b51c2244be3b1a66f2c908bcdd9 set to quiescing
I20240506 16:16:12.770321 13407 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:14.247730 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 13212
I20240506 16:16:14.283551 13335 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:16:14.525295 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 13212
I20240506 16:16:14.561483 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.130:42665
--local_ip_for_outbound_sockets=127.12.173.130
--tserver_master_addrs=127.12.173.190:43641
--webserver_port=34765
--webserver_interface=127.12.173.130
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20240506 16:16:15.255262 15054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:16:15.255786 15054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:16:15.256006 15054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:16:15.267904 15054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:16:15.268188 15054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.130
I20240506 16:16:15.282793 15054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.130:42665
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.173.130
--webserver_port=34765
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.130
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:16:15.284909 15054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:16:15.287756 15054 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:16:15.366549 15059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:16:15.370725 15060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:16:15.415447 15062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:16:15.417804 15054 server_base.cc:1034] running on GCE node
I20240506 16:16:15.419018 15054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:16:15.461174 15054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:16:15.462988 15054 hybrid_clock.cc:648] HybridClock initialized: now 1715012175462820 us; error 140 us; skew 500 ppm
I20240506 16:16:15.463557 15054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:16:15.474872 15054 webserver.cc:460] Webserver started at http://127.12.173.130:34765/ using document root <none> and password file <none>
I20240506 16:16:15.476169 15054 fs_manager.cc:359] Metadata directory not provided
I20240506 16:16:15.476460 15054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:16:15.498090 15054 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.005s sys 0.013s
I20240506 16:16:15.506016 15069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:16:15.508476 15054 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.002s
I20240506 16:16:15.508692 15054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/wal
uuid: "cea103e071de48bbbad5e030eca24b4a"
format_stamp: "Formatted at 2024-05-06 16:15:44 on dist-test-slave-g936"
I20240506 16:16:15.509831 15054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:16:15.562259 15054 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:16:15.563212 15054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:16:15.563488 15054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:16:15.565770 15054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:16:15.581066 15054 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:16:15.581239 15054 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240506 16:16:15.581378 15054 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:16:15.581460 15054 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 16:16:16.525141 15054 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.130:42665
I20240506 16:16:16.527608 15054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data/info.pb
I20240506 16:16:16.528509 15183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.130:42665 every 8 connection(s)
I20240506 16:16:16.537734 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 15054
I20240506 16:16:16.537986 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 13081
I20240506 16:16:16.572552 13203 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:16:16.599119 15184 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:16:16.599498 15184 heartbeater.cc:464] Registering TS with master...
I20240506 16:16:16.600595 15184 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:16.604300 13012 ts_manager.cc:194] Re-registered known tserver with Master: cea103e071de48bbbad5e030eca24b4a (127.12.173.130:42665)
I20240506 16:16:16.605587 13012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.130:60165
I20240506 16:16:16.923709 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 13081
W20240506 16:16:17.029881 14147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.173.129:43195 (error 108)
I20240506 16:16:17.031528 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.129:43195
--local_ip_for_outbound_sockets=127.12.173.129
--tserver_master_addrs=127.12.173.190:43641
--webserver_port=39161
--webserver_interface=127.12.173.129
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20240506 16:16:17.038406 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240506 16:16:17.478657 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240506 16:16:17.609818 15184 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
W20240506 16:16:17.627843 15195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:16:17.628237 15195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:16:17.628495 15195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:16:17.641424 15195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:16:17.641710 15195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.129
I20240506 16:16:17.656751 15195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.129:43195
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.173.129
--webserver_port=39161
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.129
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:16:17.658928 15195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:16:17.661600 15195 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:16:17.692404 15201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:16:17.693113 15204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:16:17.707994 15202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:16:17.715680 15195 server_base.cc:1034] running on GCE node
I20240506 16:16:17.717103 15195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:16:17.727521 15195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:16:17.731096 15195 hybrid_clock.cc:648] HybridClock initialized: now 1715012177730933 us; error 122 us; skew 500 ppm
I20240506 16:16:17.731580 15195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:16:17.737877 15195 webserver.cc:460] Webserver started at http://127.12.173.129:39161/ using document root <none> and password file <none>
I20240506 16:16:17.739667 15195 fs_manager.cc:359] Metadata directory not provided
I20240506 16:16:17.739974 15195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:16:17.776454 15195 fs_manager.cc:709] Time spent opening directory manager: real 0.032s user 0.008s sys 0.006s
I20240506 16:16:17.788208 15210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:16:17.791654 15195 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.004s sys 0.002s
I20240506 16:16:17.792001 15195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/wal
uuid: "15538fc3b6084a96a18216a2ec74cc3a"
format_stamp: "Formatted at 2024-05-06 16:15:44 on dist-test-slave-g936"
I20240506 16:16:17.793856 15195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:16:17.900104 15195 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:16:17.901536 15195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:16:17.901862 15195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:16:17.949796 15195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:16:17.965786 15220 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240506 16:16:17.973064 15195 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240506 16:16:17.973253 15195 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.014s user 0.007s sys 0.000s
I20240506 16:16:17.973421 15195 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240506 16:16:17.978550 15195 ts_tablet_manager.cc:590] Registered 1 tablets
I20240506 16:16:17.978874 15195 ts_tablet_manager.cc:569] Time spent register tablets: real 0.005s user 0.002s sys 0.002s
I20240506 16:16:17.982988 15220 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Bootstrap starting.
W20240506 16:16:18.019758 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240506 16:16:18.582479 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240506 16:16:19.028748 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240506 16:16:19.307077 15328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.129:43195 every 8 connection(s)
I20240506 16:16:19.307592 15195 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.129:43195
I20240506 16:16:19.310356 15195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data/info.pb
I20240506 16:16:19.315351 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 15195
I20240506 16:16:19.315587 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 13344
I20240506 16:16:19.372967 15329 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:16:19.373384 15329 heartbeater.cc:464] Registering TS with master...
I20240506 16:16:19.374677 15329 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:19.378059 13012 ts_manager.cc:194] Re-registered known tserver with Master: 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195)
I20240506 16:16:19.379720 13012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.129:38171
I20240506 16:16:19.438063 13467 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:16:19.456293 15220 log.cc:824] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Log is configured to *not* fsync() on all Append() calls
W20240506 16:16:19.626899 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
I20240506 16:16:19.811697 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 13344
I20240506 16:16:19.882706 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.131:33797
--local_ip_for_outbound_sockets=127.12.173.131
--tserver_master_addrs=127.12.173.190:43641
--webserver_port=38629
--webserver_interface=127.12.173.131
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20240506 16:16:20.195744 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20240506 16:16:20.384683 15329 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
W20240506 16:16:20.732973 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
W20240506 16:16:20.774430 15338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:16:20.774976 15338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:16:20.775208 15338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:16:20.796608 15338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:16:20.797008 15338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.131
I20240506 16:16:20.903936 15338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.131:33797
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.173.131
--webserver_port=38629
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.131
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:16:20.909487 15338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:16:20.935516 15338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:16:21.005265 15346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:16:21.013731 15347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:16:21.023792 15350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:16:21.032976 15338 server_base.cc:1034] running on GCE node
I20240506 16:16:21.035634 15338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:16:21.063428 15338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:16:21.071475 15338 hybrid_clock.cc:648] HybridClock initialized: now 1715012181071377 us; error 66 us; skew 500 ppm
I20240506 16:16:21.071997 15338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:16:21.121312 15338 webserver.cc:460] Webserver started at http://127.12.173.131:38629/ using document root <none> and password file <none>
I20240506 16:16:21.122627 15338 fs_manager.cc:359] Metadata directory not provided
I20240506 16:16:21.122889 15338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:16:21.144649 15338 fs_manager.cc:709] Time spent opening directory manager: real 0.019s user 0.004s sys 0.006s
I20240506 16:16:21.156368 15355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:16:21.158900 15338 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.007s
I20240506 16:16:21.159255 15338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/wal
uuid: "405f2b51c2244be3b1a66f2c908bcdd9"
format_stamp: "Formatted at 2024-05-06 16:15:45 on dist-test-slave-g936"
I20240506 16:16:21.161149 15338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:16:21.215620 15338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:16:21.216848 15338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:16:21.217268 15338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:16:21.224858 15338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:16:21.233654 15338 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:16:21.233886 15338 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20240506 16:16:21.234069 15338 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:16:21.234192 15338 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20240506 16:16:21.475908 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
I20240506 16:16:21.530875 15212 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Peer 15538fc3b6084a96a18216a2ec74cc3a is lagging by at least 2 ops behind the committed index
W20240506 16:16:22.061328 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
I20240506 16:16:22.299947 15338 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.131:33797
I20240506 16:16:22.302750 15338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data/info.pb
I20240506 16:16:22.303664 15468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.131:33797 every 8 connection(s)
I20240506 16:16:22.309721 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 15338
I20240506 16:16:22.443691 15469 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:16:22.444058 15469 heartbeater.cc:464] Registering TS with master...
I20240506 16:16:22.445132 15469 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:22.448176 13012 ts_manager.cc:194] Re-registered known tserver with Master: 405f2b51c2244be3b1a66f2c908bcdd9 (127.12.173.131:33797)
I20240506 16:16:22.449311 13012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.131:55349
W20240506 16:16:22.726338 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
W20240506 16:16:23.399313 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
I20240506 16:16:23.453087 15469 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
I20240506 16:16:23.967631 15262 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:24.029021 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:24.056427 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:24.061952 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:24.099831 14193 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
W20240506 16:16:24.145795 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
I20240506 16:16:24.162765 14062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:24.201164 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:24.214100 13538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:24.274067 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:24.470705 15220 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Bootstrap replaying log segment 1/1 (1.12M/3.29M this segment, stats: ops{read=664 overwritten=0 applied=663 ignored=0} inserts{seen=33100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20240506 16:16:24.544338 15341 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Peer 15538fc3b6084a96a18216a2ec74cc3a is lagging by at least 225 ops behind the committed index [suppressed 22 similar messages]
W20240506 16:16:24.822311 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
W20240506 16:16:25.355957 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
W20240506 16:16:25.879424 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
W20240506 16:16:26.434002 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20240506 16:16:26.977686 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
W20240506 16:16:27.497759 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
I20240506 16:16:27.551980 15523 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Peer 15538fc3b6084a96a18216a2ec74cc3a is lagging by at least 543 ops behind the committed index [suppressed 26 similar messages]
W20240506 16:16:28.029487 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
W20240506 16:16:28.562512 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
W20240506 16:16:29.072319 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
I20240506 16:16:29.472560 15220 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Bootstrap replaying log segment 1/1 (2.60M/3.29M this segment, stats: ops{read=1543 overwritten=0 applied=1543 ignored=0} inserts{seen=77100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:16:29.534632 14267 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.12.173.137:37831, user_credentials={real_user=slave}} blocked reactor thread for 44995.3us
W20240506 16:16:29.604223 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
W20240506 16:16:30.067548 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
W20240506 16:16:30.537307 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
I20240506 16:16:30.578927 15524 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Peer 15538fc3b6084a96a18216a2ec74cc3a is lagging by at least 829 ops behind the committed index [suppressed 29 similar messages]
W20240506 16:16:31.071799 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
I20240506 16:16:31.341351 15262 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:31.348827 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:31.372931 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:31.429445 14193 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:16:31.454057 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:31.470028 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:31.489799 13538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:31.522284 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:31.561141 14062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 16:16:31.759848 14147 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
I20240506 16:16:31.922909 15220 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1947 overwritten=0 applied=1946 ignored=0} inserts{seen=97250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240506 16:16:31.923853 15220 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Bootstrap complete.
I20240506 16:16:31.926899 15220 ts_tablet_manager.cc:1367] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Time spent bootstrapping tablet: real 13.944s user 9.083s sys 0.567s
I20240506 16:16:31.943320 15220 raft_consensus.cc:348] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:16:31.970721 15220 raft_consensus.cc:729] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15538fc3b6084a96a18216a2ec74cc3a, State: Initialized, Role: FOLLOWER
I20240506 16:16:31.971748 15220 consensus_queue.cc:260] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1946, Last appended: 1.1947, Last appended by leader: 1947, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:16:31.974915 15220 ts_tablet_manager.cc:1398] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Time spent starting tablet: real 0.048s user 0.015s sys 0.016s
I20240506 16:16:32.304955 15582 raft_consensus.cc:482] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5beda4231be140b8959d934a86e21bb1)
I20240506 16:16:32.305402 15582 raft_consensus.cc:504] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:16:32.307683 15582 leader_election.cc:288] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055), 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831)
I20240506 16:16:32.335814 13558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "15538fc3b6084a96a18216a2ec74cc3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2372 } } ignore_live_leader: false dest_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" is_pre_election: true
I20240506 16:16:32.356402 14213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "15538fc3b6084a96a18216a2ec74cc3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2372 } } ignore_live_leader: false dest_uuid: "5beda4231be140b8959d934a86e21bb1" is_pre_election: true
I20240506 16:16:32.363075 15215 leader_election.cc:302] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15538fc3b6084a96a18216a2ec74cc3a; no voters: 5beda4231be140b8959d934a86e21bb1, e6d1cddba8c94532b40c0ae9c06740ef
I20240506 16:16:32.610935 15582 raft_consensus.cc:2726] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 16:16:38.299121 15262 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:38.336055 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:38.491472 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:38.577952 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:38.629258 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:38.642927 14193 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:16:38.645114 13538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:38.693003 14062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:38.707691 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:40.175541 13012 ts_manager.cc:281] Unset tserver state for cea103e071de48bbbad5e030eca24b4a from MAINTENANCE_MODE
I20240506 16:16:40.177369 13603 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:40.177743 14558 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:40.249917 15329 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:40.308254 14258 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:40.338835 14697 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:40.497354 15469 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:40.583403 14837 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:40.815258 13022 ts_manager.cc:281] Unset tserver state for 405f2b51c2244be3b1a66f2c908bcdd9 from MAINTENANCE_MODE
I20240506 16:16:40.816603 15184 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:40.942538 13013 ts_manager.cc:281] Unset tserver state for 15538fc3b6084a96a18216a2ec74cc3a from MAINTENANCE_MODE
I20240506 16:16:40.999929 14127 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:41.184504 13603 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:41.187215 14558 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:41.255586 15329 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:41.315117 14258 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:41.344012 14697 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:41.504580 15469 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:41.592708 14837 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:41.852867 15184 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:44.193073 12982 maintenance_mode-itest.cc:745] Restarting batch of 1 tservers: e6d1cddba8c94532b40c0ae9c06740ef
I20240506 16:16:45.435739 13013 ts_manager.cc:292] Set tserver state for e6d1cddba8c94532b40c0ae9c06740ef to MAINTENANCE_MODE
I20240506 16:16:46.550688 13538 tablet_service.cc:1422] Tablet server e6d1cddba8c94532b40c0ae9c06740ef set to quiescing
I20240506 16:16:46.550971 13538 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:47.963083 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 13475
I20240506 16:16:47.992533 13597 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:16:48.505060 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 13475
W20240506 16:16:48.622112 14148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.173.132:39055 (error 108) [suppressed 22 similar messages]
I20240506 16:16:48.629886 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.132:39055
--local_ip_for_outbound_sockets=127.12.173.132
--tserver_master_addrs=127.12.173.190:43641
--webserver_port=44537
--webserver_interface=127.12.173.132
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20240506 16:16:48.643869 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.132:39055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240506 16:16:49.182036 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.132:39055: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240506 16:16:49.248656 15764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:16:49.249022 15764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:16:49.249250 15764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:16:49.260010 15764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:16:49.260272 15764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.132
I20240506 16:16:49.279469 15764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.132:39055
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.12.173.132
--webserver_port=44537
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.132
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:16:49.281491 15764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:16:49.283931 15764 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:16:49.321122 15771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:16:49.333125 15772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:16:49.352648 15764 server_base.cc:1034] running on GCE node
W20240506 16:16:49.356287 15775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:16:49.358414 15764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:16:49.373524 15764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:16:49.379072 15764 hybrid_clock.cc:648] HybridClock initialized: now 1715012209378850 us; error 178 us; skew 500 ppm
I20240506 16:16:49.379534 15764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:16:49.388468 15764 webserver.cc:460] Webserver started at http://127.12.173.132:44537/ using document root <none> and password file <none>
I20240506 16:16:49.389689 15764 fs_manager.cc:359] Metadata directory not provided
I20240506 16:16:49.389938 15764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:16:49.407543 15764 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.005s sys 0.000s
I20240506 16:16:49.436828 15780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:16:49.444113 15764 fs_manager.cc:725] Time spent opening block manager: real 0.033s user 0.006s sys 0.000s
I20240506 16:16:49.444415 15764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/wal
uuid: "e6d1cddba8c94532b40c0ae9c06740ef"
format_stamp: "Formatted at 2024-05-06 16:15:45 on dist-test-slave-g936"
I20240506 16:16:49.445663 15764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:16:49.496631 15764 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:16:49.498947 15764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:16:49.499351 15764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:16:49.509910 15764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:16:49.519174 15787 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240506 16:16:49.533819 15764 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240506 16:16:49.534052 15764 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20240506 16:16:49.534261 15764 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240506 16:16:49.539253 15764 ts_tablet_manager.cc:590] Registered 1 tablets
I20240506 16:16:49.539988 15764 ts_tablet_manager.cc:569] Time spent register tablets: real 0.006s user 0.003s sys 0.001s
I20240506 16:16:49.539883 15787 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Bootstrap starting.
W20240506 16:16:49.721458 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.132:39055: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240506 16:16:50.279942 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.132:39055: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240506 16:16:50.536214 15895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.132:39055 every 8 connection(s)
I20240506 16:16:50.536587 15764 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.132:39055
I20240506 16:16:50.538712 15764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data/info.pb
I20240506 16:16:50.539865 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 15764
I20240506 16:16:50.596084 15896 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:16:50.596413 15896 heartbeater.cc:464] Registering TS with master...
I20240506 16:16:50.597666 15896 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:16:50.605693 13013 ts_manager.cc:194] Re-registered known tserver with Master: e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055)
I20240506 16:16:50.607261 13013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.132:35765
W20240506 16:16:50.894781 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
W20240506 16:16:51.388955 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
I20240506 16:16:51.612013 15896 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
W20240506 16:16:52.021112 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20240506 16:16:52.184746 15262 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:52.253383 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:52.318240 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:52.393052 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:52.404836 14062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:52.409348 15830 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:52.413655 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:52.426028 14193 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:16:52.430315 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:52.697243 15923 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Peer e6d1cddba8c94532b40c0ae9c06740ef is lagging by at least 2 ops behind the committed index
W20240506 16:16:52.764005 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20240506 16:16:53.073876 15787 log.cc:824] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Log is configured to *not* fsync() on all Append() calls
W20240506 16:16:53.205781 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20240506 16:16:53.772997 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20240506 16:16:54.342940 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
W20240506 16:16:54.882862 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
W20240506 16:16:55.381363 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
I20240506 16:16:55.748525 15903 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Peer e6d1cddba8c94532b40c0ae9c06740ef is lagging by at least 326 ops behind the committed index [suppressed 28 similar messages]
W20240506 16:16:55.911058 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
W20240506 16:16:56.427872 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
W20240506 16:16:56.953892 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
W20240506 16:16:57.472541 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20240506 16:16:58.028273 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
I20240506 16:16:58.078831 15787 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Bootstrap replaying log segment 1/1 (1.84M/7.71M this segment, stats: ops{read=1083 overwritten=0 applied=1083 ignored=0} inserts{seen=54100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:16:58.617915 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
I20240506 16:16:58.772675 15765 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Peer e6d1cddba8c94532b40c0ae9c06740ef is lagging by at least 630 ops behind the committed index [suppressed 27 similar messages]
W20240506 16:16:59.168336 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
I20240506 16:16:59.680714 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:59.688665 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 16:16:59.715219 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
I20240506 16:16:59.715759 15262 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:59.735101 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:59.752107 14193 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:16:59.783133 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:59.841349 14062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:59.906244 15830 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:16:59.942499 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 16:17:00.370622 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20240506 16:17:00.774859 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
W20240506 16:17:01.243008 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
W20240506 16:17:01.762446 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
I20240506 16:17:01.845655 15962 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Peer e6d1cddba8c94532b40c0ae9c06740ef is lagging by at least 914 ops behind the committed index [suppressed 28 similar messages]
W20240506 16:17:02.279945 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
W20240506 16:17:02.877578 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
I20240506 16:17:03.078819 15787 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Bootstrap replaying log segment 1/1 (3.53M/7.71M this segment, stats: ops{read=2085 overwritten=0 applied=2084 ignored=0} inserts{seen=104150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:17:03.333307 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
W20240506 16:17:03.877454 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
W20240506 16:17:04.421442 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
I20240506 16:17:04.873459 15749 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Peer e6d1cddba8c94532b40c0ae9c06740ef is lagging by at least 1221 ops behind the committed index [suppressed 28 similar messages]
W20240506 16:17:04.929181 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
W20240506 16:17:05.151412 14267 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.12.173.137:37831, user_credentials={real_user=slave}} blocked reactor thread for 52783.3us
W20240506 16:17:05.400624 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 156: this message will repeat every 5th retry.
W20240506 16:17:06.056962 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 161: this message will repeat every 5th retry.
W20240506 16:17:06.602543 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 166: this message will repeat every 5th retry.
W20240506 16:17:07.107707 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 171: this message will repeat every 5th retry.
I20240506 16:17:07.186085 15262 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:07.213095 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:07.289830 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:07.309655 14193 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:17:07.356804 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:07.370512 14062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:07.378139 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:07.417346 15830 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:07.458850 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 16:17:07.665469 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 176: this message will repeat every 5th retry.
I20240506 16:17:07.964989 15765 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Peer e6d1cddba8c94532b40c0ae9c06740ef is lagging by at least 1474 ops behind the committed index [suppressed 27 similar messages]
I20240506 16:17:08.087131 15787 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Bootstrap replaying log segment 1/1 (5.08M/7.71M this segment, stats: ops{read=2996 overwritten=0 applied=2992 ignored=0} inserts{seen=149550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:17:08.227358 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 181: this message will repeat every 5th retry.
W20240506 16:17:08.805662 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 186: this message will repeat every 5th retry.
W20240506 16:17:09.253729 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 191: this message will repeat every 5th retry.
W20240506 16:17:09.758574 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 196: this message will repeat every 5th retry.
W20240506 16:17:10.333391 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 201: this message will repeat every 5th retry.
W20240506 16:17:10.831537 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 206: this message will repeat every 5th retry.
I20240506 16:17:10.990176 16044 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Peer e6d1cddba8c94532b40c0ae9c06740ef is lagging by at least 1761 ops behind the committed index [suppressed 28 similar messages]
W20240506 16:17:11.347190 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 211: this message will repeat every 5th retry.
W20240506 16:17:11.807137 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 216: this message will repeat every 5th retry.
W20240506 16:17:12.344355 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 221: this message will repeat every 5th retry.
W20240506 16:17:12.917817 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 226: this message will repeat every 5th retry.
I20240506 16:17:13.089181 15787 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Bootstrap replaying log segment 1/1 (6.81M/7.71M this segment, stats: ops{read=4012 overwritten=0 applied=4012 ignored=0} inserts{seen=200550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:17:13.456864 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 231: this message will repeat every 5th retry.
I20240506 16:17:14.083050 16085 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Peer e6d1cddba8c94532b40c0ae9c06740ef is lagging by at least 2032 ops behind the committed index [suppressed 27 similar messages]
W20240506 16:17:14.129117 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 236: this message will repeat every 5th retry.
I20240506 16:17:14.161903 15262 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:14.191392 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:14.201279 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:14.257110 14193 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:17:14.274111 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:14.379344 14062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:14.379802 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:14.387609 15830 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:14.408938 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 16:17:14.917940 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 241: this message will repeat every 5th retry.
W20240506 16:17:15.450688 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 246: this message will repeat every 5th retry.
W20240506 16:17:16.000662 14148 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 251: this message will repeat every 5th retry.
I20240506 16:17:16.126483 15787 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=4545 overwritten=0 applied=4543 ignored=0} inserts{seen=227100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20240506 16:17:16.127374 15787 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Bootstrap complete.
I20240506 16:17:16.130115 15787 ts_tablet_manager.cc:1367] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Time spent bootstrapping tablet: real 26.591s user 19.671s sys 0.949s
I20240506 16:17:16.136788 15787 raft_consensus.cc:348] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 1 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:17:16.148360 15787 raft_consensus.cc:729] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6d1cddba8c94532b40c0ae9c06740ef, State: Initialized, Role: FOLLOWER
I20240506 16:17:16.149827 15787 consensus_queue.cc:260] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4543, Last appended: 1.4545, Last appended by leader: 4545, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:17:16.174185 15787 ts_tablet_manager.cc:1398] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef: Time spent starting tablet: real 0.044s user 0.028s sys 0.012s
I20240506 16:17:16.266794 15849 raft_consensus.cc:1203] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4544->[1.4545-1.4969] Dedup: 1.4545->[1.4546-1.4969]
I20240506 16:17:16.723513 16148 raft_consensus.cc:482] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5beda4231be140b8959d934a86e21bb1)
I20240506 16:17:16.723809 16148 raft_consensus.cc:504] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:17:16.726055 16148 leader_election.cc:288] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195), 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831)
I20240506 16:17:16.776890 14213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4969 } } ignore_live_leader: false dest_uuid: "5beda4231be140b8959d934a86e21bb1" is_pre_election: true
I20240506 16:17:16.812114 15280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 4969 } } ignore_live_leader: false dest_uuid: "15538fc3b6084a96a18216a2ec74cc3a" is_pre_election: true
I20240506 16:17:16.813400 15782 leader_election.cc:302] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6d1cddba8c94532b40c0ae9c06740ef; no voters: 15538fc3b6084a96a18216a2ec74cc3a, 5beda4231be140b8959d934a86e21bb1
I20240506 16:17:16.814205 16148 raft_consensus.cc:2726] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 16:17:17.163990 16148 raft_consensus.cc:482] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5beda4231be140b8959d934a86e21bb1)
I20240506 16:17:17.164192 16148 raft_consensus.cc:504] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:17:17.179682 16148 leader_election.cc:288] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195), 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831)
I20240506 16:17:17.180145 15280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 5394 } } ignore_live_leader: false dest_uuid: "15538fc3b6084a96a18216a2ec74cc3a" is_pre_election: true
I20240506 16:17:17.180181 14213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 5394 } } ignore_live_leader: false dest_uuid: "5beda4231be140b8959d934a86e21bb1" is_pre_election: true
I20240506 16:17:17.180832 15782 leader_election.cc:302] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6d1cddba8c94532b40c0ae9c06740ef; no voters: 15538fc3b6084a96a18216a2ec74cc3a, 5beda4231be140b8959d934a86e21bb1
I20240506 16:17:17.181131 16148 raft_consensus.cc:2726] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240506 16:17:17.192950 16103 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Peer e6d1cddba8c94532b40c0ae9c06740ef is lagging by at least 1450 ops behind the committed index [suppressed 18 similar messages]
W20240506 16:17:20.189659 14267 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.12.173.137:37831, user_credentials={real_user=slave}} blocked reactor thread for 65016.2us
I20240506 16:17:21.706737 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:21.717156 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:21.733927 15262 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:21.768667 14193 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:17:21.817840 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:21.839354 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:21.863611 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:21.866009 15830 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:21.936139 14062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 16:17:22.313818 14147 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.173.129:43195, user_credentials={real_user=slave}} blocked reactor thread for 46293.5us
I20240506 16:17:23.524894 13013 ts_manager.cc:281] Unset tserver state for e6d1cddba8c94532b40c0ae9c06740ef from MAINTENANCE_MODE
I20240506 16:17:23.615468 15469 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:17:23.702560 14837 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:17:23.751917 15896 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:17:23.818459 14258 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:17:23.957568 15184 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:17:24.124526 14127 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:17:24.321130 14558 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:17:24.383545 15329 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:17:24.446317 14697 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:17:25.419364 12982 maintenance_mode-itest.cc:745] Restarting batch of 2 tservers: dc9fadb1e71e44c18dbade6135728983,5beda4231be140b8959d934a86e21bb1
I20240506 16:17:26.828666 13013 ts_manager.cc:292] Set tserver state for 5beda4231be140b8959d934a86e21bb1 to MAINTENANCE_MODE
I20240506 16:17:26.837337 13021 ts_manager.cc:292] Set tserver state for dc9fadb1e71e44c18dbade6135728983 to MAINTENANCE_MODE
I20240506 16:17:29.815114 14062 tablet_service.cc:1422] Tablet server dc9fadb1e71e44c18dbade6135728983 set to quiescing
I20240506 16:17:29.815485 14062 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:31.401463 14193 tablet_service.cc:1422] Tablet server 5beda4231be140b8959d934a86e21bb1 set to quiescing
I20240506 16:17:31.401922 14193 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:17:31.410284 16178 raft_consensus.cc:979] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: : Instructing follower e6d1cddba8c94532b40c0ae9c06740ef to start an election
I20240506 16:17:31.410450 16178 raft_consensus.cc:1067] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 1 LEADER]: Signalling peer e6d1cddba8c94532b40c0ae9c06740ef to start an election
I20240506 16:17:31.419955 16282 raft_consensus.cc:979] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: : Instructing follower e6d1cddba8c94532b40c0ae9c06740ef to start an election
I20240506 16:17:31.420094 16282 raft_consensus.cc:1067] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 1 LEADER]: Signalling peer e6d1cddba8c94532b40c0ae9c06740ef to start an election
I20240506 16:17:31.420292 16282 raft_consensus.cc:979] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: : Instructing follower 15538fc3b6084a96a18216a2ec74cc3a to start an election
I20240506 16:17:31.420388 16282 raft_consensus.cc:1067] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 1 LEADER]: Signalling peer 15538fc3b6084a96a18216a2ec74cc3a to start an election
I20240506 16:17:31.421360 15283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f"
dest_uuid: "15538fc3b6084a96a18216a2ec74cc3a"
from {username='slave'} at 127.12.173.137:53863
I20240506 16:17:31.421777 15283 raft_consensus.cc:482] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 16:17:31.421999 15283 raft_consensus.cc:3037] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 1 FOLLOWER]: Advancing to term 2
I20240506 16:17:31.426115 15850 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f"
dest_uuid: "e6d1cddba8c94532b40c0ae9c06740ef"
from {username='slave'} at 127.12.173.137:37929
I20240506 16:17:31.426445 15850 raft_consensus.cc:482] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 16:17:31.426584 15850 raft_consensus.cc:3037] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 1 FOLLOWER]: Advancing to term 2
I20240506 16:17:31.429912 15849 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f"
dest_uuid: "e6d1cddba8c94532b40c0ae9c06740ef"
from {username='slave'} at 127.12.173.137:37929
I20240506 16:17:31.433161 15850 raft_consensus.cc:504] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:17:31.433665 15849 raft_consensus.cc:482] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240506 16:17:31.433768 15849 raft_consensus.cc:3037] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 2 FOLLOWER]: Advancing to term 3
I20240506 16:17:31.436601 15849 raft_consensus.cc:504] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:17:31.441622 15280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 8190 } } ignore_live_leader: true dest_uuid: "15538fc3b6084a96a18216a2ec74cc3a"
I20240506 16:17:31.442409 15850 leader_election.cc:288] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [CANDIDATE]: Term 2 election: Requested vote from peers 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195), 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831)
I20240506 16:17:31.443331 14213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 8190 } } ignore_live_leader: true dest_uuid: "5beda4231be140b8959d934a86e21bb1"
I20240506 16:17:31.444348 15281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 8190 } } ignore_live_leader: true dest_uuid: "15538fc3b6084a96a18216a2ec74cc3a"
I20240506 16:17:31.445653 15849 leader_election.cc:288] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [CANDIDATE]: Term 3 election: Requested vote from peers 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195), 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831)
I20240506 16:17:31.446358 14213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" candidate_term: 3 candidate_status { last_received { term: 1 index: 8190 } } ignore_live_leader: true dest_uuid: "5beda4231be140b8959d934a86e21bb1"
I20240506 16:17:31.448642 15283 raft_consensus.cc:504] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:17:31.449182 15781 leader_election.cc:302] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6d1cddba8c94532b40c0ae9c06740ef; no voters: 15538fc3b6084a96a18216a2ec74cc3a, 5beda4231be140b8959d934a86e21bb1
I20240506 16:17:31.461860 16312 raft_consensus.cc:2726] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240506 16:17:31.462863 15280 raft_consensus.cc:2379] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e6d1cddba8c94532b40c0ae9c06740ef in current term 2: Already voted for candidate 15538fc3b6084a96a18216a2ec74cc3a in this term.
I20240506 16:17:31.463843 15782 leader_election.cc:302] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6d1cddba8c94532b40c0ae9c06740ef; no voters: 15538fc3b6084a96a18216a2ec74cc3a, 5beda4231be140b8959d934a86e21bb1
I20240506 16:17:31.474325 16312 raft_consensus.cc:2726] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240506 16:17:31.489514 15283 leader_election.cc:288] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [CANDIDATE]: Term 2 election: Requested vote from peers e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055), 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831)
I20240506 16:17:31.491380 14213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "15538fc3b6084a96a18216a2ec74cc3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8190 } } ignore_live_leader: true dest_uuid: "5beda4231be140b8959d934a86e21bb1"
I20240506 16:17:31.502846 15283 raft_consensus.cc:1226] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 2 FOLLOWER]: Rejecting Update request from peer 5beda4231be140b8959d934a86e21bb1 for earlier term 1. Current term is 2. Ops: [1.8191-1.8191]
I20240506 16:17:31.503762 16278 consensus_queue.cc:1046] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 }, Status: INVALID_TERM, Last received: 1.8190, Next index: 8191, Last known committed idx: 8190, Time since last communication: 0.000s
I20240506 16:17:31.504200 16278 raft_consensus.cc:3032] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 1 LEADER]: Stepping down as leader of term 1
I20240506 16:17:31.504284 16278 raft_consensus.cc:729] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5beda4231be140b8959d934a86e21bb1, State: Running, Role: LEADER
I20240506 16:17:31.504606 16278 consensus_queue.cc:260] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8190, Committed index: 8190, Last appended: 1.8193, Last appended by leader: 8193, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:17:31.516250 16278 logging.cc:424] LogThrottler TrackedPeer Lag: suppressed but not reported on 3 messages since previous log ~14 seconds ago
I20240506 16:17:31.516495 16278 logging.cc:424] LogThrottler TrackedPeer Lag: suppressed but not reported on 13 messages since previous log ~60 seconds ago
I20240506 16:17:31.516992 16278 raft_consensus.cc:3037] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 1 FOLLOWER]: Advancing to term 2
I20240506 16:17:31.523903 15849 raft_consensus.cc:1226] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 3 FOLLOWER]: Rejecting Update request from peer 5beda4231be140b8959d934a86e21bb1 for earlier term 1. Current term is 3. Ops: [1.8191-1.8191]
I20240506 16:17:31.545078 15849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "15538fc3b6084a96a18216a2ec74cc3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8190 } } ignore_live_leader: true dest_uuid: "e6d1cddba8c94532b40c0ae9c06740ef"
I20240506 16:17:31.545621 15849 raft_consensus.cc:2354] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 15538fc3b6084a96a18216a2ec74cc3a for earlier term 2. Current term is 3.
I20240506 16:17:31.546553 15215 leader_election.cc:398] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [CANDIDATE]: Term 2 election: Vote denied by peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055) with higher term. Message: Invalid argument: T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 15538fc3b6084a96a18216a2ec74cc3a for earlier term 2. Current term is 3.
I20240506 16:17:31.546784 15215 leader_election.cc:401] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240506 16:17:31.570351 16316 raft_consensus.cc:3037] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 2 FOLLOWER]: Advancing to term 3
I20240506 16:17:31.576879 16316 raft_consensus.cc:2726] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055) with higher term. Message: Invalid argument: T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 15538fc3b6084a96a18216a2ec74cc3a for earlier term 2. Current term is 3.
W20240506 16:17:31.765329 16278 raft_consensus.cc:659] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: failed to trigger leader election: Illegal state: leader elections are disabled
I20240506 16:17:31.912325 16316 raft_consensus.cc:482] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240506 16:17:31.912567 16316 raft_consensus.cc:504] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:17:31.913519 16316 leader_election.cc:288] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055), 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831)
I20240506 16:17:31.914379 15849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "15538fc3b6084a96a18216a2ec74cc3a" candidate_term: 4 candidate_status { last_received { term: 1 index: 8190 } } ignore_live_leader: false dest_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" is_pre_election: true
I20240506 16:17:31.914868 15849 raft_consensus.cc:2454] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15538fc3b6084a96a18216a2ec74cc3a in term 3.
I20240506 16:17:31.915413 15215 leader_election.cc:302] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15538fc3b6084a96a18216a2ec74cc3a, e6d1cddba8c94532b40c0ae9c06740ef; no voters:
I20240506 16:17:31.916196 14213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "15538fc3b6084a96a18216a2ec74cc3a" candidate_term: 4 candidate_status { last_received { term: 1 index: 8190 } } ignore_live_leader: false dest_uuid: "5beda4231be140b8959d934a86e21bb1" is_pre_election: true
I20240506 16:17:31.916483 14213 raft_consensus.cc:2396] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 15538fc3b6084a96a18216a2ec74cc3a for term 4 because replica has last-logged OpId of term: 1 index: 8193, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8190.
I20240506 16:17:31.918256 16316 raft_consensus.cc:2781] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240506 16:17:31.918392 16316 raft_consensus.cc:482] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240506 16:17:31.918493 16316 raft_consensus.cc:3037] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 3 FOLLOWER]: Advancing to term 4
I20240506 16:17:31.921466 16316 raft_consensus.cc:504] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:17:31.922789 16316 leader_election.cc:288] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [CANDIDATE]: Term 4 election: Requested vote from peers e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055), 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831)
I20240506 16:17:31.922825 15849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "15538fc3b6084a96a18216a2ec74cc3a" candidate_term: 4 candidate_status { last_received { term: 1 index: 8190 } } ignore_live_leader: false dest_uuid: "e6d1cddba8c94532b40c0ae9c06740ef"
I20240506 16:17:31.923180 15849 raft_consensus.cc:3037] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 3 FOLLOWER]: Advancing to term 4
I20240506 16:17:31.924018 14213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "15538fc3b6084a96a18216a2ec74cc3a" candidate_term: 4 candidate_status { last_received { term: 1 index: 8190 } } ignore_live_leader: false dest_uuid: "5beda4231be140b8959d934a86e21bb1"
I20240506 16:17:31.924202 14213 raft_consensus.cc:3037] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 2 FOLLOWER]: Advancing to term 4
I20240506 16:17:31.926568 14213 raft_consensus.cc:2396] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 15538fc3b6084a96a18216a2ec74cc3a for term 4 because replica has last-logged OpId of term: 1 index: 8193, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 8190.
I20240506 16:17:31.928838 15849 raft_consensus.cc:2454] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15538fc3b6084a96a18216a2ec74cc3a in term 4.
I20240506 16:17:31.929863 15215 leader_election.cc:302] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15538fc3b6084a96a18216a2ec74cc3a, e6d1cddba8c94532b40c0ae9c06740ef; no voters: 5beda4231be140b8959d934a86e21bb1
I20240506 16:17:31.930392 16316 raft_consensus.cc:2781] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 4 FOLLOWER]: Leader election won for term 4
I20240506 16:17:31.930907 16316 raft_consensus.cc:686] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [term 4 LEADER]: Becoming Leader. State: Replica: 15538fc3b6084a96a18216a2ec74cc3a, State: Running, Role: LEADER
I20240506 16:17:31.931458 16316 consensus_queue.cc:237] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8190, Committed index: 8190, Last appended: 1.8190, Last appended by leader: 8190, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:17:31.936458 13021 catalog_manager.cc:5484] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a reported cstate change: term changed from 1 to 4, leader changed from 5beda4231be140b8959d934a86e21bb1 (127.12.173.137) to 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129). New cstate: current_term: 4 leader_uuid: "15538fc3b6084a96a18216a2ec74cc3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } health_report { overall_health: UNKNOWN } } }
I20240506 16:17:32.031199 15849 raft_consensus.cc:1261] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 4 FOLLOWER]: Refusing update from remote peer 15538fc3b6084a96a18216a2ec74cc3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8190. Preceding OpId from leader: term: 4 index: 8191. (index mismatch)
I20240506 16:17:32.032150 16316 consensus_queue.cc:1035] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8191, Last known committed idx: 8190, Time since last communication: 0.000s
I20240506 16:17:32.040588 14213 raft_consensus.cc:1261] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 4 FOLLOWER]: Refusing update from remote peer 15538fc3b6084a96a18216a2ec74cc3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8193. Preceding OpId from leader: term: 4 index: 8191. (term mismatch)
I20240506 16:17:32.040761 14213 pending_rounds.cc:76] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Aborting all ops after (but not including) 8190
I20240506 16:17:32.040824 14213 pending_rounds.cc:97] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Aborting uncommitted WRITE_OP operation due to leader change: 1.8191
W20240506 16:17:32.041345 14213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49682: Aborted: Op aborted by new leader
I20240506 16:17:32.042162 14213 pending_rounds.cc:97] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Aborting uncommitted WRITE_OP operation due to leader change: 1.8192
W20240506 16:17:32.042515 14213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49682: Aborted: Op aborted by new leader
I20240506 16:17:32.042901 14213 pending_rounds.cc:97] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Aborting uncommitted WRITE_OP operation due to leader change: 1.8193
W20240506 16:17:32.043228 14213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:49682: Aborted: Op aborted by new leader
I20240506 16:17:32.045603 16317 consensus_queue.cc:1035] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8191, Last known committed idx: 8190, Time since last communication: 0.000s
W20240506 16:17:32.337497 15211 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.12.173.137:37831, user_credentials={real_user=slave}} blocked reactor thread for 62333.9us
I20240506 16:17:34.071398 14193 tablet_service.cc:1422] Tablet server 5beda4231be140b8959d934a86e21bb1 set to quiescing
I20240506 16:17:34.071640 14193 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20240506 16:17:36.446280 14193 tablet_service.cc:1422] Tablet server 5beda4231be140b8959d934a86e21bb1 set to quiescing
I20240506 16:17:36.446533 14193 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20240506 16:17:40.536609 14193 tablet_service.cc:1422] Tablet server 5beda4231be140b8959d934a86e21bb1 set to quiescing
I20240506 16:17:40.536873 14193 tablet_service.cc:1429] Tablet server has 0 leaders and 2 scanners
I20240506 16:17:43.967392 13010 maintenance_manager.cc:419] P e31cda4593704d9c9f2ef36e57bde84a: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033399
I20240506 16:17:44.060379 13009 maintenance_manager.cc:643] P e31cda4593704d9c9f2ef36e57bde84a: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.092s user 0.031s sys 0.009s Metrics: {"bytes_written":7332,"cfile_init":1,"drs_written":1,"lbm_read_time_us":158,"lbm_reads_lt_1ms":4,"lbm_write_time_us":16641,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":14,"thread_start_us":10262,"threads_started":1,"wal-append.queue_time_us":10075}
I20240506 16:17:44.066224 13010 maintenance_manager.cc:419] P e31cda4593704d9c9f2ef36e57bde84a: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 672 bytes on disk
I20240506 16:17:44.067420 13009 maintenance_manager.cc:643] P e31cda4593704d9c9f2ef36e57bde84a: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
I20240506 16:17:44.615897 14193 tablet_service.cc:1422] Tablet server 5beda4231be140b8959d934a86e21bb1 set to quiescing
I20240506 16:17:44.616153 14193 tablet_service.cc:1429] Tablet server has 0 leaders and 2 scanners
W20240506 16:17:45.562106 14268 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.12.173.129:43195, user_credentials={real_user=slave}} blocked reactor thread for 68104.1us
I20240506 16:17:48.571501 14259 maintenance_manager.cc:419] P 5beda4231be140b8959d934a86e21bb1: Scheduling FlushMRSOp(5ea58089cc7849fbabaa4bde5c75d45f): perf score=0.049377
I20240506 16:17:48.674561 14193 tablet_service.cc:1422] Tablet server 5beda4231be140b8959d934a86e21bb1 set to quiescing
I20240506 16:17:48.674829 14193 tablet_service.cc:1429] Tablet server has 0 leaders and 1 scanners
W20240506 16:17:49.026095 14268 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.12.173.129:43195, user_credentials={real_user=slave}} blocked reactor thread for 95752.9us
I20240506 16:17:53.182843 14193 tablet_service.cc:1422] Tablet server 5beda4231be140b8959d934a86e21bb1 set to quiescing
I20240506 16:17:53.183146 14193 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:17:55.749876 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 13999
I20240506 16:17:55.825753 14121 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:17:56.350009 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 13999
I20240506 16:17:56.393293 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.136:34037
--local_ip_for_outbound_sockets=127.12.173.136
--tserver_master_addrs=127.12.173.190:43641
--webserver_port=36367
--webserver_interface=127.12.173.136
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20240506 16:17:57.213037 16492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:17:57.213528 16492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:17:57.214089 16492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:17:57.226325 16492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:17:57.226665 16492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.136
I20240506 16:17:57.241601 16492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.136:34037
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data/info.pb
--webserver_interface=127.12.173.136
--webserver_port=36367
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.136
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:17:57.243775 16492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:17:57.246454 16492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:17:57.318137 16497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:17:57.320029 16498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:17:57.333002 16492 server_base.cc:1034] running on GCE node
W20240506 16:17:57.335686 16500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:17:57.336695 16492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:17:57.368955 16492 hybrid_clock.cc:648] HybridClock initialized: now 1715012277368857 us; error 89 us; skew 500 ppm
I20240506 16:17:57.369532 16492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:17:57.581496 16492 webserver.cc:460] Webserver started at http://127.12.173.136:36367/ using document root <none> and password file <none>
I20240506 16:17:57.582764 16492 fs_manager.cc:359] Metadata directory not provided
I20240506 16:17:57.583010 16492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:17:57.597338 16492 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.005s sys 0.006s
I20240506 16:17:57.764959 16509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:17:57.770581 16492 fs_manager.cc:725] Time spent opening block manager: real 0.171s user 0.046s sys 0.088s
I20240506 16:17:57.770943 16492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/wal
uuid: "dc9fadb1e71e44c18dbade6135728983"
format_stamp: "Formatted at 2024-05-06 16:15:47 on dist-test-slave-g936"
I20240506 16:17:57.772394 16492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:17:57.920426 16492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:17:57.921785 16492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:17:57.922221 16492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:17:57.943615 16492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:17:57.952513 16492 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240506 16:17:57.952747 16492 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20240506 16:17:57.953051 16492 ts_tablet_manager.cc:590] Registered 0 tablets
I20240506 16:17:57.953239 16492 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240506 16:17:58.769101 16492 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.136:34037
I20240506 16:17:58.771402 16492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data/info.pb
I20240506 16:17:58.772091 16626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.136:34037 every 8 connection(s)
I20240506 16:17:58.779656 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 16492
I20240506 16:17:58.779870 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 14130
I20240506 16:17:58.842311 14252 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:17:58.866556 16627 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:17:58.867105 16627 heartbeater.cc:464] Registering TS with master...
I20240506 16:17:58.868391 16627 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:17:58.870518 13012 ts_manager.cc:194] Re-registered known tserver with Master: dc9fadb1e71e44c18dbade6135728983 (127.12.173.136:34037)
I20240506 16:17:58.871870 13012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.136:33539
I20240506 16:17:59.428025 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 14130
W20240506 16:17:59.534822 15211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.173.137:37831 (error 108)
I20240506 16:17:59.535945 12982 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGIku5t/build/asan/bin/kudu
/tmp/dist-test-taskGIku5t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.173.137:37831
--local_ip_for_outbound_sockets=127.12.173.137
--tserver_master_addrs=127.12.173.190:43641
--webserver_port=39563
--webserver_interface=127.12.173.137
--builtin_ntp_servers=127.12.173.148:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20240506 16:17:59.538707 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.137:37831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 16:17:59.882440 16627 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
W20240506 16:18:00.010728 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.137:37831: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240506 16:18:00.145267 16640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:18:00.145779 16640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240506 16:18:00.145989 16640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:18:00.160012 16640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240506 16:18:00.160353 16640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.173.137
I20240506 16:18:00.174631 16640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.173.148:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.173.137:37831
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data/info.pb
--webserver_interface=127.12.173.137
--webserver_port=39563
--tserver_master_addrs=127.12.173.190:43641
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.173.137
--log_dir=/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 1f0c0b0172fb211e74848c87fce983e45280e66d
build type FASTDEBUG
built by None at 06 May 2024 15:46:02 UTC on efd1337a2aa3
build id 163
ASAN enabled
I20240506 16:18:00.176697 16640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240506 16:18:00.188019 16640 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240506 16:18:00.259343 16647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240506 16:18:00.289711 16650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:18:00.290911 16640 server_base.cc:1034] running on GCE node
W20240506 16:18:00.303088 16648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240506 16:18:00.306051 16640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240506 16:18:00.314397 16640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240506 16:18:00.315971 16640 hybrid_clock.cc:648] HybridClock initialized: now 1715012280315861 us; error 87 us; skew 500 ppm
I20240506 16:18:00.316454 16640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240506 16:18:00.319993 16640 webserver.cc:460] Webserver started at http://127.12.173.137:39563/ using document root <none> and password file <none>
I20240506 16:18:00.324594 16640 fs_manager.cc:359] Metadata directory not provided
I20240506 16:18:00.324851 16640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240506 16:18:00.335326 16640 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.006s sys 0.000s
I20240506 16:18:00.356931 16658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240506 16:18:00.362527 16640 fs_manager.cc:725] Time spent opening block manager: real 0.025s user 0.004s sys 0.000s
I20240506 16:18:00.362774 16640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data,/tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/wal
uuid: "5beda4231be140b8959d934a86e21bb1"
format_stamp: "Formatted at 2024-05-06 16:15:48 on dist-test-slave-g936"
I20240506 16:18:00.364017 16640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/wal
metadata directory: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/wal
1 data directories: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240506 16:18:00.450546 16640 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240506 16:18:00.451478 16640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240506 16:18:00.451764 16640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240506 16:18:00.452896 16640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240506 16:18:00.455633 16665 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240506 16:18:00.483049 16640 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240506 16:18:00.483211 16640 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.028s user 0.001s sys 0.000s
I20240506 16:18:00.483352 16640 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240506 16:18:00.487830 16640 ts_tablet_manager.cc:590] Registered 1 tablets
I20240506 16:18:00.487934 16640 ts_tablet_manager.cc:569] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20240506 16:18:00.489017 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap starting.
W20240506 16:18:00.572047 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.137:37831: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240506 16:18:01.164227 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.137:37831: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240506 16:18:01.278745 16665 log.cc:824] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Log is configured to *not* fsync() on all Append() calls
W20240506 16:18:01.663264 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.137:37831: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240506 16:18:01.859401 16640 rpc_server.cc:307] RPC server started. Bound to: 127.12.173.137:37831
I20240506 16:18:01.859445 16775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.173.137:37831 every 8 connection(s)
I20240506 16:18:01.861409 16640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data/info.pb
I20240506 16:18:01.867250 12982 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGIku5t/build/asan/bin/kudu as pid 16640
I20240506 16:18:01.984470 16776 heartbeater.cc:347] Connected to a master server at 127.12.173.190:43641
I20240506 16:18:01.984791 16776 heartbeater.cc:464] Registering TS with master...
I20240506 16:18:01.986035 16776 heartbeater.cc:511] Master 127.12.173.190:43641 requested a full tablet report, sending...
I20240506 16:18:01.989132 13012 ts_manager.cc:194] Re-registered known tserver with Master: 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831)
I20240506 16:18:01.990748 13012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.173.137:52189
W20240506 16:18:02.343863 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
W20240506 16:18:02.950335 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20240506 16:18:03.001302 16776 heartbeater.cc:503] Master 127.12.173.190:43641 was elected leader, sending a full tablet report...
I20240506 16:18:03.382908 15262 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:18:03.454077 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:03.474720 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:03.544692 16708 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:03.595118 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:03.634610 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:03.720672 15830 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 16:18:03.730736 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20240506 16:18:03.749617 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:03.769693 16560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:03.851418 16476 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 3 ops behind the committed index
W20240506 16:18:04.367154 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20240506 16:18:04.840415 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20240506 16:18:05.131076 14268 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.12.173.129:43195, user_credentials={real_user=slave}} blocked reactor thread for 51039.6us
W20240506 16:18:05.415598 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
W20240506 16:18:05.946981 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
I20240506 16:18:06.293463 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replaying log segment 1/3 (1.37M/8.00M this segment, stats: ops{read=790 overwritten=0 applied=790 ignored=0} inserts{seen=39450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:18:06.528056 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
I20240506 16:18:07.059978 16828 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 319 ops behind the committed index [suppressed 26 similar messages]
W20240506 16:18:07.197404 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
W20240506 16:18:07.682226 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
W20240506 16:18:08.159621 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
W20240506 16:18:08.664397 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20240506 16:18:09.230190 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
W20240506 16:18:09.800334 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
I20240506 16:18:10.123041 16832 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 609 ops behind the committed index [suppressed 28 similar messages]
W20240506 16:18:10.379030 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
I20240506 16:18:10.643908 15262 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:18:10.777197 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:10.827679 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:10.872917 16708 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:10.994401 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:11.016669 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:11.041380 15830 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:11.043499 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:11.053385 16560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 16:18:11.136453 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
I20240506 16:18:11.300408 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replaying log segment 1/3 (3.03M/8.00M this segment, stats: ops{read=1751 overwritten=0 applied=1751 ignored=0} inserts{seen=87500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:18:11.640306 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20240506 16:18:12.201772 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
W20240506 16:18:12.707903 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
I20240506 16:18:13.222812 16894 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 877 ops behind the committed index [suppressed 26 similar messages]
W20240506 16:18:13.251480 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
W20240506 16:18:13.791735 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
W20240506 16:18:14.327033 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
W20240506 16:18:14.910602 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
W20240506 16:18:15.250414 14268 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.12.173.129:43195, user_credentials={real_user=slave}} blocked reactor thread for 48060.4us
W20240506 16:18:15.470844 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
W20240506 16:18:16.026224 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
I20240506 16:18:16.241853 16858 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 1177 ops behind the committed index [suppressed 27 similar messages]
I20240506 16:18:16.300490 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replaying log segment 1/3 (4.96M/8.00M this segment, stats: ops{read=2868 overwritten=0 applied=2865 ignored=0} inserts{seen=143200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:18:16.483844 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
W20240506 16:18:17.046556 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 156: this message will repeat every 5th retry.
W20240506 16:18:17.653501 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 161: this message will repeat every 5th retry.
W20240506 16:18:18.150331 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 166: this message will repeat every 5th retry.
I20240506 16:18:18.343523 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:18.452472 16708 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:18.456596 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:18.490746 15262 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:18:18.496762 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:18.529912 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:18.585665 16560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:18.603513 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:18.622560 15830 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 16:18:18.862416 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 171: this message will repeat every 5th retry.
I20240506 16:18:19.333675 16919 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 1431 ops behind the committed index [suppressed 25 similar messages]
W20240506 16:18:19.475081 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 176: this message will repeat every 5th retry.
W20240506 16:18:20.056656 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 181: this message will repeat every 5th retry.
W20240506 16:18:20.597087 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 186: this message will repeat every 5th retry.
W20240506 16:18:21.200018 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 191: this message will repeat every 5th retry.
I20240506 16:18:21.302521 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replaying log segment 1/3 (6.47M/8.00M this segment, stats: ops{read=3738 overwritten=0 applied=3738 ignored=0} inserts{seen=186850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:18:21.722965 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 196: this message will repeat every 5th retry.
W20240506 16:18:22.232415 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 201: this message will repeat every 5th retry.
I20240506 16:18:22.380924 16964 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 1724 ops behind the committed index [suppressed 27 similar messages]
W20240506 16:18:22.528437 14268 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.12.173.129:43195, user_credentials={real_user=slave}} blocked reactor thread for 47287.2us
W20240506 16:18:22.735379 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 206: this message will repeat every 5th retry.
W20240506 16:18:23.252645 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 211: this message will repeat every 5th retry.
W20240506 16:18:23.783703 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 216: this message will repeat every 5th retry.
W20240506 16:18:24.268221 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 221: this message will repeat every 5th retry.
W20240506 16:18:24.815724 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 226: this message will repeat every 5th retry.
W20240506 16:18:25.393689 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 231: this message will repeat every 5th retry.
I20240506 16:18:25.433365 16965 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 1990 ops behind the committed index [suppressed 28 similar messages]
I20240506 16:18:25.620956 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replayed 1/3 log segments. Stats: ops{read=4621 overwritten=0 applied=4620 ignored=0} inserts{seen=230950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240506 16:18:25.712491 15262 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:18:25.724042 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:25.774052 16708 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:25.823565 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:25.823593 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:25.849897 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:25.873956 15830 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:26.003438 16560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:26.028438 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 16:18:26.040443 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 236: this message will repeat every 5th retry.
I20240506 16:18:26.307904 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replaying log segment 2/3 (124.4K/8.00M this segment, stats: ops{read=4691 overwritten=0 applied=4691 ignored=0} inserts{seen=234500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:18:26.522589 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 241: this message will repeat every 5th retry.
W20240506 16:18:27.102260 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 246: this message will repeat every 5th retry.
W20240506 16:18:27.661448 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 251: this message will repeat every 5th retry.
W20240506 16:18:28.171118 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 256: this message will repeat every 5th retry.
I20240506 16:18:28.534260 17023 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 2279 ops behind the committed index [suppressed 27 similar messages]
W20240506 16:18:28.690515 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 261: this message will repeat every 5th retry.
W20240506 16:18:29.262210 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 266: this message will repeat every 5th retry.
W20240506 16:18:29.727255 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 271: this message will repeat every 5th retry.
W20240506 16:18:30.234325 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 276: this message will repeat every 5th retry.
W20240506 16:18:30.819566 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 281: this message will repeat every 5th retry.
I20240506 16:18:31.310654 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replaying log segment 2/3 (1.87M/8.00M this segment, stats: ops{read=5699 overwritten=0 applied=5699 ignored=0} inserts{seen=284900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:18:31.348291 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 286: this message will repeat every 5th retry.
I20240506 16:18:31.557570 16965 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 2572 ops behind the committed index [suppressed 27 similar messages]
I20240506 16:18:31.976428 15334 maintenance_manager.cc:419] P 15538fc3b6084a96a18216a2ec74cc3a: Scheduling FlushMRSOp(5ea58089cc7849fbabaa4bde5c75d45f): perf score=0.067787
W20240506 16:18:31.985338 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 291: this message will repeat every 5th retry.
W20240506 16:18:32.760335 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 296: this message will repeat every 5th retry.
I20240506 16:18:33.340230 16708 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:33.372512 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:33.386174 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:33.386301 15262 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:18:33.467439 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:33.484318 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:33.500083 16560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:33.513351 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:33.565572 15830 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 16:18:33.657930 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 301: this message will repeat every 5th retry.
W20240506 16:18:34.280347 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 306: this message will repeat every 5th retry.
I20240506 16:18:34.599278 17024 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 2774 ops behind the committed index [suppressed 20 similar messages]
W20240506 16:18:34.985592 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 311: this message will repeat every 5th retry.
W20240506 16:18:35.475078 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 316: this message will repeat every 5th retry.
W20240506 16:18:36.029034 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 321: this message will repeat every 5th retry.
I20240506 16:18:36.311136 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replaying log segment 2/3 (3.04M/8.00M this segment, stats: ops{read=6374 overwritten=0 applied=6373 ignored=0} inserts{seen=318600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:18:36.704782 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 326: this message will repeat every 5th retry.
W20240506 16:18:37.166429 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 331: this message will repeat every 5th retry.
I20240506 16:18:37.644002 16969 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 3012 ops behind the committed index [suppressed 26 similar messages]
W20240506 16:18:37.716756 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 336: this message will repeat every 5th retry.
W20240506 16:18:38.269129 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 341: this message will repeat every 5th retry.
W20240506 16:18:38.809666 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 346: this message will repeat every 5th retry.
W20240506 16:18:39.304009 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 351: this message will repeat every 5th retry.
W20240506 16:18:39.824345 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 356: this message will repeat every 5th retry.
I20240506 16:18:40.675871 17041 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 3217 ops behind the committed index [suppressed 24 similar messages]
W20240506 16:18:40.729440 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 361: this message will repeat every 5th retry.
I20240506 16:18:41.116686 15262 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:18:41.157934 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:41.160480 16708 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:41.170274 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:41.213766 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:41.312278 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replaying log segment 2/3 (4.42M/8.00M this segment, stats: ops{read=7173 overwritten=0 applied=7172 ignored=0} inserts{seen=358550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20240506 16:18:41.313939 16560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:41.346032 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:41.350982 15830 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:41.362845 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 16:18:41.453334 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 366: this message will repeat every 5th retry.
W20240506 16:18:42.203043 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 371: this message will repeat every 5th retry.
W20240506 16:18:42.926273 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 376: this message will repeat every 5th retry.
W20240506 16:18:43.438589 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 381: this message will repeat every 5th retry.
I20240506 16:18:43.720456 17024 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 3432 ops behind the committed index [suppressed 22 similar messages]
W20240506 16:18:44.043301 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 386: this message will repeat every 5th retry.
W20240506 16:18:44.582216 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 391: this message will repeat every 5th retry.
W20240506 16:18:45.144591 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 396: this message will repeat every 5th retry.
W20240506 16:18:45.690881 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 401: this message will repeat every 5th retry.
W20240506 16:18:46.259680 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 406: this message will repeat every 5th retry.
I20240506 16:18:46.313329 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replaying log segment 2/3 (5.80M/8.00M this segment, stats: ops{read=7970 overwritten=0 applied=7969 ignored=0} inserts{seen=398400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:18:46.751580 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 411: this message will repeat every 5th retry.
I20240506 16:18:46.838102 17141 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 3670 ops behind the committed index [suppressed 27 similar messages]
W20240506 16:18:47.368942 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 416: this message will repeat every 5th retry.
W20240506 16:18:47.893431 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 421: this message will repeat every 5th retry.
W20240506 16:18:48.608971 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 426: this message will repeat every 5th retry.
W20240506 16:18:49.361398 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 431: this message will repeat every 5th retry.
I20240506 16:18:49.907122 17155 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 3880 ops behind the committed index [suppressed 22 similar messages]
I20240506 16:18:49.979449 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:50.031788 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:50.049779 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:50.067112 16708 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:50.067240 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:50.081864 15262 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:18:50.145057 16560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:50.206312 15830 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:50.247380 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 16:18:50.253417 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 436: this message will repeat every 5th retry.
W20240506 16:18:51.028076 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 441: this message will repeat every 5th retry.
I20240506 16:18:51.326798 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replaying log segment 2/3 (6.71M/8.00M this segment, stats: ops{read=8507 overwritten=3 applied=8503 ignored=0} inserts{seen=425050 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:18:51.643180 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 446: this message will repeat every 5th retry.
W20240506 16:18:52.212906 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 451: this message will repeat every 5th retry.
W20240506 16:18:52.672884 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 456: this message will repeat every 5th retry.
I20240506 16:18:52.909497 17155 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 4093 ops behind the committed index [suppressed 23 similar messages]
W20240506 16:18:53.123778 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 461: this message will repeat every 5th retry.
W20240506 16:18:53.606043 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 466: this message will repeat every 5th retry.
W20240506 16:18:54.171155 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 471: this message will repeat every 5th retry.
W20240506 16:18:54.702332 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 476: this message will repeat every 5th retry.
W20240506 16:18:55.328713 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 481: this message will repeat every 5th retry.
I20240506 16:18:55.843943 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replayed 2/3 log segments. Stats: ops{read=9281 overwritten=3 applied=9275 ignored=0} inserts{seen=463650 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
W20240506 16:18:55.928752 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 486: this message will repeat every 5th retry.
I20240506 16:18:55.971664 17209 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 4320 ops behind the committed index [suppressed 27 similar messages]
I20240506 16:18:56.331668 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replaying log segment 3/3 (129.7K/1.77M this segment, stats: ops{read=9357 overwritten=3 applied=9353 ignored=0} inserts{seen=467550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:18:56.712940 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 491: this message will repeat every 5th retry.
W20240506 16:18:57.474782 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 496: this message will repeat every 5th retry.
W20240506 16:18:58.149199 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 501: this message will repeat every 5th retry.
I20240506 16:18:58.544896 15262 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20240506 16:18:58.557587 14772 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:58.622895 16708 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:58.630028 15404 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:58.676671 14630 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:58.689041 14491 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:58.699810 15117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:58.702275 16560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240506 16:18:58.760157 15830 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240506 16:18:58.857024 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 506: this message will repeat every 5th retry.
I20240506 16:18:58.990747 17209 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 4505 ops behind the committed index [suppressed 20 similar messages]
W20240506 16:18:59.529443 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 511: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/maintenance_mode-itest.cc:753: Failure
Failed
Bad status: Timed out: cluster didn't become healthy: Running 'cluster ksck 127.12.173.190:43641' did not succeed in 60 seconds: Runtime error: error running 'cluster ksck 127.12.173.190:43641': stdout: Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
e31cda4593704d9c9f2ef36e57bde84a | 127.12.173.190:43641 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.173.148:40193 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server States
Server | State
----------------------------------+------------------
5beda4231be140b8959d934a86e21bb1 | MAINTENANCE_MODE
dc9fadb1e71e44c18dbade6135728983 | MAINTENANCE_MODE
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
15538fc3b6084a96a18216a2ec74cc3a | 127.12.173.129:43195 | HEALTHY | /L0 | 1 | 3
18c7d19aee15430b8c3a0ef90b23fee5 | 127.12.173.135:43823 | HEALTHY | /L1 | 0 | 0
405f2b51c2244be3b1a66f2c908bcdd9 | 127.12.173.131:33797 | HEALTHY | /L0 | 0 | 0
5beda4231be140b8959d934a86e21bb1 | 127.12.173.137:37831 | HEALTHY | /L2 | 0 | 0
c0471a9ec7ba4ea38e8df76c78d8b38f | 127.12.173.134:36633 | HEALTHY | /L1 | 0 | 0
cea103e071de48bbbad5e030eca24b4a | 127.12.173.130:42665 | HEALTHY | /L0 | 0 | 0
dc9fadb1e71e44c18dbade6135728983 | 127.12.173.136:34037 | HEALTHY | /L2 | 0 | 0
e6d1cddba8c94532b40c0ae9c06740ef | 127.12.173.132:39055 | HEALTHY | /L1 | 0 | 0
fd8906dd0f884e22911dce7df910b6d4 | 127.12.173.133:42687 | HEALTHY | /L1 | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/L2 | 2
/L0 | 3
/L1 | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 9 server(s) checked
local_ip_for_outbound_sockets | 127.12.173.129 | experimental | 127.12.173.129:43195
local_ip_for_outbound_sockets | 127.12.173.130 | experimental | 127.12.173.130:42665
local_ip_for_outbound_sockets | 127.12.173.131 | experimental | 127.12.173.131:33797
local_ip_for_outbound_sockets | 127.12.173.132 | experimental | 127.12.173.132:39055
local_ip_for_outbound_sockets | 127.12.173.133 | experimental | 127.12.173.133:42687
local_ip_for_outbound_sockets | 127.12.173.134 | experimental | 127.12.173.134:36633
local_ip_for_outbound_sockets | 127.12.173.135 | experimental | 127.12.173.135:43823
local_ip_for_outbound_sockets | 127.12.173.136 | experimental | 127.12.173.136:34037
local_ip_for_outbound_sockets | 127.12.173.137 | experimental | 127.12.173.137:37831
never_fsync | true | unsafe,advanced | all 9 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 9 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked
server_dump_info_format | pb | hidden | all 9 server(s) checked
server_dump_info_path | /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/data/info.pb | hidden | 127.12.173.129:43195
server_dump_info_path | /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-1/data/info.pb | hidden | 127.12.173.130:42665
server_dump_info_path | /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-2/data/info.pb | hidden | 127.12.173.131:33797
server_dump_info_path | /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-3/data/info.pb | hidden | 127.12.173.132:39055
server_dump_info_path | /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-4/data/info.pb | hidden | 127.12.173.133:42687
server_dump_info_path | /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-5/data/info.pb | hidden | 127.12.173.134:36633
server_dump_info_path | /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-6/data/info.pb | hidden | 127.12.173.135:43823
server_dump_info_path | /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-7/data/info.pb | hidden | 127.12.173.136:34037
server_dump_info_path | /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-8/data/info.pb | hidden | 127.12.173.137:37831
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.12.173.148:40193 | all 9 server(s) checked
time_source | builtin | all 9 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------
1.18.0-SNAPSHOT | all 10 server(s) checked
Tablet Summary
Tablet 5ea58089cc7849fbabaa4bde5c75d45f of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): RUNNING [LEADER]
e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): RUNNING
5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): not running
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap replaying log segment 3/3 (129.7K/1.77M this segment, stats: ops{read=9357 overwritten=3 applied=9353 ignored=0} inserts{seen=467550 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
All reported replicas are:
A = 15538fc3b6084a96a18216a2ec74cc3a
B = e6d1cddba8c94532b40c0ae9c06740ef
C = 5beda4231be140b8959d934a86e21bb1
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 4 | -1 | Yes
B | A* B C | 4 | -1 | Yes
C | A B C | 4 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 9
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
, stderr: W20240506 16:18:57.801632 17214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240506 16:18:57.802156 17214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240506 16:18:57.865914 17214 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Runtime error: ksck discovered errors
: /tmp/dist-test-taskGIku5t/build/asan/bin/kudu: process exited with non-zero status 1
W20240506 16:18:59.994772 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 516: this message will repeat every 5th retry.
W20240506 16:19:00.496311 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 521: this message will repeat every 5th retry.
W20240506 16:19:00.969245 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 526: this message will repeat every 5th retry.
I20240506 16:19:01.332648 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replaying log segment 3/3 (1.54M/1.77M this segment, stats: ops{read=10203 overwritten=3 applied=10195 ignored=0} inserts{seen=509650 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20240506 16:19:01.547958 15211 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a -> Peer 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831): Couldn't send request to peer 5beda4231be140b8959d934a86e21bb1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 531: this message will repeat every 5th retry.
I20240506 16:19:01.874433 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap replayed 3/3 log segments. Stats: ops{read=10339 overwritten=3 applied=10325 ignored=0} inserts{seen=516150 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 11 replicates
I20240506 16:19:01.875394 16665 tablet_bootstrap.cc:492] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Bootstrap complete.
I20240506 16:19:01.881680 16665 ts_tablet_manager.cc:1367] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Time spent bootstrapping tablet: real 61.393s user 43.382s sys 1.406s
I20240506 16:19:01.892324 16665 raft_consensus.cc:348] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 4 FOLLOWER]: Replica starting. Triggering 11 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:19:01.943619 16665 raft_consensus.cc:729] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5beda4231be140b8959d934a86e21bb1, State: Initialized, Role: FOLLOWER
I20240506 16:19:01.944712 16665 consensus_queue.cc:260] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10325, Last appended: 4.10336, Last appended by leader: 10336, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:19:01.966624 16665 ts_tablet_manager.cc:1398] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1: Time spent starting tablet: real 0.085s user 0.043s sys 0.036s
I20240506 16:19:02.022008 17268 consensus_queue.cc:786] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a [LEADER]: Peer 5beda4231be140b8959d934a86e21bb1 is lagging by at least 4555 ops behind the committed index [suppressed 27 similar messages]
I20240506 16:19:02.052333 16728 raft_consensus.cc:1203] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 4.10335->[4.10336-4.10760] Dedup: 4.10336->[4.10337-4.10760]
I20240506 16:19:05.359223 15217 maintenance_manager.cc:643] P 15538fc3b6084a96a18216a2ec74cc3a: FlushMRSOp(5ea58089cc7849fbabaa4bde5c75d45f) complete. Timing: real 33.381s user 22.091s sys 0.442s Metrics: {"bytes_written":18537513,"cfile_init":1,"dirs.queue_time_us":11909,"dirs.run_cpu_time_us":322,"dirs.run_wall_time_us":14964,"drs_written":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4,"lbm_write_time_us":149420,"lbm_writes_1-10_ms":8,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":801,"peak_mem_usage":0,"rows_written":668100,"thread_start_us":19862,"threads_started":2,"wal-append.queue_time_us":9412}
I20240506 16:19:05.361418 15334 maintenance_manager.cc:419] P 15538fc3b6084a96a18216a2ec74cc3a: Scheduling UndoDeltaBlockGCOp(5ea58089cc7849fbabaa4bde5c75d45f): 7657634 bytes on disk
I20240506 16:19:05.362507 15217 maintenance_manager.cc:643] P 15538fc3b6084a96a18216a2ec74cc3a: UndoDeltaBlockGCOp(5ea58089cc7849fbabaa4bde5c75d45f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4}
I20240506 16:19:10.342491 12982 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20240506 16:19:10.343086 12982 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 15538fc3b6084a96a18216a2ec74cc3a and pid 15195
I20240506 16:19:10.372105 15334 maintenance_manager.cc:419] P 15538fc3b6084a96a18216a2ec74cc3a: Scheduling LogGCOp(5ea58089cc7849fbabaa4bde5c75d45f): free 16783062 bytes of WAL
I20240506 16:19:10.372563 15217 log_reader.cc:386] T 5ea58089cc7849fbabaa4bde5c75d45f: removed 2 log segments from log reader
I20240506 16:19:10.372650 15217 log.cc:1076] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Deleting log segment in path: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/wal/wals/5ea58089cc7849fbabaa4bde5c75d45f/wal-000000001 (ops 1-4899)
I20240506 16:19:10.372795 15217 log.cc:1076] T 5ea58089cc7849fbabaa4bde5c75d45f P 15538fc3b6084a96a18216a2ec74cc3a: Deleting log segment in path: /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0/minicluster-data/ts-0/wal/wals/5ea58089cc7849fbabaa4bde5c75d45f/wal-000000002 (ops 4900-9592)
I20240506 16:19:10.377287 15217 maintenance_manager.cc:643] P 15538fc3b6084a96a18216a2ec74cc3a: LogGCOp(5ea58089cc7849fbabaa4bde5c75d45f) complete. Timing: real 0.005s user 0.001s sys 0.004s Metrics: {}
************************ BEGIN STACKS **************************
[New LWP 15197]
[New LWP 15198]
[New LWP 15199]
[New LWP 15200]
[New LWP 15206]
[New LWP 15207]
[New LWP 15208]
[New LWP 15211]
[New LWP 15214]
[New LWP 15215]
[New LWP 15216]
[New LWP 15217]
[New LWP 15219]
[New LWP 15221]
[New LWP 15222]
[New LWP 15223]
[New LWP 15224]
[New LWP 15225]
[New LWP 15226]
[New LWP 15227]
[New LWP 15228]
[New LWP 15229]
[New LWP 15230]
[New LWP 15231]
[New LWP 15232]
[New LWP 15233]
[New LWP 15234]
[New LWP 15235]
[New LWP 15236]
[New LWP 15237]
[New LWP 15238]
[New LWP 15239]
[New LWP 15240]
[New LWP 15241]
[New LWP 15242]
[New LWP 15243]
[New LWP 15244]
[New LWP 15245]
[New LWP 15246]
[New LWP 15247]
[New LWP 15248]
[New LWP 15249]
[New LWP 15250]
[New LWP 15251]
[New LWP 15252]
[New LWP 15253]
[New LWP 15254]
[New LWP 15255]
[New LWP 15256]
[New LWP 15257]
[New LWP 15258]
[New LWP 15259]
[New LWP 15260]
[New LWP 15261]
[New LWP 15262]
[New LWP 15263]
[New LWP 15264]
[New LWP 15265]
[New LWP 15266]
[New LWP 15267]
[New LWP 15268]
[New LWP 15269]
[New LWP 15270]
[New LWP 15271]
[New LWP 15272]
[New LWP 15273]
[New LWP 15274]
[New LWP 15275]
[New LWP 15276]
[New LWP 15277]
[New LWP 15278]
[New LWP 15279]
[New LWP 15280]
[New LWP 15281]
[New LWP 15283]
[New LWP 15284]
[New LWP 15285]
[New LWP 15287]
[New LWP 15288]
[New LWP 15289]
[New LWP 15290]
[New LWP 15291]
[New LWP 15292]
[New LWP 15293]
[New LWP 15294]
[New LWP 15295]
[New LWP 15296]
[New LWP 15297]
[New LWP 15298]
[New LWP 15299]
[New LWP 15300]
[New LWP 15301]
[New LWP 15302]
[New LWP 15303]
[New LWP 15304]
[New LWP 15305]
[New LWP 15306]
[New LWP 15307]
[New LWP 15308]
[New LWP 15309]
[New LWP 15310]
[New LWP 15311]
[New LWP 15312]
[New LWP 15313]
[New LWP 15314]
[New LWP 15315]
[New LWP 15316]
[New LWP 15317]
[New LWP 15318]
[New LWP 15319]
[New LWP 15320]
[New LWP 15321]
[New LWP 15322]
[New LWP 15323]
[New LWP 15324]
[New LWP 15325]
[New LWP 15326]
[New LWP 15327]
[New LWP 15328]
[New LWP 15329]
[New LWP 15334]
[New LWP 17283]
[New LWP 17289]
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
0x00007fa5f4366d50 in ?? ()
Id Target Id Frame
* 1 LWP 15195 "kudu" 0x00007fa5f4366d50 in ?? ()
2 LWP 15197 "kudu" 0x00007fa5f4362fb9 in ?? ()
3 LWP 15198 "kudu" 0x00007fa5f4362fb9 in ?? ()
4 LWP 15199 "kudu" 0x00007fa5f4362fb9 in ?? ()
5 LWP 15200 "kernel-watcher-" 0x00007fa5f4362fb9 in ?? ()
6 LWP 15206 "ntp client-1520" 0x00007fa5f43669e2 in ?? ()
7 LWP 15207 "file cache-evic" 0x00007fa5f4362fb9 in ?? ()
8 LWP 15208 "sq_acceptor" 0x00007fa5ed18dcb9 in ?? ()
9 LWP 15211 "rpc reactor-152" 0x00007fa5ed19aa47 in ?? ()
10 LWP 15214 "rpc reactor-152" 0x00007fa5ed19aa47 in ?? ()
11 LWP 15215 "rpc reactor-152" 0x00007fa5ed19aa47 in ?? ()
12 LWP 15216 "rpc reactor-152" 0x00007fa5ed19aa47 in ?? ()
13 LWP 15217 "MaintenanceMgr " 0x00007fa5f4362ad3 in ?? ()
14 LWP 15219 "txn-status-mana" 0x00007fa5f4362fb9 in ?? ()
15 LWP 15221 "collect_and_rem" 0x00007fa5f4362fb9 in ?? ()
16 LWP 15222 "tc-session-exp-" 0x00007fa5f4362fb9 in ?? ()
17 LWP 15223 "rpc worker-1522" 0x00007fa5f4362ad3 in ?? ()
18 LWP 15224 "rpc worker-1522" 0x00007fa5f4362ad3 in ?? ()
19 LWP 15225 "rpc worker-1522" 0x00007fa5f4362ad3 in ?? ()
20 LWP 15226 "rpc worker-1522" 0x00007fa5f4362ad3 in ?? ()
21 LWP 15227 "rpc worker-1522" 0x00007fa5f4362ad3 in ?? ()
22 LWP 15228 "rpc worker-1522" 0x00007fa5f4362ad3 in ?? ()
23 LWP 15229 "rpc worker-1522" 0x00007fa5f4362ad3 in ?? ()
24 LWP 15230 "rpc worker-1523" 0x00007fa5f4362ad3 in ?? ()
25 LWP 15231 "rpc worker-1523" 0x00007fa5f4362ad3 in ?? ()
26 LWP 15232 "rpc worker-1523" 0x00007fa5f4362ad3 in ?? ()
27 LWP 15233 "rpc worker-1523" 0x00007fa5f4362ad3 in ?? ()
28 LWP 15234 "rpc worker-1523" 0x00007fa5f4362ad3 in ?? ()
29 LWP 15235 "rpc worker-1523" 0x00007fa5f4362ad3 in ?? ()
30 LWP 15236 "rpc worker-1523" 0x00007fa5f4362ad3 in ?? ()
31 LWP 15237 "rpc worker-1523" 0x00007fa5f4362ad3 in ?? ()
32 LWP 15238 "rpc worker-1523" 0x00007fa5f4362ad3 in ?? ()
33 LWP 15239 "rpc worker-1523" 0x00007fa5f4362ad3 in ?? ()
34 LWP 15240 "rpc worker-1524" 0x00007fa5f4362ad3 in ?? ()
35 LWP 15241 "rpc worker-1524" 0x00007fa5f4362ad3 in ?? ()
36 LWP 15242 "rpc worker-1524" 0x00007fa5f4362ad3 in ?? ()
37 LWP 15243 "rpc worker-1524" 0x00007fa5f4362ad3 in ?? ()
38 LWP 15244 "rpc worker-1524" 0x00007fa5f4362ad3 in ?? ()
39 LWP 15245 "rpc worker-1524" 0x00007fa5f4362ad3 in ?? ()
40 LWP 15246 "rpc worker-1524" 0x00007fa5f4362ad3 in ?? ()
41 LWP 15247 "rpc worker-1524" 0x00007fa5f4362ad3 in ?? ()
42 LWP 15248 "rpc worker-1524" 0x00007fa5f4362ad3 in ?? ()
43 LWP 15249 "rpc worker-1524" 0x00007fa5f4362ad3 in ?? ()
44 LWP 15250 "rpc worker-1525" 0x00007fa5f4362ad3 in ?? ()
45 LWP 15251 "rpc worker-1525" 0x00007fa5f4362ad3 in ?? ()
46 LWP 15252 "rpc worker-1525" 0x00007fa5f4362ad3 in ?? ()
47 LWP 15253 "rpc worker-1525" 0x00007fa5f4362ad3 in ?? ()
48 LWP 15254 "rpc worker-1525" 0x00007fa5f4362ad3 in ?? ()
49 LWP 15255 "rpc worker-1525" 0x00007fa5f4362ad3 in ?? ()
50 LWP 15256 "rpc worker-1525" 0x00007fa5f4362ad3 in ?? ()
51 LWP 15257 "rpc worker-1525" 0x00007fa5f4362ad3 in ?? ()
52 LWP 15258 "rpc worker-1525" 0x00007fa5f4362ad3 in ?? ()
53 LWP 15259 "rpc worker-1525" 0x00007fa5f4362ad3 in ?? ()
54 LWP 15260 "rpc worker-1526" 0x00007fa5f4362ad3 in ?? ()
55 LWP 15261 "rpc worker-1526" 0x00007fa5f4362ad3 in ?? ()
56 LWP 15262 "rpc worker-1526" 0x00007fa5f4362ad3 in ?? ()
57 LWP 15263 "rpc worker-1526" 0x00007fa5f4362ad3 in ?? ()
58 LWP 15264 "rpc worker-1526" 0x00007fa5f4362ad3 in ?? ()
59 LWP 15265 "rpc worker-1526" 0x00007fa5f4362ad3 in ?? ()
60 LWP 15266 "rpc worker-1526" 0x00007fa5f4362ad3 in ?? ()
61 LWP 15267 "rpc worker-1526" 0x00007fa5f4362ad3 in ?? ()
62 LWP 15268 "rpc worker-1526" 0x00007fa5f4362ad3 in ?? ()
63 LWP 15269 "rpc worker-1526" 0x00007fa5f4362ad3 in ?? ()
64 LWP 15270 "rpc worker-1527" 0x00007fa5f4362ad3 in ?? ()
65 LWP 15271 "rpc worker-1527" 0x00007fa5f4362ad3 in ?? ()
66 LWP 15272 "rpc worker-1527" 0x00007fa5f4362ad3 in ?? ()
67 LWP 15273 "rpc worker-1527" 0x00007fa5f4362ad3 in ?? ()
68 LWP 15274 "rpc worker-1527" 0x00007fa5f4362ad3 in ?? ()
69 LWP 15275 "rpc worker-1527" 0x00007fa5f4362ad3 in ?? ()
70 LWP 15276 "rpc worker-1527" 0x00007fa5f4362ad3 in ?? ()
71 LWP 15277 "rpc worker-1527" 0x00007fa5f4362ad3 in ?? ()
72 LWP 15278 "rpc worker-1527" 0x00007fa5f4362ad3 in ?? ()
73 LWP 15279 "rpc worker-1527" 0x00007fa5f4362ad3 in ?? ()
74 LWP 15280 "rpc worker-1528" 0x00007fa5f4362ad3 in ?? ()
75 LWP 15281 "rpc worker-1528" 0x00007fa5f4362ad3 in ?? ()
76 LWP 15283 "rpc worker-1528" 0x00007fa5f4362ad3 in ?? ()
77 LWP 15284 "rpc worker-1528" 0x00007fa5f4362ad3 in ?? ()
78 LWP 15285 "rpc worker-1528" 0x00007fa5f4362ad3 in ?? ()
79 LWP 15287 "rpc worker-1528" 0x00007fa5f4362ad3 in ?? ()
80 LWP 15288 "rpc worker-1528" 0x00007fa5f4362ad3 in ?? ()
81 LWP 15289 "rpc worker-1528" 0x00007fa5f4362ad3 in ?? ()
82 LWP 15290 "rpc worker-1529" 0x00007fa5f4362ad3 in ?? ()
83 LWP 15291 "rpc worker-1529" 0x00007fa5f4362ad3 in ?? ()
84 LWP 15292 "rpc worker-1529" 0x00007fa5f4362ad3 in ?? ()
85 LWP 15293 "rpc worker-1529" 0x00007fa5f4362ad3 in ?? ()
86 LWP 15294 "rpc worker-1529" 0x00007fa5f4362ad3 in ?? ()
87 LWP 15295 "rpc worker-1529" 0x00007fa5f4362ad3 in ?? ()
88 LWP 15296 "rpc worker-1529" 0x00007fa5f4362ad3 in ?? ()
89 LWP 15297 "rpc worker-1529" 0x00007fa5f4362ad3 in ?? ()
90 LWP 15298 "rpc worker-1529" 0x00007fa5f4362ad3 in ?? ()
91 LWP 15299 "rpc worker-1529" 0x00007fa5f4362ad3 in ?? ()
92 LWP 15300 "rpc worker-1530" 0x00007fa5f4362ad3 in ?? ()
93 LWP 15301 "rpc worker-1530" 0x00007fa5f4362ad3 in ?? ()
94 LWP 15302 "rpc worker-1530" 0x00007fa5f4362ad3 in ?? ()
95 LWP 15303 "rpc worker-1530" 0x00007fa5f4362ad3 in ?? ()
96 LWP 15304 "rpc worker-1530" 0x00007fa5f4362ad3 in ?? ()
97 LWP 15305 "rpc worker-1530" 0x00007fa5f4362ad3 in ?? ()
98 LWP 15306 "rpc worker-1530" 0x00007fa5f4362ad3 in ?? ()
99 LWP 15307 "rpc worker-1530" 0x00007fa5f4362ad3 in ?? ()
100 LWP 15308 "rpc worker-1530" 0x00007fa5f4362ad3 in ?? ()
101 LWP 15309 "rpc worker-1530" 0x00007fa5f4362ad3 in ?? ()
102 LWP 15310 "rpc worker-1531" 0x00007fa5f4362ad3 in ?? ()
103 LWP 15311 "rpc worker-1531" 0x00007fa5f4362ad3 in ?? ()
104 LWP 15312 "rpc worker-1531" 0x00007fa5f4362ad3 in ?? ()
105 LWP 15313 "rpc worker-1531" 0x00007fa5f4362ad3 in ?? ()
106 LWP 15314 "rpc worker-1531" 0x00007fa5f4362ad3 in ?? ()
107 LWP 15315 "rpc worker-1531" 0x00007fa5f4362ad3 in ?? ()
108 LWP 15316 "rpc worker-1531" 0x00007fa5f4362ad3 in ?? ()
109 LWP 15317 "rpc worker-1531" 0x00007fa5f4362ad3 in ?? ()
110 LWP 15318 "rpc worker-1531" 0x00007fa5f4362ad3 in ?? ()
111 LWP 15319 "rpc worker-1531" 0x00007fa5f4362ad3 in ?? ()
112 LWP 15320 "rpc worker-1532" 0x00007fa5f4362ad3 in ?? ()
113 LWP 15321 "rpc worker-1532" 0x00007fa5f4362ad3 in ?? ()
114 LWP 15322 "rpc worker-1532" 0x00007fa5f4362ad3 in ?? ()
115 LWP 15323 "rpc worker-1532" 0x00007fa5f4362ad3 in ?? ()
116 LWP 15324 "rpc worker-1532" 0x00007fa5f4362ad3 in ?? ()
117 LWP 15325 "diag-logger-153" 0x00007fa5f4362fb9 in ?? ()
118 LWP 15326 "result-tracker-" 0x00007fa5f4362fb9 in ?? ()
119 LWP 15327 "excess-log-dele" 0x00007fa5f4362fb9 in ?? ()
120 LWP 15328 "acceptor-15328" 0x00007fa5ed19c0c7 in ?? ()
121 LWP 15329 "heartbeat-15329" 0x00007fa5f4362fb9 in ?? ()
122 LWP 15334 "maintenance_sch" 0x00007fa5f4362fb9 in ?? ()
123 LWP 17283 "raft [worker]-1" 0x00007fa5f4362fb9 in ?? ()
124 LWP 17289 "raft [worker]-1" 0x00007fa5f4362fb9 in ?? ()
Thread 124 (LWP 17289):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000000d in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00007fa5a37a4b24 in ?? ()
#5 0x00007fa5a37a47b0 in ?? ()
#6 0x000000000000001b in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fa5a37a47d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 123 (LWP 17283):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000005a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fa5a245ab20 in ?? ()
#5 0x00007fa5a245a7b0 in ?? ()
#6 0x00000000000000b4 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 15334):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000002af in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x0000616000017b84 in ?? ()
#5 0x00007fa5a8cbf550 in ?? ()
#6 0x000000000000055f in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fa5a8cbf570 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007fa5a8cbf5f0 in ?? ()
#12 0x0000000000997b05 in __sanitizer::theDepot ()
#13 0x0000000000000000 in ?? ()
Thread 121 (LWP 15329):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000000af in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x0000613000018b44 in ?? ()
#5 0x00007fa5a9cef5b0 in ?? ()
#6 0x000000000000015f in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fa5a9cef5d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0xffffffff0000006e in ?? ()
#12 0x00000ff4b539decc in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 120 (LWP 15328):
#0 0x00007fa5ed19c0c7 in ?? ()
#1 0x00006140000316a8 in ?? ()
#2 0x00007fa5aa516d50 in ?? ()
#3 0x00007fa5aa516d80 in ?? ()
#4 0x00007fa5aa516e80 in ?? ()
#5 0x00007fa5aa516d70 in ?? ()
#6 0x00007fa5aa516de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x0000000000894fab in __sanitizer::theDepot ()
#9 0x00007fa5f0fb6da0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 15327):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x00007fa5aad2ff00 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffe9d468990 in ?? ()
#5 0x00007fa5aad2fef0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 15326):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x00007fa5ab5490c0 in ?? ()
#2 0x00000000000000ab in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000861f0 in ?? ()
#5 0x00007fa5ab5490b0 in ?? ()
#6 0x0000000000000156 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 15325):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 15324):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x000000000000002b in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002ffffc in ?? ()
#4 0x00007fa5ac578e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5ac578e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002fffb0 in ?? ()
#9 0x00007fa5f4362770 in ?? ()
#10 0x00007fa5ac578e70 in ?? ()
#11 0x00000ff4b58af1d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 15323):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000015 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002fd00c in ?? ()
#4 0x00007fa5acd90e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5acd90e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002fcfc0 in ?? ()
#9 0x00007fa5f4362770 in ?? ()
#10 0x00007fa5acd90e70 in ?? ()
#11 0x00000ff4b59b21d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 15322):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 15321):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 15320):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 15319):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 15318):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 15317):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 15316):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 15315):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 15314):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 15313):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 15312):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 15311):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 15310):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 15309):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 15308):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 15307):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 15306):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 15305):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 15304):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 15303):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 15302):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 15301):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 15300):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 15299):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 15298):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 15297):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 15296):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 15295):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 15294):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 15293):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 15292):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 15291):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 15290):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 15289):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 15288):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 15287):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 15285):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 15284):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 15283):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000c88 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001e6848 in ?? ()
#4 0x00007fa5c0949e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5c0949e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 15281):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x00000000000003af in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001e004c in ?? ()
#4 0x00007fa5c1161e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5c1161e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001e0000 in ?? ()
#9 0x00007fa5f4362770 in ?? ()
#10 0x00007fa5c1161e70 in ?? ()
#11 0x00000ff4b822c3d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 74 (LWP 15280):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000b24 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001d6858 in ?? ()
#4 0x00007fa5c1979e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5c1979e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 73 (LWP 15279):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 15278):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 15277):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 15276):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 15275):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 15274):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 15273):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 15272):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 15271):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 15270):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 15269):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 15268):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 15267):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 15266):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 15265):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 15264):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 15263):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 15262):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000150008 in ?? ()
#4 0x00007fa5cab2ae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5cab2ae70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 15261):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 15260):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 15259):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 15258):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 15257):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 15256):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 15255):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 15254):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 15253):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 15252):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 15251):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 15250):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 15249):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 15248):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 15247):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 15246):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 15245):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 15244):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 15243):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 15242):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x000000000000181d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000c67cc in ?? ()
#4 0x00007fa5d44f3e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5d44f3e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000c6780 in ?? ()
#9 0x00007fa5f4362770 in ?? ()
#10 0x00007fa5d44f3e70 in ?? ()
#11 0x00000ff4ba89e7d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 15241):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000001ce8 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000b67d8 in ?? ()
#4 0x00007fa5d4d0be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5d4d0be70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 34 (LWP 15240):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000001f48 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000affd8 in ?? ()
#4 0x00007fa5d5523e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5d5523e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 33 (LWP 15239):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000001a8a in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000acfe8 in ?? ()
#4 0x00007fa5d5d3be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5d5d3be70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 32 (LWP 15238):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000001b41 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007fa5d6553e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5d6553e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007fa5f4362770 in ?? ()
#10 0x00007fa5d6553e70 in ?? ()
#11 0x00000ff4bacaa7d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 31 (LWP 15237):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000a37 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000967fc in ?? ()
#4 0x00007fa5d6d6be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5d6d6be70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000967b0 in ?? ()
#9 0x00007fa5f4362770 in ?? ()
#10 0x00007fa5d6d6be70 in ?? ()
#11 0x00000ff4badad7d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 30 (LWP 15236):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000947 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00008fffc in ?? ()
#4 0x00007fa5d7583e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5d7583e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00008ffb0 in ?? ()
#9 0x00007fa5f4362770 in ?? ()
#10 0x00007fa5d7583e70 in ?? ()
#11 0x00000ff4baeb07d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 29 (LWP 15235):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 15234):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 15233):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 15232):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 15231):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 15230):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 15229):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 15228):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 15227):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 15226):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 15225):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 15224):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 15223):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 15222):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x00007fa5de704c80 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019ae0 in ?? ()
#5 0x00007fa5de704c70 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 15221):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x00007fa5def2c1c0 in ?? ()
#2 0x0000000000000022 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000cc98 in ?? ()
#5 0x00007fa5def2c210 in ?? ()
#6 0x0000000000000044 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 15219):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x00007fa5dff891c0 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000f9a8 in ?? ()
#5 0x00007fa5dff891b0 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 15217):
#0 0x00007fa5f4362ad3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x00007fa5e07b0b20 in ?? ()
#4 0x00007fa5e07b0830 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa5e07b0850 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 12 (LWP 15216):
#0 0x00007fa5ed19aa47 in ?? ()
#1 0x00007fa5e0fb22e0 in ?? ()
#2 0x0000619000059180 in ?? ()
#3 0x00007fa5e0fb22d0 in ?? ()
#4 0x00007fa5e0fb24e0 in ?? ()
#5 0x00007fa5e0fb2320 in ?? ()
#6 0x0000614000031a98 in ?? ()
#7 0x00007fa5e0fb23a0 in ?? ()
#8 0x00007fa5ef7e1de0 in ?? ()
#9 0x3faffc2f4b088000 in ?? ()
#10 0x0000619000059180 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059180 in ?? ()
#13 0x421780bb34000000 in ?? ()
#14 0x00007fa500000000 in ?? ()
#15 0x41d98e3f99a4af49 in ?? ()
#16 0x00000ff53c1ee474 in ?? ()
#17 0x00007fa5e0fb2380 in ?? ()
#18 0x00007fa5ef7e549e in ?? ()
#19 0x00007fa5e0fb2350 in ?? ()
#20 0x3faffc2f4b088000 in ?? ()
#21 0x00000000e0fb23a0 in ?? ()
#22 0x0000619000059180 in ?? ()
#23 0x0000614000031a98 in ?? ()
#24 0x3faffc2f4b088000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 15215):
#0 0x00007fa5ed19aa47 in ?? ()
#1 0x00007fa5e17b32e0 in ?? ()
#2 0x0000619000058c80 in ?? ()
#3 0x00007fa5e17b32d0 in ?? ()
#4 0x00007fa5e17b34e0 in ?? ()
#5 0x00007fa5e17b3320 in ?? ()
#6 0x0000614000031898 in ?? ()
#7 0x00007fa5e17b33a0 in ?? ()
#8 0x00007fa5ef7e1de0 in ?? ()
#9 0x3fa54050af7e8000 in ?? ()
#10 0x0000619000058c80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058c80 in ?? ()
#13 0x4259d251ab300000 in ?? ()
#14 0x00007fa500000000 in ?? ()
#15 0x41d98e3f99a4af46 in ?? ()
#16 0x00000ff53c2ee674 in ?? ()
#17 0x00007fa5e17b3380 in ?? ()
#18 0x00007fa5ef7e549e in ?? ()
#19 0x00007fa5e17b3350 in ?? ()
#20 0x3fa54050af7e8000 in ?? ()
#21 0x00000000e17b33a0 in ?? ()
#22 0x0000619000058c80 in ?? ()
#23 0x0000614000031898 in ?? ()
#24 0x3fa54050af7e8000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 15214):
#0 0x00007fa5ed19aa47 in ?? ()
#1 0x00007fa5e1fb42e0 in ?? ()
#2 0x0000619000058780 in ?? ()
#3 0x00007fa5e1fb42d0 in ?? ()
#4 0x00007fa5e1fb44e0 in ?? ()
#5 0x00007fa5e1fb4320 in ?? ()
#6 0x0000614000031698 in ?? ()
#7 0x00007fa5e1fb43a0 in ?? ()
#8 0x00007fa5ef7e1de0 in ?? ()
#9 0x3fa61f3ec4a30000 in ?? ()
#10 0x0000619000058780 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058780 in ?? ()
#13 0x4211d44908000000 in ?? ()
#14 0x00007fa500000000 in ?? ()
#15 0x41d98e3f99a4af46 in ?? ()
#16 0x00000ff53c3ee874 in ?? ()
#17 0x00007fa5e1fb4380 in ?? ()
#18 0x00007fa5ef7e549e in ?? ()
#19 0x00007fa5e1fb4350 in ?? ()
#20 0x3fa61f3ec4a30000 in ?? ()
#21 0x00000000e1fb43a0 in ?? ()
#22 0x0000619000058780 in ?? ()
#23 0x0000614000031698 in ?? ()
#24 0x3fa61f3ec4a30000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 15211):
#0 0x00007fa5ed19aa47 in ?? ()
#1 0x00007fa5e3fad2e0 in ?? ()
#2 0x0000619000058280 in ?? ()
#3 0x00007fa5e3fad2d0 in ?? ()
#4 0x00007fa5e3fad4e0 in ?? ()
#5 0x00007fa5e3fad320 in ?? ()
#6 0x0000614000031498 in ?? ()
#7 0x00007fa5e3fad3a0 in ?? ()
#8 0x00007fa5ef7e1de0 in ?? ()
#9 0x3facca001b3c0000 in ?? ()
#10 0x0000619000058280 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058280 in ?? ()
#13 0x4256abc75d500000 in ?? ()
#14 0x00007fa500000000 in ?? ()
#15 0x41d98e3f99a4af45 in ?? ()
#16 0x00000ff53c7eda74 in ?? ()
#17 0x00007fa5e3fad380 in ?? ()
#18 0x00007fa5ef7e549e in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 15208):
#0 0x00007fa5ed18dcb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fa5e59b7788 in ?? ()
#3 0x0000602000015cb0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x000000000088cbf1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 15207):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x00007fa5e51b6080 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000c2c0 in ?? ()
#5 0x00007fa5e51b6070 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 15206):
#0 0x00007fa5f43669e2 in ?? ()
#1 0x00007fa5e49b3b60 in ?? ()
#2 0x00007fa5e49b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fa5e49b3bb0 in ?? ()
#6 0x0000000001760060 in ?? ()
#7 0x00007fa5e49b3c10 in ?? ()
#8 0x0000611000087690 in ?? ()
#9 0x00007fa5e49b3c00 in ?? ()
#10 0x00000000008870e7 in __sanitizer::theDepot ()
#11 0x00007fa5f9cc638c in ?? ()
#12 0x00007fa5f9cb5c19 in ?? ()
#13 0x00007fa5f9cba206 in ?? ()
#14 0x00007fa5f9cbec30 in ?? ()
#15 0x00007fa5f9cbea59 in ?? ()
#16 0x0000000000a60bcd in __sanitizer::theDepot ()
#17 0x00007fa5f0fd701e in ?? ()
#18 0x00007fa5f435c6db in ?? ()
#19 0x00000ff53c92e67c in ?? ()
#20 0x00007fa5e49b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 15200):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x00007fa5e69b8c40 in ?? ()
#2 0x000000000000035f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000d1a8 in ?? ()
#5 0x00007fa5e69b8c30 in ?? ()
#6 0x00000000000006be in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 15199):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000171b8 in ?? ()
#5 0x00007fa5e71ba410 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 3 (LWP 15198):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000017038 in ?? ()
#5 0x00007fa5e79bb410 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (LWP 15197):
#0 0x00007fa5f4362fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fa5e81bc410 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 15195):
#0 0x00007fa5f4366d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20240506 16:19:11.201109 12982 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID cea103e071de48bbbad5e030eca24b4a and pid 15054
************************ BEGIN STACKS **************************
[New LWP 15055]
[New LWP 15056]
[New LWP 15057]
[New LWP 15058]
[New LWP 15065]
[New LWP 15066]
[New LWP 15067]
[New LWP 15070]
[New LWP 15071]
[New LWP 15072]
[New LWP 15073]
[New LWP 15074]
[New LWP 15075]
[New LWP 15076]
[New LWP 15077]
[New LWP 15078]
[New LWP 15079]
[New LWP 15080]
[New LWP 15081]
[New LWP 15082]
[New LWP 15083]
[New LWP 15084]
[New LWP 15085]
[New LWP 15086]
[New LWP 15087]
[New LWP 15088]
[New LWP 15089]
[New LWP 15090]
[New LWP 15091]
[New LWP 15092]
[New LWP 15093]
[New LWP 15094]
[New LWP 15095]
[New LWP 15096]
[New LWP 15097]
[New LWP 15098]
[New LWP 15099]
[New LWP 15100]
[New LWP 15101]
[New LWP 15102]
[New LWP 15103]
[New LWP 15104]
[New LWP 15105]
[New LWP 15106]
[New LWP 15107]
[New LWP 15108]
[New LWP 15109]
[New LWP 15110]
[New LWP 15111]
[New LWP 15112]
[New LWP 15113]
[New LWP 15114]
[New LWP 15115]
[New LWP 15116]
[New LWP 15117]
[New LWP 15118]
[New LWP 15119]
[New LWP 15120]
[New LWP 15121]
[New LWP 15122]
[New LWP 15123]
[New LWP 15124]
[New LWP 15125]
[New LWP 15126]
[New LWP 15127]
[New LWP 15128]
[New LWP 15129]
[New LWP 15130]
[New LWP 15131]
[New LWP 15132]
[New LWP 15133]
[New LWP 15134]
[New LWP 15135]
[New LWP 15136]
[New LWP 15137]
[New LWP 15138]
[New LWP 15139]
[New LWP 15140]
[New LWP 15141]
[New LWP 15142]
[New LWP 15143]
[New LWP 15144]
[New LWP 15145]
[New LWP 15146]
[New LWP 15147]
[New LWP 15148]
[New LWP 15149]
[New LWP 15150]
[New LWP 15151]
[New LWP 15152]
[New LWP 15153]
[New LWP 15154]
[New LWP 15155]
[New LWP 15156]
[New LWP 15157]
[New LWP 15160]
[New LWP 15161]
[New LWP 15162]
[New LWP 15163]
[New LWP 15164]
[New LWP 15165]
[New LWP 15166]
[New LWP 15167]
[New LWP 15168]
[New LWP 15169]
[New LWP 15170]
[New LWP 15171]
[New LWP 15172]
[New LWP 15173]
[New LWP 15174]
[New LWP 15175]
[New LWP 15176]
[New LWP 15177]
[New LWP 15178]
[New LWP 15179]
[New LWP 15180]
[New LWP 15181]
[New LWP 15182]
[New LWP 15183]
[New LWP 15184]
[New LWP 15185]
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
0x00007f60a162fd50 in ?? ()
Id Target Id Frame
* 1 LWP 15054 "kudu" 0x00007f60a162fd50 in ?? ()
2 LWP 15055 "kudu" 0x00007f60a162bfb9 in ?? ()
3 LWP 15056 "kudu" 0x00007f60a162bfb9 in ?? ()
4 LWP 15057 "kudu" 0x00007f60a162bfb9 in ?? ()
5 LWP 15058 "kernel-watcher-" 0x00007f60a162bfb9 in ?? ()
6 LWP 15065 "ntp client-1506" 0x00007f60a162f9e2 in ?? ()
7 LWP 15066 "file cache-evic" 0x00007f60a162bfb9 in ?? ()
8 LWP 15067 "sq_acceptor" 0x00007f609a456cb9 in ?? ()
9 LWP 15070 "rpc reactor-150" 0x00007f609a463a47 in ?? ()
10 LWP 15071 "rpc reactor-150" 0x00007f609a463a47 in ?? ()
11 LWP 15072 "rpc reactor-150" 0x00007f609a463a47 in ?? ()
12 LWP 15073 "rpc reactor-150" 0x00007f609a463a47 in ?? ()
13 LWP 15074 "MaintenanceMgr " 0x00007f60a162bad3 in ?? ()
14 LWP 15075 "txn-status-mana" 0x00007f60a162bfb9 in ?? ()
15 LWP 15076 "collect_and_rem" 0x00007f60a162bfb9 in ?? ()
16 LWP 15077 "tc-session-exp-" 0x00007f60a162bfb9 in ?? ()
17 LWP 15078 "rpc worker-1507" 0x00007f60a162bad3 in ?? ()
18 LWP 15079 "rpc worker-1507" 0x00007f60a162bad3 in ?? ()
19 LWP 15080 "rpc worker-1508" 0x00007f60a162bad3 in ?? ()
20 LWP 15081 "rpc worker-1508" 0x00007f60a162bad3 in ?? ()
21 LWP 15082 "rpc worker-1508" 0x00007f60a162bad3 in ?? ()
22 LWP 15083 "rpc worker-1508" 0x00007f60a162bad3 in ?? ()
23 LWP 15084 "rpc worker-1508" 0x00007f60a162bad3 in ?? ()
24 LWP 15085 "rpc worker-1508" 0x00007f60a162bad3 in ?? ()
25 LWP 15086 "rpc worker-1508" 0x00007f60a162bad3 in ?? ()
26 LWP 15087 "rpc worker-1508" 0x00007f60a162bad3 in ?? ()
27 LWP 15088 "rpc worker-1508" 0x00007f60a162bad3 in ?? ()
28 LWP 15089 "rpc worker-1508" 0x00007f60a162bad3 in ?? ()
29 LWP 15090 "rpc worker-1509" 0x00007f60a162bad3 in ?? ()
30 LWP 15091 "rpc worker-1509" 0x00007f60a162bad3 in ?? ()
31 LWP 15092 "rpc worker-1509" 0x00007f60a162bad3 in ?? ()
32 LWP 15093 "rpc worker-1509" 0x00007f60a162bad3 in ?? ()
33 LWP 15094 "rpc worker-1509" 0x00007f60a162bad3 in ?? ()
34 LWP 15095 "rpc worker-1509" 0x00007f60a162bad3 in ?? ()
35 LWP 15096 "rpc worker-1509" 0x00007f60a162bad3 in ?? ()
36 LWP 15097 "rpc worker-1509" 0x00007f60a162bad3 in ?? ()
37 LWP 15098 "rpc worker-1509" 0x00007f60a162bad3 in ?? ()
38 LWP 15099 "rpc worker-1509" 0x00007f60a162bad3 in ?? ()
39 LWP 15100 "rpc worker-1510" 0x00007f60a162bad3 in ?? ()
40 LWP 15101 "rpc worker-1510" 0x00007f60a162bad3 in ?? ()
41 LWP 15102 "rpc worker-1510" 0x00007f60a162bad3 in ?? ()
42 LWP 15103 "rpc worker-1510" 0x00007f60a162bad3 in ?? ()
43 LWP 15104 "rpc worker-1510" 0x00007f60a162bad3 in ?? ()
44 LWP 15105 "rpc worker-1510" 0x00007f60a162bad3 in ?? ()
45 LWP 15106 "rpc worker-1510" 0x00007f60a162bad3 in ?? ()
46 LWP 15107 "rpc worker-1510" 0x00007f60a162bad3 in ?? ()
47 LWP 15108 "rpc worker-1510" 0x00007f60a162bad3 in ?? ()
48 LWP 15109 "rpc worker-1510" 0x00007f60a162bad3 in ?? ()
49 LWP 15110 "rpc worker-1511" 0x00007f60a162bad3 in ?? ()
50 LWP 15111 "rpc worker-1511" 0x00007f60a162bad3 in ?? ()
51 LWP 15112 "rpc worker-1511" 0x00007f60a162bad3 in ?? ()
52 LWP 15113 "rpc worker-1511" 0x00007f60a162bad3 in ?? ()
53 LWP 15114 "rpc worker-1511" 0x00007f60a162bad3 in ?? ()
54 LWP 15115 "rpc worker-1511" 0x00007f60a162bad3 in ?? ()
55 LWP 15116 "rpc worker-1511" 0x00007f60a162bad3 in ?? ()
56 LWP 15117 "rpc worker-1511" 0x00007f60a162bad3 in ?? ()
57 LWP 15118 "rpc worker-1511" 0x00007f60a162bad3 in ?? ()
58 LWP 15119 "rpc worker-1511" 0x00007f60a162bad3 in ?? ()
59 LWP 15120 "rpc worker-1512" 0x00007f60a162bad3 in ?? ()
60 LWP 15121 "rpc worker-1512" 0x00007f60a162bad3 in ?? ()
61 LWP 15122 "rpc worker-1512" 0x00007f60a162bad3 in ?? ()
62 LWP 15123 "rpc worker-1512" 0x00007f60a162bad3 in ?? ()
63 LWP 15124 "rpc worker-1512" 0x00007f60a162bad3 in ?? ()
64 LWP 15125 "rpc worker-1512" 0x00007f60a162bad3 in ?? ()
65 LWP 15126 "rpc worker-1512" 0x00007f60a162bad3 in ?? ()
66 LWP 15127 "rpc worker-1512" 0x00007f60a162bad3 in ?? ()
67 LWP 15128 "rpc worker-1512" 0x00007f60a162bad3 in ?? ()
68 LWP 15129 "rpc worker-1512" 0x00007f60a162bad3 in ?? ()
69 LWP 15130 "rpc worker-1513" 0x00007f60a162bad3 in ?? ()
70 LWP 15131 "rpc worker-1513" 0x00007f60a162bad3 in ?? ()
71 LWP 15132 "rpc worker-1513" 0x00007f60a162bad3 in ?? ()
72 LWP 15133 "rpc worker-1513" 0x00007f60a162bad3 in ?? ()
73 LWP 15134 "rpc worker-1513" 0x00007f60a162bad3 in ?? ()
74 LWP 15135 "rpc worker-1513" 0x00007f60a162bad3 in ?? ()
75 LWP 15136 "rpc worker-1513" 0x00007f60a162bad3 in ?? ()
76 LWP 15137 "rpc worker-1513" 0x00007f60a162bad3 in ?? ()
77 LWP 15138 "rpc worker-1513" 0x00007f60a162bad3 in ?? ()
78 LWP 15139 "rpc worker-1513" 0x00007f60a162bad3 in ?? ()
79 LWP 15140 "rpc worker-1514" 0x00007f60a162bad3 in ?? ()
80 LWP 15141 "rpc worker-1514" 0x00007f60a162bad3 in ?? ()
81 LWP 15142 "rpc worker-1514" 0x00007f60a162bad3 in ?? ()
82 LWP 15143 "rpc worker-1514" 0x00007f60a162bad3 in ?? ()
83 LWP 15144 "rpc worker-1514" 0x00007f60a162bad3 in ?? ()
84 LWP 15145 "rpc worker-1514" 0x00007f60a162bad3 in ?? ()
85 LWP 15146 "rpc worker-1514" 0x00007f60a162bad3 in ?? ()
86 LWP 15147 "rpc worker-1514" 0x00007f60a162bad3 in ?? ()
87 LWP 15148 "rpc worker-1514" 0x00007f60a162bad3 in ?? ()
88 LWP 15149 "rpc worker-1514" 0x00007f60a162bad3 in ?? ()
89 LWP 15150 "rpc worker-1515" 0x00007f60a162bad3 in ?? ()
90 LWP 15151 "rpc worker-1515" 0x00007f60a162bad3 in ?? ()
91 LWP 15152 "rpc worker-1515" 0x00007f60a162bad3 in ?? ()
92 LWP 15153 "rpc worker-1515" 0x00007f60a162bad3 in ?? ()
93 LWP 15154 "rpc worker-1515" 0x00007f60a162bad3 in ?? ()
94 LWP 15155 "rpc worker-1515" 0x00007f60a162bad3 in ?? ()
95 LWP 15156 "rpc worker-1515" 0x00007f60a162bad3 in ?? ()
96 LWP 15157 "rpc worker-1515" 0x00007f60a162bad3 in ?? ()
97 LWP 15160 "rpc worker-1516" 0x00007f60a162bad3 in ?? ()
98 LWP 15161 "rpc worker-1516" 0x00007f60a162bad3 in ?? ()
99 LWP 15162 "rpc worker-1516" 0x00007f60a162bad3 in ?? ()
100 LWP 15163 "rpc worker-1516" 0x00007f60a162bad3 in ?? ()
101 LWP 15164 "rpc worker-1516" 0x00007f60a162bad3 in ?? ()
102 LWP 15165 "rpc worker-1516" 0x00007f60a162bad3 in ?? ()
103 LWP 15166 "rpc worker-1516" 0x00007f60a162bad3 in ?? ()
104 LWP 15167 "rpc worker-1516" 0x00007f60a162bad3 in ?? ()
105 LWP 15168 "rpc worker-1516" 0x00007f60a162bad3 in ?? ()
106 LWP 15169 "rpc worker-1516" 0x00007f60a162bad3 in ?? ()
107 LWP 15170 "rpc worker-1517" 0x00007f60a162bad3 in ?? ()
108 LWP 15171 "rpc worker-1517" 0x00007f60a162bad3 in ?? ()
109 LWP 15172 "rpc worker-1517" 0x00007f60a162bad3 in ?? ()
110 LWP 15173 "rpc worker-1517" 0x00007f60a162bad3 in ?? ()
111 LWP 15174 "rpc worker-1517" 0x00007f60a162bad3 in ?? ()
112 LWP 15175 "rpc worker-1517" 0x00007f60a162bad3 in ?? ()
113 LWP 15176 "rpc worker-1517" 0x00007f60a162bad3 in ?? ()
114 LWP 15177 "rpc worker-1517" 0x00007f60a162bad3 in ?? ()
115 LWP 15178 "rpc worker-1517" 0x00007f60a162bad3 in ?? ()
116 LWP 15179 "rpc worker-1517" 0x00007f60a162bad3 in ?? ()
117 LWP 15180 "diag-logger-151" 0x00007f60a162bfb9 in ?? ()
118 LWP 15181 "result-tracker-" 0x00007f60a162bfb9 in ?? ()
119 LWP 15182 "excess-log-dele" 0x00007f60a162bfb9 in ?? ()
120 LWP 15183 "acceptor-15183" 0x00007f609a4650c7 in ?? ()
121 LWP 15184 "heartbeat-15184" 0x00007f60a162bfb9 in ?? ()
122 LWP 15185 "maintenance_sch" 0x00007f60a162bfb9 in ?? ()
Thread 122 (LWP 15185):
#0 0x00007f60a162bfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000002bc in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017b80 in ?? ()
#5 0x00007f6056ff8550 in ?? ()
#6 0x0000000000000578 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 15184):
#0 0x00007f60a162bfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000000ae in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000018b40 in ?? ()
#5 0x00007f60578105b0 in ?? ()
#6 0x000000000000015c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 15183):
#0 0x00007f609a4650c7 in ?? ()
#1 0x0000614000031ea8 in ?? ()
#2 0x00007f6058037d50 in ?? ()
#3 0x00007f6058037d80 in ?? ()
#4 0x00007f6058037e80 in ?? ()
#5 0x00007f6058037d70 in ?? ()
#6 0x00007f6058037de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x0000000000894fab in __sanitizer::theDepot ()
#9 0x00007f609e27fda0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 15182):
#0 0x00007f60a162bfb9 in ?? ()
#1 0x00007f6058851f00 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffe4b161b70 in ?? ()
#5 0x00007f6058851ef0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 15181):
#0 0x00007f60a162bfb9 in ?? ()
#1 0x00007f605906a0c0 in ?? ()
#2 0x00000000000000af in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000861f0 in ?? ()
#5 0x00007f605906a0b0 in ?? ()
#6 0x000000000000015e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 15180):
#0 0x00007f60a162bfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 15179):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007f605a099e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f605a099e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 15178):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000028 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002a6858 in ?? ()
#4 0x00007f605a8b1e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f605a8b1e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 15177):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 15176):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 15175):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 15174):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 15173):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 15172):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 15171):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 15170):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 15169):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 15168):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 15167):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 15166):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 15165):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 15164):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 15163):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 15162):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 15161):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 15160):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 15157):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 15156):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 15155):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 15154):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 15153):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 15152):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 15151):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 15150):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 15149):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 15148):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 15147):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 15146):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 15145):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 15144):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 15143):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 15142):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 15141):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 15140):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 15139):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 15138):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 15137):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007f606e45ae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f606e45ae70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 15136):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 15135):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 15134):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 15133):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 15132):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 15131):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 15130):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 15129):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 15128):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 15127):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 15126):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 15125):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 15124):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 15123):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 15122):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 15121):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 15120):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 15119):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 15118):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 15117):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007f6077e23e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f6077e23e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 15116):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 15115):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 15114):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 15113):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 15112):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 15111):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 15110):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 15109):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 15108):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 15107):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 15106):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 15105):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 15104):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 15103):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 15102):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 15101):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 15100):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 15099):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 15098):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 15097):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007f6082004e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f6082004e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 15096):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 15095):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 15094):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 15093):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 15092):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 15091):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 15090):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 15089):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 15088):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 15087):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 15086):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 15085):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 15084):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 15083):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 15082):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 15081):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 15080):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 15079):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 15078):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 15077):
#0 0x00007f60a162bfb9 in ?? ()
#1 0x00007f608c217c80 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006130000193e0 in ?? ()
#5 0x00007f608c217c70 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 15076):
#0 0x00007f60a162bfb9 in ?? ()
#1 0x00007f608ca3f1c0 in ?? ()
#2 0x0000000000000023 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000cc98 in ?? ()
#5 0x00007f608ca3f210 in ?? ()
#6 0x0000000000000046 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 15075):
#0 0x00007f60a162bfb9 in ?? ()
#1 0x00007f608d2551c0 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000f9a8 in ?? ()
#5 0x00007f608d2551b0 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 15074):
#0 0x00007f60a162bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 15073):
#0 0x00007f609a463a47 in ?? ()
#1 0x00007f608e2942e0 in ?? ()
#2 0x0000619000059180 in ?? ()
#3 0x00007f608e2942d0 in ?? ()
#4 0x00007f608e2944e0 in ?? ()
#5 0x00007f608e294320 in ?? ()
#6 0x0000614000031e98 in ?? ()
#7 0x00007f608e2943a0 in ?? ()
#8 0x00007f609caaade0 in ?? ()
#9 0x3fb95a3e9c2c8000 in ?? ()
#10 0x0000619000059180 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059180 in ?? ()
#13 0x421e09b6e5000000 in ?? ()
#14 0x00007f6000000000 in ?? ()
#15 0x41d98e3f99a4af7e in ?? ()
#16 0x00000fec91c4a874 in ?? ()
#17 0x00007f608e294380 in ?? ()
#18 0x00007f609caae49e in ?? ()
#19 0x00007f608e294350 in ?? ()
#20 0x3fb95a3e9c2c8000 in ?? ()
#21 0x000000008e2943a0 in ?? ()
#22 0x0000619000059180 in ?? ()
#23 0x0000614000031e98 in ?? ()
#24 0x3fb95a3e9c2c8000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 15072):
#0 0x00007f609a463a47 in ?? ()
#1 0x00007f608eaab2e0 in ?? ()
#2 0x0000619000058c80 in ?? ()
#3 0x00007f608eaab2d0 in ?? ()
#4 0x00007f608eaab4e0 in ?? ()
#5 0x00007f608eaab320 in ?? ()
#6 0x0000614000031c98 in ?? ()
#7 0x00007f608eaab3a0 in ?? ()
#8 0x00007f609caaade0 in ?? ()
#9 0x3fb958ce73698000 in ?? ()
#10 0x0000619000058c80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058c80 in ?? ()
#13 0x42255be446000000 in ?? ()
#14 0x00007f6000000000 in ?? ()
#15 0x41d98e3f99a4af44 in ?? ()
#16 0x00000fec91d4d674 in ?? ()
#17 0x00007f608eaab380 in ?? ()
#18 0x00007f609caae49e in ?? ()
#19 0x00007f608eaab350 in ?? ()
#20 0x3fb958ce73698000 in ?? ()
#21 0x000000008eaab3a0 in ?? ()
#22 0x0000619000058c80 in ?? ()
#23 0x0000614000031c98 in ?? ()
#24 0x3fb958ce73698000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 15071):
#0 0x00007f609a463a47 in ?? ()
#1 0x00007f608f2b42e0 in ?? ()
#2 0x0000619000058780 in ?? ()
#3 0x00007f608f2b42d0 in ?? ()
#4 0x00007f608f2b44e0 in ?? ()
#5 0x00007f608f2b4320 in ?? ()
#6 0x0000614000031a98 in ?? ()
#7 0x00007f608f2b43a0 in ?? ()
#8 0x00007f609caaade0 in ?? ()
#9 0x3fb947a969e70000 in ?? ()
#10 0x0000619000058780 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058780 in ?? ()
#13 0x4214b53cb3000000 in ?? ()
#14 0x00007f6000000000 in ?? ()
#15 0x41d98e3f99a4af47 in ?? ()
#16 0x00000fec91e4e874 in ?? ()
#17 0x00007f608f2b4380 in ?? ()
#18 0x00007f609caae49e in ?? ()
#19 0x00007f608f2b4350 in ?? ()
#20 0x3fb947a969e70000 in ?? ()
#21 0x000000008f2b43a0 in ?? ()
#22 0x0000619000058780 in ?? ()
#23 0x0000614000031a98 in ?? ()
#24 0x3fb947a969e70000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 15070):
#0 0x00007f609a463a47 in ?? ()
#1 0x00007f60912ad2e0 in ?? ()
#2 0x0000619000058280 in ?? ()
#3 0x00007f60912ad2d0 in ?? ()
#4 0x00007f60912ad4e0 in ?? ()
#5 0x00007f60912ad320 in ?? ()
#6 0x0000614000031898 in ?? ()
#7 0x00007f60912ad3a0 in ?? ()
#8 0x00007f609caaade0 in ?? ()
#9 0x3fb96ccee8508000 in ?? ()
#10 0x0000619000058280 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058280 in ?? ()
#13 0x42280493a5000000 in ?? ()
#14 0x00007f6000000000 in ?? ()
#15 0x41d98e3f99a4af45 in ?? ()
#16 0x00000fec9224da74 in ?? ()
#17 0x00007f60912ad380 in ?? ()
#18 0x00007f609caae49e in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 15067):
#0 0x00007f609a456cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f6092cb7788 in ?? ()
#3 0x0000602000017cb0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x000000000088cbf1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 15066):
#0 0x00007f60a162bfb9 in ?? ()
#1 0x00007f60924b6080 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000c2c0 in ?? ()
#5 0x00007f60924b6070 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 15065):
#0 0x00007f60a162f9e2 in ?? ()
#1 0x00007f6091cb3b60 in ?? ()
#2 0x00007f6091cb3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f6091cb3bb0 in ?? ()
#6 0x0000000001760060 in ?? ()
#7 0x00007f6091cb3c10 in ?? ()
#8 0x0000611000087690 in ?? ()
#9 0x00007f6091cb3c00 in ?? ()
#10 0x00000000008870e7 in __sanitizer::theDepot ()
#11 0x00007f60a6f8f38c in ?? ()
#12 0x00007f60a6f7ec19 in ?? ()
#13 0x00007f60a6f83206 in ?? ()
#14 0x00007f60a6f87c30 in ?? ()
#15 0x00007f60a6f87a59 in ?? ()
#16 0x0000000000a60bcd in __sanitizer::theDepot ()
#17 0x00007f609e2a001e in ?? ()
#18 0x00007f60a16256db in ?? ()
#19 0x00000fec9238e67c in ?? ()
#20 0x00007f6091cb3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 15058):
#0 0x00007f60a162bfb9 in ?? ()
#1 0x00007f6093cb8c40 in ?? ()
#2 0x0000000000000370 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000d1a8 in ?? ()
#5 0x00007f6093cb8c30 in ?? ()
#6 0x00000000000006e0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 15057):
#0 0x00007f60a162bfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000171b8 in ?? ()
#5 0x00007f60944ba410 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 3 (LWP 15056):
#0 0x00007f60a162bfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000017038 in ?? ()
#5 0x00007f6094cbb410 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (LWP 15055):
#0 0x00007f60a162bfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f60954bc410 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 15054):
#0 0x00007f60a162fd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20240506 16:19:12.039045 12982 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID 405f2b51c2244be3b1a66f2c908bcdd9 and pid 15338
************************ BEGIN STACKS **************************
[New LWP 15342]
[New LWP 15343]
[New LWP 15344]
[New LWP 15345]
[New LWP 15351]
[New LWP 15352]
[New LWP 15353]
[New LWP 15356]
[New LWP 15357]
[New LWP 15358]
[New LWP 15359]
[New LWP 15360]
[New LWP 15361]
[New LWP 15362]
[New LWP 15363]
[New LWP 15364]
[New LWP 15365]
[New LWP 15367]
[New LWP 15368]
[New LWP 15369]
[New LWP 15370]
[New LWP 15371]
[New LWP 15372]
[New LWP 15373]
[New LWP 15374]
[New LWP 15375]
[New LWP 15376]
[New LWP 15377]
[New LWP 15378]
[New LWP 15379]
[New LWP 15380]
[New LWP 15381]
[New LWP 15382]
[New LWP 15383]
[New LWP 15384]
[New LWP 15385]
[New LWP 15386]
[New LWP 15387]
[New LWP 15388]
[New LWP 15389]
[New LWP 15390]
[New LWP 15391]
[New LWP 15392]
[New LWP 15393]
[New LWP 15394]
[New LWP 15395]
[New LWP 15396]
[New LWP 15397]
[New LWP 15398]
[New LWP 15399]
[New LWP 15400]
[New LWP 15401]
[New LWP 15402]
[New LWP 15403]
[New LWP 15404]
[New LWP 15405]
[New LWP 15406]
[New LWP 15407]
[New LWP 15408]
[New LWP 15409]
[New LWP 15410]
[New LWP 15411]
[New LWP 15412]
[New LWP 15413]
[New LWP 15414]
[New LWP 15415]
[New LWP 15416]
[New LWP 15417]
[New LWP 15418]
[New LWP 15419]
[New LWP 15420]
[New LWP 15421]
[New LWP 15422]
[New LWP 15423]
[New LWP 15424]
[New LWP 15425]
[New LWP 15426]
[New LWP 15427]
[New LWP 15428]
[New LWP 15429]
[New LWP 15430]
[New LWP 15431]
[New LWP 15432]
[New LWP 15433]
[New LWP 15434]
[New LWP 15435]
[New LWP 15436]
[New LWP 15437]
[New LWP 15438]
[New LWP 15439]
[New LWP 15440]
[New LWP 15441]
[New LWP 15442]
[New LWP 15443]
[New LWP 15444]
[New LWP 15445]
[New LWP 15446]
[New LWP 15447]
[New LWP 15448]
[New LWP 15449]
[New LWP 15450]
[New LWP 15451]
[New LWP 15452]
[New LWP 15453]
[New LWP 15454]
[New LWP 15455]
[New LWP 15456]
[New LWP 15457]
[New LWP 15458]
[New LWP 15459]
[New LWP 15460]
[New LWP 15461]
[New LWP 15462]
[New LWP 15463]
[New LWP 15464]
[New LWP 15465]
[New LWP 15466]
[New LWP 15467]
[New LWP 15468]
[New LWP 15469]
[New LWP 15472]
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
0x00007f8f0a127d50 in ?? ()
Id Target Id Frame
* 1 LWP 15338 "kudu" 0x00007f8f0a127d50 in ?? ()
2 LWP 15342 "kudu" 0x00007f8f0a123fb9 in ?? ()
3 LWP 15343 "kudu" 0x00007f8f0a123fb9 in ?? ()
4 LWP 15344 "kudu" 0x00007f8f0a123fb9 in ?? ()
5 LWP 15345 "kernel-watcher-" 0x00007f8f0a123fb9 in ?? ()
6 LWP 15351 "ntp client-1535" 0x00007f8f0a1279e2 in ?? ()
7 LWP 15352 "file cache-evic" 0x00007f8f0a123fb9 in ?? ()
8 LWP 15353 "sq_acceptor" 0x00007f8f02f4ecb9 in ?? ()
9 LWP 15356 "rpc reactor-153" 0x00007f8f02f5ba47 in ?? ()
10 LWP 15357 "rpc reactor-153" 0x00007f8f02f5ba47 in ?? ()
11 LWP 15358 "rpc reactor-153" 0x00007f8f02f5ba47 in ?? ()
12 LWP 15359 "rpc reactor-153" 0x00007f8f02f5ba47 in ?? ()
13 LWP 15360 "MaintenanceMgr " 0x00007f8f0a123ad3 in ?? ()
14 LWP 15361 "txn-status-mana" 0x00007f8f0a123fb9 in ?? ()
15 LWP 15362 "collect_and_rem" 0x00007f8f0a123fb9 in ?? ()
16 LWP 15363 "tc-session-exp-" 0x00007f8f0a123fb9 in ?? ()
17 LWP 15364 "rpc worker-1536" 0x00007f8f0a123ad3 in ?? ()
18 LWP 15365 "rpc worker-1536" 0x00007f8f0a123ad3 in ?? ()
19 LWP 15367 "rpc worker-1536" 0x00007f8f0a123ad3 in ?? ()
20 LWP 15368 "rpc worker-1536" 0x00007f8f0a123ad3 in ?? ()
21 LWP 15369 "rpc worker-1536" 0x00007f8f0a123ad3 in ?? ()
22 LWP 15370 "rpc worker-1537" 0x00007f8f0a123ad3 in ?? ()
23 LWP 15371 "rpc worker-1537" 0x00007f8f0a123ad3 in ?? ()
24 LWP 15372 "rpc worker-1537" 0x00007f8f0a123ad3 in ?? ()
25 LWP 15373 "rpc worker-1537" 0x00007f8f0a123ad3 in ?? ()
26 LWP 15374 "rpc worker-1537" 0x00007f8f0a123ad3 in ?? ()
27 LWP 15375 "rpc worker-1537" 0x00007f8f0a123ad3 in ?? ()
28 LWP 15376 "rpc worker-1537" 0x00007f8f0a123ad3 in ?? ()
29 LWP 15377 "rpc worker-1537" 0x00007f8f0a123ad3 in ?? ()
30 LWP 15378 "rpc worker-1537" 0x00007f8f0a123ad3 in ?? ()
31 LWP 15379 "rpc worker-1537" 0x00007f8f0a123ad3 in ?? ()
32 LWP 15380 "rpc worker-1538" 0x00007f8f0a123ad3 in ?? ()
33 LWP 15381 "rpc worker-1538" 0x00007f8f0a123ad3 in ?? ()
34 LWP 15382 "rpc worker-1538" 0x00007f8f0a123ad3 in ?? ()
35 LWP 15383 "rpc worker-1538" 0x00007f8f0a123ad3 in ?? ()
36 LWP 15384 "rpc worker-1538" 0x00007f8f0a123ad3 in ?? ()
37 LWP 15385 "rpc worker-1538" 0x00007f8f0a123ad3 in ?? ()
38 LWP 15386 "rpc worker-1538" 0x00007f8f0a123ad3 in ?? ()
39 LWP 15387 "rpc worker-1538" 0x00007f8f0a123ad3 in ?? ()
40 LWP 15388 "rpc worker-1538" 0x00007f8f0a123ad3 in ?? ()
41 LWP 15389 "rpc worker-1538" 0x00007f8f0a123ad3 in ?? ()
42 LWP 15390 "rpc worker-1539" 0x00007f8f0a123ad3 in ?? ()
43 LWP 15391 "rpc worker-1539" 0x00007f8f0a123ad3 in ?? ()
44 LWP 15392 "rpc worker-1539" 0x00007f8f0a123ad3 in ?? ()
45 LWP 15393 "rpc worker-1539" 0x00007f8f0a123ad3 in ?? ()
46 LWP 15394 "rpc worker-1539" 0x00007f8f0a123ad3 in ?? ()
47 LWP 15395 "rpc worker-1539" 0x00007f8f0a123ad3 in ?? ()
48 LWP 15396 "rpc worker-1539" 0x00007f8f0a123ad3 in ?? ()
49 LWP 15397 "rpc worker-1539" 0x00007f8f0a123ad3 in ?? ()
50 LWP 15398 "rpc worker-1539" 0x00007f8f0a123ad3 in ?? ()
51 LWP 15399 "rpc worker-1539" 0x00007f8f0a123ad3 in ?? ()
52 LWP 15400 "rpc worker-1540" 0x00007f8f0a123ad3 in ?? ()
53 LWP 15401 "rpc worker-1540" 0x00007f8f0a123ad3 in ?? ()
54 LWP 15402 "rpc worker-1540" 0x00007f8f0a123ad3 in ?? ()
55 LWP 15403 "rpc worker-1540" 0x00007f8f0a123ad3 in ?? ()
56 LWP 15404 "rpc worker-1540" 0x00007f8f0a123ad3 in ?? ()
57 LWP 15405 "rpc worker-1540" 0x00007f8f0a123ad3 in ?? ()
58 LWP 15406 "rpc worker-1540" 0x00007f8f0a123ad3 in ?? ()
59 LWP 15407 "rpc worker-1540" 0x00007f8f0a123ad3 in ?? ()
60 LWP 15408 "rpc worker-1540" 0x00007f8f0a123ad3 in ?? ()
61 LWP 15409 "rpc worker-1540" 0x00007f8f0a123ad3 in ?? ()
62 LWP 15410 "rpc worker-1541" 0x00007f8f0a123ad3 in ?? ()
63 LWP 15411 "rpc worker-1541" 0x00007f8f0a123ad3 in ?? ()
64 LWP 15412 "rpc worker-1541" 0x00007f8f0a123ad3 in ?? ()
65 LWP 15413 "rpc worker-1541" 0x00007f8f0a123ad3 in ?? ()
66 LWP 15414 "rpc worker-1541" 0x00007f8f0a123ad3 in ?? ()
67 LWP 15415 "rpc worker-1541" 0x00007f8f0a123ad3 in ?? ()
68 LWP 15416 "rpc worker-1541" 0x00007f8f0a123ad3 in ?? ()
69 LWP 15417 "rpc worker-1541" 0x00007f8f0a123ad3 in ?? ()
70 LWP 15418 "rpc worker-1541" 0x00007f8f0a123ad3 in ?? ()
71 LWP 15419 "rpc worker-1541" 0x00007f8f0a123ad3 in ?? ()
72 LWP 15420 "rpc worker-1542" 0x00007f8f0a123ad3 in ?? ()
73 LWP 15421 "rpc worker-1542" 0x00007f8f0a123ad3 in ?? ()
74 LWP 15422 "rpc worker-1542" 0x00007f8f0a123ad3 in ?? ()
75 LWP 15423 "rpc worker-1542" 0x00007f8f0a123ad3 in ?? ()
76 LWP 15424 "rpc worker-1542" 0x00007f8f0a123ad3 in ?? ()
77 LWP 15425 "rpc worker-1542" 0x00007f8f0a123ad3 in ?? ()
78 LWP 15426 "rpc worker-1542" 0x00007f8f0a123ad3 in ?? ()
79 LWP 15427 "rpc worker-1542" 0x00007f8f0a123ad3 in ?? ()
80 LWP 15428 "rpc worker-1542" 0x00007f8f0a123ad3 in ?? ()
81 LWP 15429 "rpc worker-1542" 0x00007f8f0a123ad3 in ?? ()
82 LWP 15430 "rpc worker-1543" 0x00007f8f0a123ad3 in ?? ()
83 LWP 15431 "rpc worker-1543" 0x00007f8f0a123ad3 in ?? ()
84 LWP 15432 "rpc worker-1543" 0x00007f8f0a123ad3 in ?? ()
85 LWP 15433 "rpc worker-1543" 0x00007f8f0a123ad3 in ?? ()
86 LWP 15434 "rpc worker-1543" 0x00007f8f0a123ad3 in ?? ()
87 LWP 15435 "rpc worker-1543" 0x00007f8f0a123ad3 in ?? ()
88 LWP 15436 "rpc worker-1543" 0x00007f8f0a123ad3 in ?? ()
89 LWP 15437 "rpc worker-1543" 0x00007f8f0a123ad3 in ?? ()
90 LWP 15438 "rpc worker-1543" 0x00007f8f0a123ad3 in ?? ()
91 LWP 15439 "rpc worker-1543" 0x00007f8f0a123ad3 in ?? ()
92 LWP 15440 "rpc worker-1544" 0x00007f8f0a123ad3 in ?? ()
93 LWP 15441 "rpc worker-1544" 0x00007f8f0a123ad3 in ?? ()
94 LWP 15442 "rpc worker-1544" 0x00007f8f0a123ad3 in ?? ()
95 LWP 15443 "rpc worker-1544" 0x00007f8f0a123ad3 in ?? ()
96 LWP 15444 "rpc worker-1544" 0x00007f8f0a123ad3 in ?? ()
97 LWP 15445 "rpc worker-1544" 0x00007f8f0a123ad3 in ?? ()
98 LWP 15446 "rpc worker-1544" 0x00007f8f0a123ad3 in ?? ()
99 LWP 15447 "rpc worker-1544" 0x00007f8f0a123ad3 in ?? ()
100 LWP 15448 "rpc worker-1544" 0x00007f8f0a123ad3 in ?? ()
101 LWP 15449 "rpc worker-1544" 0x00007f8f0a123ad3 in ?? ()
102 LWP 15450 "rpc worker-1545" 0x00007f8f0a123ad3 in ?? ()
103 LWP 15451 "rpc worker-1545" 0x00007f8f0a123ad3 in ?? ()
104 LWP 15452 "rpc worker-1545" 0x00007f8f0a123ad3 in ?? ()
105 LWP 15453 "rpc worker-1545" 0x00007f8f0a123ad3 in ?? ()
106 LWP 15454 "rpc worker-1545" 0x00007f8f0a123ad3 in ?? ()
107 LWP 15455 "rpc worker-1545" 0x00007f8f0a123ad3 in ?? ()
108 LWP 15456 "rpc worker-1545" 0x00007f8f0a123ad3 in ?? ()
109 LWP 15457 "rpc worker-1545" 0x00007f8f0a123ad3 in ?? ()
110 LWP 15458 "rpc worker-1545" 0x00007f8f0a123ad3 in ?? ()
111 LWP 15459 "rpc worker-1545" 0x00007f8f0a123ad3 in ?? ()
112 LWP 15460 "rpc worker-1546" 0x00007f8f0a123ad3 in ?? ()
113 LWP 15461 "rpc worker-1546" 0x00007f8f0a123ad3 in ?? ()
114 LWP 15462 "rpc worker-1546" 0x00007f8f0a123ad3 in ?? ()
115 LWP 15463 "rpc worker-1546" 0x00007f8f0a123ad3 in ?? ()
116 LWP 15464 "rpc worker-1546" 0x00007f8f0a123ad3 in ?? ()
117 LWP 15465 "diag-logger-154" 0x00007f8f0a123fb9 in ?? ()
118 LWP 15466 "result-tracker-" 0x00007f8f0a123fb9 in ?? ()
119 LWP 15467 "excess-log-dele" 0x00007f8f0a123fb9 in ?? ()
120 LWP 15468 "acceptor-15468" 0x00007f8f02f5d0c7 in ?? ()
121 LWP 15469 "heartbeat-15469" 0x00007f8f0a123fb9 in ?? ()
122 LWP 15472 "maintenance_sch" 0x00007f8f0a123fb9 in ?? ()
Thread 122 (LWP 15472):
#0 0x00007f8f0a123fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000002a8 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017b80 in ?? ()
#5 0x00007f8ebfaf0550 in ?? ()
#6 0x0000000000000550 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 15469):
#0 0x00007f8f0a123fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000000a9 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000018b40 in ?? ()
#5 0x00007f8ec03085b0 in ?? ()
#6 0x0000000000000152 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 15468):
#0 0x00007f8f02f5d0c7 in ?? ()
#1 0x0000614000031aa8 in ?? ()
#2 0x00007f8ec0b2fd50 in ?? ()
#3 0x00007f8ec0b2fd80 in ?? ()
#4 0x00007f8ec0b2fe80 in ?? ()
#5 0x00007f8ec0b2fd70 in ?? ()
#6 0x00007f8ec0b2fde0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x0000000000894fab in __sanitizer::theDepot ()
#9 0x00007f8f06d77da0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 15467):
#0 0x00007f8f0a123fb9 in ?? ()
#1 0x00007f8ec1349f00 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffd3de89770 in ?? ()
#5 0x00007f8ec1349ef0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 15466):
#0 0x00007f8f0a123fb9 in ?? ()
#1 0x00007f8ec1b620c0 in ?? ()
#2 0x00000000000000aa in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000861f0 in ?? ()
#5 0x00007f8ec1b620b0 in ?? ()
#6 0x0000000000000154 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 15465):
#0 0x00007f8f0a123fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 15464):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000017 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007f8ec2b91e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f8ec2b91e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007f8f0a123770 in ?? ()
#10 0x00007f8ec2b91e70 in ?? ()
#11 0x00000ff1d85723d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 15463):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000029 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007f8ec33a9e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f8ec33a9e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007f8f0a123770 in ?? ()
#10 0x00007f8ec33a9e70 in ?? ()
#11 0x00000ff1d86753d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 15462):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 15461):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 15460):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 15459):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 15458):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 15457):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 15456):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 15455):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 15454):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 15453):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 15452):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 15451):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 15450):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 15449):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 15448):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 15447):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 15446):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 15445):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 15444):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 15443):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 15442):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 15441):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 15440):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 15439):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 15438):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 15437):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 15436):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 15435):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 15434):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 15433):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 15432):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 15431):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 15430):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 15429):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 15428):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 15427):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 15426):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 15425):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 15424):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007f8ed6f52e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f8ed6f52e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 15423):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 15422):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 15421):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 15420):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 15419):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 15418):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 15417):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 15416):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 15415):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 15414):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 15413):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 15412):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 15411):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 15410):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 15409):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 15408):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 15407):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 15406):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 15405):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 15404):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007f8ee091be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f8ee091be70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 15403):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 15402):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 15401):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 15400):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 15399):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 15398):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 15397):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 15396):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 15395):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 15394):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 15393):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 15392):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 15391):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 15390):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 15389):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 15388):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 15387):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 15386):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 15385):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 15384):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007f8eeaafce50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f8eeaafce70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 15383):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 15382):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 15381):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 15380):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 15379):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 15378):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 15377):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 15376):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 15375):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 15374):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 15373):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 15372):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 15371):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 15370):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 15369):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 15368):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 15367):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 15365):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 15364):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 15363):
#0 0x00007f8f0a123fb9 in ?? ()
#1 0x00007f8ef4bc1c80 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006130000193e0 in ?? ()
#5 0x00007f8ef4bc1c70 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 15362):
#0 0x00007f8f0a123fb9 in ?? ()
#1 0x00007f8ef53c31c0 in ?? ()
#2 0x0000000000000022 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000cc98 in ?? ()
#5 0x00007f8ef53c3210 in ?? ()
#6 0x0000000000000044 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 15361):
#0 0x00007f8f0a123fb9 in ?? ()
#1 0x00007f8ef5bc31c0 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000f9a8 in ?? ()
#5 0x00007f8ef5bc31b0 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 15360):
#0 0x00007f8f0a123ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 15359):
#0 0x00007f8f02f5ba47 in ?? ()
#1 0x00007f8ef6bc62e0 in ?? ()
#2 0x0000619000059180 in ?? ()
#3 0x00007f8ef6bc62d0 in ?? ()
#4 0x00007f8ef6bc64e0 in ?? ()
#5 0x00007f8ef6bc6320 in ?? ()
#6 0x0000614000031c98 in ?? ()
#7 0x00007f8ef6bc63a0 in ?? ()
#8 0x00007f8f055a2de0 in ?? ()
#9 0x3fb98026e7718000 in ?? ()
#10 0x0000619000059180 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059180 in ?? ()
#13 0x422746ea2d800000 in ?? ()
#14 0x00007f8e00000000 in ?? ()
#15 0x41d98e3f99a4af45 in ?? ()
#16 0x00000ff25ed70c74 in ?? ()
#17 0x00007f8ef6bc6380 in ?? ()
#18 0x00007f8f055a649e in ?? ()
#19 0x00007f8ef6bc6350 in ?? ()
#20 0x3fb98026e7718000 in ?? ()
#21 0x00000000f6bc63a0 in ?? ()
#22 0x0000619000059180 in ?? ()
#23 0x0000614000031c98 in ?? ()
#24 0x3fb98026e7718000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 15358):
#0 0x00007f8f02f5ba47 in ?? ()
#1 0x00007f8ef73c72e0 in ?? ()
#2 0x0000619000058c80 in ?? ()
#3 0x00007f8ef73c72d0 in ?? ()
#4 0x00007f8ef73c74e0 in ?? ()
#5 0x00007f8ef73c7320 in ?? ()
#6 0x0000614000031a98 in ?? ()
#7 0x00007f8ef73c73a0 in ?? ()
#8 0x00007f8f055a2de0 in ?? ()
#9 0x3fb97df056660000 in ?? ()
#10 0x0000619000058c80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058c80 in ?? ()
#13 0x422b371156000000 in ?? ()
#14 0x00007f8e00000000 in ?? ()
#15 0x41d98e3f99a4af45 in ?? ()
#16 0x00000ff25ee70e74 in ?? ()
#17 0x00007f8ef73c7380 in ?? ()
#18 0x00007f8f055a649e in ?? ()
#19 0x00007f8ef73c7350 in ?? ()
#20 0x3fb97df056660000 in ?? ()
#21 0x00000000f73c73a0 in ?? ()
#22 0x0000619000058c80 in ?? ()
#23 0x0000614000031a98 in ?? ()
#24 0x3fb97df056660000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 15357):
#0 0x00007f8f02f5ba47 in ?? ()
#1 0x00007f8ef97b32e0 in ?? ()
#2 0x0000619000058780 in ?? ()
#3 0x00007f8ef97b32d0 in ?? ()
#4 0x00007f8ef97b34e0 in ?? ()
#5 0x00007f8ef97b3320 in ?? ()
#6 0x0000614000031898 in ?? ()
#7 0x00007f8ef97b33a0 in ?? ()
#8 0x00007f8f055a2de0 in ?? ()
#9 0x3fb9522525c8c000 in ?? ()
#10 0x0000619000058780 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058780 in ?? ()
#13 0x422261c965800000 in ?? ()
#14 0x00007f8e00000000 in ?? ()
#15 0x41d98e3f99a4af45 in ?? ()
#16 0x00000ff25f2ee674 in ?? ()
#17 0x00007f8ef97b3380 in ?? ()
#18 0x00007f8f055a649e in ?? ()
#19 0x00007f8ef97b3350 in ?? ()
#20 0x3fb9522525c8c000 in ?? ()
#21 0x00000000f97b33a0 in ?? ()
#22 0x0000619000058780 in ?? ()
#23 0x0000614000031898 in ?? ()
#24 0x3fb9522525c8c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 15356):
#0 0x00007f8f02f5ba47 in ?? ()
#1 0x00007f8ef7bc82e0 in ?? ()
#2 0x0000619000058280 in ?? ()
#3 0x00007f8ef7bc82d0 in ?? ()
#4 0x00007f8ef7bc84e0 in ?? ()
#5 0x00007f8ef7bc8320 in ?? ()
#6 0x0000614000031698 in ?? ()
#7 0x00007f8ef7bc83a0 in ?? ()
#8 0x00007f8f055a2de0 in ?? ()
#9 0x3fb956123ca60000 in ?? ()
#10 0x0000619000058280 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058280 in ?? ()
#13 0x421c4292c8000000 in ?? ()
#14 0x00007f8e00000000 in ?? ()
#15 0x41d98e3f99a4af49 in ?? ()
#16 0x00000ff25ef71074 in ?? ()
#17 0x00007f8ef7bc8380 in ?? ()
#18 0x00007f8f055a649e in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 15353):
#0 0x00007f8f02f4ecb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f8efb7b7788 in ?? ()
#3 0x0000602000017cb0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x000000000088cbf1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 15352):
#0 0x00007f8f0a123fb9 in ?? ()
#1 0x00007f8efafb6080 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000c2c0 in ?? ()
#5 0x00007f8efafb6070 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 15351):
#0 0x00007f8f0a1279e2 in ?? ()
#1 0x00007f8ef9fb2b60 in ?? ()
#2 0x00007f8ef9fb2c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f8ef9fb2bb0 in ?? ()
#6 0x0000000001760060 in ?? ()
#7 0x00007f8ef9fb2c10 in ?? ()
#8 0x0000611000087690 in ?? ()
#9 0x00007f8ef9fb2c00 in ?? ()
#10 0x00000000008870e7 in __sanitizer::theDepot ()
#11 0x00007f8f0fa8738c in ?? ()
#12 0x00007f8f0fa76c19 in ?? ()
#13 0x00007f8f0fa7b206 in ?? ()
#14 0x00007f8f0fa7fc30 in ?? ()
#15 0x00007f8f0fa7fa59 in ?? ()
#16 0x0000000000a60bcd in __sanitizer::theDepot ()
#17 0x00007f8f06d9801e in ?? ()
#18 0x00007f8f0a11d6db in ?? ()
#19 0x00000ff25f3ee47c in ?? ()
#20 0x00007f8ef9fb2400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 15345):
#0 0x00007f8f0a123fb9 in ?? ()
#1 0x00007f8efc7b8c40 in ?? ()
#2 0x0000000000000357 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000d1a8 in ?? ()
#5 0x00007f8efc7b8c30 in ?? ()
#6 0x00000000000006ae in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 15344):
#0 0x00007f8f0a123fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000171b8 in ?? ()
#5 0x00007f8efcfba410 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 3 (LWP 15343):
#0 0x00007f8f0a123fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000017038 in ?? ()
#5 0x00007f8efd7bb410 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (LWP 15342):
#0 0x00007f8f0a123fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f8efdfbc410 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 15338):
#0 0x00007f8f0a127d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20240506 16:19:12.833483 12982 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 3 with UUID e6d1cddba8c94532b40c0ae9c06740ef and pid 15764
************************ BEGIN STACKS **************************
[New LWP 15767]
[New LWP 15768]
[New LWP 15769]
[New LWP 15770]
[New LWP 15776]
[New LWP 15777]
[New LWP 15778]
[New LWP 15781]
[New LWP 15782]
[New LWP 15783]
[New LWP 15784]
[New LWP 15785]
[New LWP 15786]
[New LWP 15788]
[New LWP 15789]
[New LWP 15790]
[New LWP 15791]
[New LWP 15792]
[New LWP 15793]
[New LWP 15794]
[New LWP 15795]
[New LWP 15797]
[New LWP 15798]
[New LWP 15799]
[New LWP 15800]
[New LWP 15801]
[New LWP 15802]
[New LWP 15803]
[New LWP 15804]
[New LWP 15805]
[New LWP 15806]
[New LWP 15807]
[New LWP 15808]
[New LWP 15809]
[New LWP 15810]
[New LWP 15811]
[New LWP 15812]
[New LWP 15813]
[New LWP 15814]
[New LWP 15815]
[New LWP 15816]
[New LWP 15817]
[New LWP 15818]
[New LWP 15819]
[New LWP 15820]
[New LWP 15821]
[New LWP 15822]
[New LWP 15823]
[New LWP 15824]
[New LWP 15825]
[New LWP 15826]
[New LWP 15827]
[New LWP 15828]
[New LWP 15829]
[New LWP 15830]
[New LWP 15831]
[New LWP 15832]
[New LWP 15833]
[New LWP 15834]
[New LWP 15835]
[New LWP 15836]
[New LWP 15837]
[New LWP 15838]
[New LWP 15839]
[New LWP 15840]
[New LWP 15841]
[New LWP 15842]
[New LWP 15843]
[New LWP 15844]
[New LWP 15845]
[New LWP 15846]
[New LWP 15847]
[New LWP 15848]
[New LWP 15849]
[New LWP 15850]
[New LWP 15851]
[New LWP 15852]
[New LWP 15853]
[New LWP 15854]
[New LWP 15855]
[New LWP 15856]
[New LWP 15857]
[New LWP 15858]
[New LWP 15859]
[New LWP 15860]
[New LWP 15861]
[New LWP 15862]
[New LWP 15863]
[New LWP 15864]
[New LWP 15865]
[New LWP 15866]
[New LWP 15867]
[New LWP 15868]
[New LWP 15869]
[New LWP 15870]
[New LWP 15871]
[New LWP 15872]
[New LWP 15874]
[New LWP 15875]
[New LWP 15876]
[New LWP 15877]
[New LWP 15878]
[New LWP 15879]
[New LWP 15880]
[New LWP 15881]
[New LWP 15882]
[New LWP 15883]
[New LWP 15884]
[New LWP 15885]
[New LWP 15886]
[New LWP 15887]
[New LWP 15888]
[New LWP 15889]
[New LWP 15890]
[New LWP 15891]
[New LWP 15892]
[New LWP 15893]
[New LWP 15894]
[New LWP 15895]
[New LWP 15896]
[New LWP 15899]
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
0x00007f109ec48d50 in ?? ()
Id Target Id Frame
* 1 LWP 15764 "kudu" 0x00007f109ec48d50 in ?? ()
2 LWP 15767 "kudu" 0x00007f109ec44fb9 in ?? ()
3 LWP 15768 "kudu" 0x00007f109ec44fb9 in ?? ()
4 LWP 15769 "kudu" 0x00007f109ec44fb9 in ?? ()
5 LWP 15770 "kernel-watcher-" 0x00007f109ec44fb9 in ?? ()
6 LWP 15776 "ntp client-1577" 0x00007f109ec489e2 in ?? ()
7 LWP 15777 "file cache-evic" 0x00007f109ec44fb9 in ?? ()
8 LWP 15778 "sq_acceptor" 0x00007f1097a6fcb9 in ?? ()
9 LWP 15781 "rpc reactor-157" 0x00007f1097a7ca47 in ?? ()
10 LWP 15782 "rpc reactor-157" 0x00007f1097a7ca47 in ?? ()
11 LWP 15783 "rpc reactor-157" 0x00007f1097a7ca47 in ?? ()
12 LWP 15784 "rpc reactor-157" 0x00007f1097a7ca47 in ?? ()
13 LWP 15785 "MaintenanceMgr " 0x00007f109ec44ad3 in ?? ()
14 LWP 15786 "txn-status-mana" 0x00007f109ec44fb9 in ?? ()
15 LWP 15788 "collect_and_rem" 0x00007f109ec44fb9 in ?? ()
16 LWP 15789 "tc-session-exp-" 0x00007f109ec44fb9 in ?? ()
17 LWP 15790 "rpc worker-1579" 0x00007f109ec44ad3 in ?? ()
18 LWP 15791 "rpc worker-1579" 0x00007f109ec44ad3 in ?? ()
19 LWP 15792 "rpc worker-1579" 0x00007f109ec44ad3 in ?? ()
20 LWP 15793 "rpc worker-1579" 0x00007f109ec44ad3 in ?? ()
21 LWP 15794 "rpc worker-1579" 0x00007f109ec44ad3 in ?? ()
22 LWP 15795 "rpc worker-1579" 0x00007f109ec44ad3 in ?? ()
23 LWP 15797 "rpc worker-1579" 0x00007f109ec44ad3 in ?? ()
24 LWP 15798 "rpc worker-1579" 0x00007f109ec44ad3 in ?? ()
25 LWP 15799 "rpc worker-1579" 0x00007f109ec44ad3 in ?? ()
26 LWP 15800 "rpc worker-1580" 0x00007f109ec44ad3 in ?? ()
27 LWP 15801 "rpc worker-1580" 0x00007f109ec44ad3 in ?? ()
28 LWP 15802 "rpc worker-1580" 0x00007f109ec44ad3 in ?? ()
29 LWP 15803 "rpc worker-1580" 0x00007f109ec44ad3 in ?? ()
30 LWP 15804 "rpc worker-1580" 0x00007f109ec44ad3 in ?? ()
31 LWP 15805 "rpc worker-1580" 0x00007f109ec44ad3 in ?? ()
32 LWP 15806 "rpc worker-1580" 0x00007f109ec44ad3 in ?? ()
33 LWP 15807 "rpc worker-1580" 0x00007f109ec44ad3 in ?? ()
34 LWP 15808 "rpc worker-1580" 0x00007f109ec44ad3 in ?? ()
35 LWP 15809 "rpc worker-1580" 0x00007f109ec44ad3 in ?? ()
36 LWP 15810 "rpc worker-1581" 0x00007f109ec44ad3 in ?? ()
37 LWP 15811 "rpc worker-1581" 0x00007f109ec44ad3 in ?? ()
38 LWP 15812 "rpc worker-1581" 0x00007f109ec44ad3 in ?? ()
39 LWP 15813 "rpc worker-1581" 0x00007f109ec44ad3 in ?? ()
40 LWP 15814 "rpc worker-1581" 0x00007f109ec44ad3 in ?? ()
41 LWP 15815 "rpc worker-1581" 0x00007f109ec44ad3 in ?? ()
42 LWP 15816 "rpc worker-1581" 0x00007f109ec44ad3 in ?? ()
43 LWP 15817 "rpc worker-1581" 0x00007f109ec44ad3 in ?? ()
44 LWP 15818 "rpc worker-1581" 0x00007f109ec44ad3 in ?? ()
45 LWP 15819 "rpc worker-1581" 0x00007f109ec44ad3 in ?? ()
46 LWP 15820 "rpc worker-1582" 0x00007f109ec44ad3 in ?? ()
47 LWP 15821 "rpc worker-1582" 0x00007f109ec44ad3 in ?? ()
48 LWP 15822 "rpc worker-1582" 0x00007f109ec44ad3 in ?? ()
49 LWP 15823 "rpc worker-1582" 0x00007f109ec44ad3 in ?? ()
50 LWP 15824 "rpc worker-1582" 0x00007f109ec44ad3 in ?? ()
51 LWP 15825 "rpc worker-1582" 0x00007f109ec44ad3 in ?? ()
52 LWP 15826 "rpc worker-1582" 0x00007f109ec44ad3 in ?? ()
53 LWP 15827 "rpc worker-1582" 0x00007f109ec44ad3 in ?? ()
54 LWP 15828 "rpc worker-1582" 0x00007f109ec44ad3 in ?? ()
55 LWP 15829 "rpc worker-1582" 0x00007f109ec44ad3 in ?? ()
56 LWP 15830 "rpc worker-1583" 0x00007f109ec44ad3 in ?? ()
57 LWP 15831 "rpc worker-1583" 0x00007f109ec44ad3 in ?? ()
58 LWP 15832 "rpc worker-1583" 0x00007f109ec44ad3 in ?? ()
59 LWP 15833 "rpc worker-1583" 0x00007f109ec44ad3 in ?? ()
60 LWP 15834 "rpc worker-1583" 0x00007f109ec44ad3 in ?? ()
61 LWP 15835 "rpc worker-1583" 0x00007f109ec44ad3 in ?? ()
62 LWP 15836 "rpc worker-1583" 0x00007f109ec44ad3 in ?? ()
63 LWP 15837 "rpc worker-1583" 0x00007f109ec44ad3 in ?? ()
64 LWP 15838 "rpc worker-1583" 0x00007f109ec44ad3 in ?? ()
65 LWP 15839 "rpc worker-1583" 0x00007f109ec44ad3 in ?? ()
66 LWP 15840 "rpc worker-1584" 0x00007f109ec44ad3 in ?? ()
67 LWP 15841 "rpc worker-1584" 0x00007f109ec44ad3 in ?? ()
68 LWP 15842 "rpc worker-1584" 0x00007f109ec44ad3 in ?? ()
69 LWP 15843 "rpc worker-1584" 0x00007f109ec44ad3 in ?? ()
70 LWP 15844 "rpc worker-1584" 0x00007f109ec44ad3 in ?? ()
71 LWP 15845 "rpc worker-1584" 0x00007f109ec44ad3 in ?? ()
72 LWP 15846 "rpc worker-1584" 0x00007f109ec44ad3 in ?? ()
73 LWP 15847 "rpc worker-1584" 0x00007f109ec44ad3 in ?? ()
74 LWP 15848 "rpc worker-1584" 0x00007f109ec44ad3 in ?? ()
75 LWP 15849 "rpc worker-1584" 0x00007f109ec44ad3 in ?? ()
76 LWP 15850 "rpc worker-1585" 0x00007f109ec44ad3 in ?? ()
77 LWP 15851 "rpc worker-1585" 0x00007f109ec44ad3 in ?? ()
78 LWP 15852 "rpc worker-1585" 0x00007f109ec44ad3 in ?? ()
79 LWP 15853 "rpc worker-1585" 0x00007f109ec44ad3 in ?? ()
80 LWP 15854 "rpc worker-1585" 0x00007f109ec44ad3 in ?? ()
81 LWP 15855 "rpc worker-1585" 0x00007f109ec44ad3 in ?? ()
82 LWP 15856 "rpc worker-1585" 0x00007f109ec44ad3 in ?? ()
83 LWP 15857 "rpc worker-1585" 0x00007f109ec44ad3 in ?? ()
84 LWP 15858 "rpc worker-1585" 0x00007f109ec44ad3 in ?? ()
85 LWP 15859 "rpc worker-1585" 0x00007f109ec44ad3 in ?? ()
86 LWP 15860 "rpc worker-1586" 0x00007f109ec44ad3 in ?? ()
87 LWP 15861 "rpc worker-1586" 0x00007f109ec44ad3 in ?? ()
88 LWP 15862 "rpc worker-1586" 0x00007f109ec44ad3 in ?? ()
89 LWP 15863 "rpc worker-1586" 0x00007f109ec44ad3 in ?? ()
90 LWP 15864 "rpc worker-1586" 0x00007f109ec44ad3 in ?? ()
91 LWP 15865 "rpc worker-1586" 0x00007f109ec44ad3 in ?? ()
92 LWP 15866 "rpc worker-1586" 0x00007f109ec44ad3 in ?? ()
93 LWP 15867 "rpc worker-1586" 0x00007f109ec44ad3 in ?? ()
94 LWP 15868 "rpc worker-1586" 0x00007f109ec44ad3 in ?? ()
95 LWP 15869 "rpc worker-1586" 0x00007f109ec44ad3 in ?? ()
96 LWP 15870 "rpc worker-1587" 0x00007f109ec44ad3 in ?? ()
97 LWP 15871 "rpc worker-1587" 0x00007f109ec44ad3 in ?? ()
98 LWP 15872 "rpc worker-1587" 0x00007f109ec44ad3 in ?? ()
99 LWP 15874 "rpc worker-1587" 0x00007f109ec44ad3 in ?? ()
100 LWP 15875 "rpc worker-1587" 0x00007f109ec44ad3 in ?? ()
101 LWP 15876 "rpc worker-1587" 0x00007f109ec44ad3 in ?? ()
102 LWP 15877 "rpc worker-1587" 0x00007f109ec44ad3 in ?? ()
103 LWP 15878 "rpc worker-1587" 0x00007f109ec44ad3 in ?? ()
104 LWP 15879 "rpc worker-1587" 0x00007f109ec44ad3 in ?? ()
105 LWP 15880 "rpc worker-1588" 0x00007f109ec44ad3 in ?? ()
106 LWP 15881 "rpc worker-1588" 0x00007f109ec44ad3 in ?? ()
107 LWP 15882 "rpc worker-1588" 0x00007f109ec44ad3 in ?? ()
108 LWP 15883 "rpc worker-1588" 0x00007f109ec44ad3 in ?? ()
109 LWP 15884 "rpc worker-1588" 0x00007f109ec44ad3 in ?? ()
110 LWP 15885 "rpc worker-1588" 0x00007f109ec44ad3 in ?? ()
111 LWP 15886 "rpc worker-1588" 0x00007f109ec44ad3 in ?? ()
112 LWP 15887 "rpc worker-1588" 0x00007f109ec44ad3 in ?? ()
113 LWP 15888 "rpc worker-1588" 0x00007f109ec44ad3 in ?? ()
114 LWP 15889 "rpc worker-1588" 0x00007f109ec44ad3 in ?? ()
115 LWP 15890 "rpc worker-1589" 0x00007f109ec44ad3 in ?? ()
116 LWP 15891 "rpc worker-1589" 0x00007f109ec44ad3 in ?? ()
117 LWP 15892 "diag-logger-158" 0x00007f109ec44fb9 in ?? ()
118 LWP 15893 "result-tracker-" 0x00007f109ec44fb9 in ?? ()
119 LWP 15894 "excess-log-dele" 0x00007f109ec44fb9 in ?? ()
120 LWP 15895 "acceptor-15895" 0x00007f1097a7e0c7 in ?? ()
121 LWP 15896 "heartbeat-15896" 0x00007f109ec44fb9 in ?? ()
122 LWP 15899 "maintenance_sch" 0x00007f109ec44fb9 in ?? ()
Thread 122 (LWP 15899):
#0 0x00007f109ec44fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000023a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017b80 in ?? ()
#5 0x00007f10535aa550 in ?? ()
#6 0x0000000000000474 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 15896):
#0 0x00007f109ec44fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000090 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x0000613000018b44 in ?? ()
#5 0x00007f10545d15b0 in ?? ()
#6 0x0000000000000121 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f10545d15d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0xffffffff0000006e in ?? ()
#12 0x00000fe20a8ba2cc in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 120 (LWP 15895):
#0 0x00007f1097a7e0c7 in ?? ()
#1 0x00006140000418a8 in ?? ()
#2 0x00007f1054df8d50 in ?? ()
#3 0x00007f1054df8d80 in ?? ()
#4 0x00007f1054df8e80 in ?? ()
#5 0x00007f1054df8d70 in ?? ()
#6 0x00007f1054df8de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x0000000000894fab in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 15894):
#0 0x00007f109ec44fb9 in ?? ()
#1 0x00007f1055611f00 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff1156e830 in ?? ()
#5 0x00007f1055611ef0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 15893):
#0 0x00007f109ec44fb9 in ?? ()
#1 0x00007f1055e2b0c0 in ?? ()
#2 0x000000000000008e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000861f0 in ?? ()
#5 0x00007f1055e2b0b0 in ?? ()
#6 0x000000000000011c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 15892):
#0 0x00007f109ec44fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 15891):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x000000000000001c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002e6818 in ?? ()
#4 0x00007f1056e5ae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1056e5ae70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 15890):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002e0018 in ?? ()
#4 0x00007f1057672e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1057672e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 15889):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 15888):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 15887):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 15886):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 15885):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 15884):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 15883):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 15882):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 15881):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 15880):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 15879):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 15878):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 15877):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 15876):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 15875):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 15874):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 15872):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 15871):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 15870):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 15869):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 15868):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 15867):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 15866):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 15865):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 15864):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 15863):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 15862):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 15861):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 15860):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 15859):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 15858):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 15857):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 15856):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 15855):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 15854):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 15853):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 15852):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 15851):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 15850):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000e1d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001cd06c in ?? ()
#4 0x00007f106b22be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f106b22be70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001cd020 in ?? ()
#9 0x00007f109ec44770 in ?? ()
#10 0x00007f106b22be70 in ?? ()
#11 0x00000fe20d6457d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 15849):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x00000000000012d2 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001c6868 in ?? ()
#4 0x00007f106ba43e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f106ba43e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 15848):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000eec in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001c0068 in ?? ()
#4 0x00007f106c25be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f106c25be70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 73 (LWP 15847):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 15846):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 15845):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 15844):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 15843):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 15842):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 15841):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 15840):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 15839):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 15838):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 15837):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 15836):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 15835):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 15834):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 15833):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 15832):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 15831):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 15830):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x000000000000000d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00013d02c in ?? ()
#4 0x00007f107540ce50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f107540ce70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00013cfe0 in ?? ()
#9 0x00007f109ec44770 in ?? ()
#10 0x00007f107540ce70 in ?? ()
#11 0x00000fe20ea819d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 15829):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 15828):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 15827):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 15826):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 15825):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 15824):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 15823):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 15822):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 15821):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 15820):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 15819):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 15818):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 15817):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 15816):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 15815):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 15814):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 15813):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 15812):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 15811):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 15810):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x000000000000000d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000affdc in ?? ()
#4 0x00007f107edd5e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f107edd5e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000aff90 in ?? ()
#9 0x00007f109ec44770 in ?? ()
#10 0x00007f107edd5e70 in ?? ()
#11 0x00000fe20fdbabd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 15809):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 15808):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 15807):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 15806):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 15805):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 15804):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 15803):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 15802):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 15801):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 15800):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 15799):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 15798):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 15797):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 15795):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 15794):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 15793):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 15792):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 15791):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 15790):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 15789):
#0 0x00007f109ec44fb9 in ?? ()
#1 0x00007f1088ec0c80 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019ae0 in ?? ()
#5 0x00007f1088ec0c70 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 15788):
#0 0x00007f109ec44fb9 in ?? ()
#1 0x00007f10896c21c0 in ?? ()
#2 0x000000000000001c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000cc98 in ?? ()
#5 0x00007f10896c2210 in ?? ()
#6 0x0000000000000038 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 15786):
#0 0x00007f109ec44fb9 in ?? ()
#1 0x00007f108a6c31c0 in ?? ()
#2 0x000000000000000e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000f9a8 in ?? ()
#5 0x00007f108a6c31b0 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 15785):
#0 0x00007f109ec44ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 15784):
#0 0x00007f1097a7ca47 in ?? ()
#1 0x00007f108b6c62e0 in ?? ()
#2 0x0000619000059180 in ?? ()
#3 0x00007f108b6c62d0 in ?? ()
#4 0x00007f108b6c64e0 in ?? ()
#5 0x00007f108b6c6320 in ?? ()
#6 0x0000614000041898 in ?? ()
#7 0x00007f108b6c63a0 in ?? ()
#8 0x00007f109a0c3de0 in ?? ()
#9 0x3fb953dc463e8000 in ?? ()
#10 0x0000619000059180 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059180 in ?? ()
#13 0x4215064ca7000000 in ?? ()
#14 0x00007f1000000000 in ?? ()
#15 0x41d98e3f99a4af46 in ?? ()
#16 0x00000fe2916d0c74 in ?? ()
#17 0x00007f108b6c6380 in ?? ()
#18 0x00007f109a0c749e in ?? ()
#19 0x00007f108b6c6350 in ?? ()
#20 0x3fb953dc463e8000 in ?? ()
#21 0x000000008b6c63a0 in ?? ()
#22 0x0000619000059180 in ?? ()
#23 0x0000614000041898 in ?? ()
#24 0x3fb953dc463e8000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 15783):
#0 0x00007f1097a7ca47 in ?? ()
#1 0x00007f108bec72e0 in ?? ()
#2 0x0000619000058c80 in ?? ()
#3 0x00007f108bec72d0 in ?? ()
#4 0x00007f108bec74e0 in ?? ()
#5 0x00007f108bec7320 in ?? ()
#6 0x0000614000041698 in ?? ()
#7 0x00007f108bec73a0 in ?? ()
#8 0x00007f109a0c3de0 in ?? ()
#9 0x3fb95cbf18dc0000 in ?? ()
#10 0x0000619000058c80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058c80 in ?? ()
#13 0x420da09264000000 in ?? ()
#14 0x00007f1000000000 in ?? ()
#15 0x41d98e3f99a4af45 in ?? ()
#16 0x00000fe2917d0e74 in ?? ()
#17 0x00007f108bec7380 in ?? ()
#18 0x00007f109a0c749e in ?? ()
#19 0x00007f108bec7350 in ?? ()
#20 0x3fb95cbf18dc0000 in ?? ()
#21 0x000000008bec73a0 in ?? ()
#22 0x0000619000058c80 in ?? ()
#23 0x0000614000041698 in ?? ()
#24 0x3fb95cbf18dc0000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 15782):
#0 0x00007f1097a7ca47 in ?? ()
#1 0x00007f108c6c82e0 in ?? ()
#2 0x0000619000058780 in ?? ()
#3 0x00007f108c6c82d0 in ?? ()
#4 0x00007f108c6c84e0 in ?? ()
#5 0x00007f108c6c8320 in ?? ()
#6 0x0000614000041498 in ?? ()
#7 0x00007f108c6c83a0 in ?? ()
#8 0x00007f109a0c3de0 in ?? ()
#9 0x3fb953617b60c000 in ?? ()
#10 0x0000619000058780 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058780 in ?? ()
#13 0x421d9f6137000000 in ?? ()
#14 0x00007f1000000000 in ?? ()
#15 0x41d98e3f99a4af48 in ?? ()
#16 0x00000fe2918d1074 in ?? ()
#17 0x00007f108c6c8380 in ?? ()
#18 0x00007f109a0c749e in ?? ()
#19 0x00007f108c6c8350 in ?? ()
#20 0x3fb953617b60c000 in ?? ()
#21 0x000000008c6c83a0 in ?? ()
#22 0x0000619000058780 in ?? ()
#23 0x0000614000041498 in ?? ()
#24 0x3fb953617b60c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 15781):
#0 0x00007f1097a7ca47 in ?? ()
#1 0x00007f108e2b32e0 in ?? ()
#2 0x0000619000058280 in ?? ()
#3 0x00007f108e2b32d0 in ?? ()
#4 0x00007f108e2b34e0 in ?? ()
#5 0x00007f108e2b3320 in ?? ()
#6 0x0000614000041298 in ?? ()
#7 0x00007f108e2b33a0 in ?? ()
#8 0x00007f109a0c3de0 in ?? ()
#9 0x3fb97863d8210000 in ?? ()
#10 0x0000619000058280 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058280 in ?? ()
#13 0x42193fc41f000000 in ?? ()
#14 0x00007f1000000000 in ?? ()
#15 0x41d98e3f99a4af46 in ?? ()
#16 0x00000fe291c4e674 in ?? ()
#17 0x00007f108e2b3380 in ?? ()
#18 0x00007f109a0c749e in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 15778):
#0 0x00007f1097a6fcb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f10902b7788 in ?? ()
#3 0x0000602000017cb0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x000000000088cbf1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 15777):
#0 0x00007f109ec44fb9 in ?? ()
#1 0x00007f108fab6080 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000c2c0 in ?? ()
#5 0x00007f108fab6070 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 15776):
#0 0x00007f109ec489e2 in ?? ()
#1 0x00007f108eab2b60 in ?? ()
#2 0x00007f108eab2c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f108eab2bb0 in ?? ()
#6 0x0000000001760060 in ?? ()
#7 0x00007f108eab2c10 in ?? ()
#8 0x0000611000087690 in ?? ()
#9 0x00007f108eab2c00 in ?? ()
#10 0x00000000008870e7 in __sanitizer::theDepot ()
#11 0x00007f10a45a838c in ?? ()
#12 0x00007f10a4597c19 in ?? ()
#13 0x00007f10a459c206 in ?? ()
#14 0x00007f10a45a0c30 in ?? ()
#15 0x00007f10a45a0a59 in ?? ()
#16 0x0000000000a60bcd in __sanitizer::theDepot ()
#17 0x00007f109b8b901e in ?? ()
#18 0x00007f109ec3e6db in ?? ()
#19 0x00000fe291d4e47c in ?? ()
#20 0x00007f108eab2400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 15770):
#0 0x00007f109ec44fb9 in ?? ()
#1 0x00007f10912b8c40 in ?? ()
#2 0x00000000000002cf in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000d1a8 in ?? ()
#5 0x00007f10912b8c30 in ?? ()
#6 0x000000000000059e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 15769):
#0 0x00007f109ec44fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000171b8 in ?? ()
#5 0x00007f1091aba410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 3 (LWP 15768):
#0 0x00007f109ec44fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000017038 in ?? ()
#5 0x00007f10922bb410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (LWP 15767):
#0 0x00007f109ec44fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f1092abc410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 15764):
#0 0x00007f109ec48d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20240506 16:19:13.654240 12982 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 4 with UUID fd8906dd0f884e22911dce7df910b6d4 and pid 14565
************************ BEGIN STACKS **************************
[New LWP 14567]
[New LWP 14568]
[New LWP 14569]
[New LWP 14570]
[New LWP 14576]
[New LWP 14577]
[New LWP 14578]
[New LWP 14582]
[New LWP 14583]
[New LWP 14584]
[New LWP 14585]
[New LWP 14586]
[New LWP 14587]
[New LWP 14588]
[New LWP 14589]
[New LWP 14590]
[New LWP 14591]
[New LWP 14592]
[New LWP 14593]
[New LWP 14594]
[New LWP 14595]
[New LWP 14596]
[New LWP 14597]
[New LWP 14598]
[New LWP 14599]
[New LWP 14600]
[New LWP 14601]
[New LWP 14602]
[New LWP 14603]
[New LWP 14604]
[New LWP 14605]
[New LWP 14606]
[New LWP 14607]
[New LWP 14608]
[New LWP 14609]
[New LWP 14610]
[New LWP 14611]
[New LWP 14612]
[New LWP 14613]
[New LWP 14614]
[New LWP 14615]
[New LWP 14616]
[New LWP 14617]
[New LWP 14618]
[New LWP 14619]
[New LWP 14620]
[New LWP 14621]
[New LWP 14622]
[New LWP 14623]
[New LWP 14625]
[New LWP 14626]
[New LWP 14627]
[New LWP 14628]
[New LWP 14629]
[New LWP 14630]
[New LWP 14631]
[New LWP 14632]
[New LWP 14633]
[New LWP 14634]
[New LWP 14635]
[New LWP 14636]
[New LWP 14637]
[New LWP 14638]
[New LWP 14639]
[New LWP 14640]
[New LWP 14641]
[New LWP 14642]
[New LWP 14643]
[New LWP 14644]
[New LWP 14645]
[New LWP 14646]
[New LWP 14647]
[New LWP 14648]
[New LWP 14649]
[New LWP 14650]
[New LWP 14651]
[New LWP 14652]
[New LWP 14653]
[New LWP 14654]
[New LWP 14655]
[New LWP 14656]
[New LWP 14657]
[New LWP 14658]
[New LWP 14659]
[New LWP 14660]
[New LWP 14661]
[New LWP 14662]
[New LWP 14663]
[New LWP 14664]
[New LWP 14665]
[New LWP 14666]
[New LWP 14667]
[New LWP 14668]
[New LWP 14669]
[New LWP 14670]
[New LWP 14671]
[New LWP 14672]
[New LWP 14673]
[New LWP 14674]
[New LWP 14675]
[New LWP 14676]
[New LWP 14678]
[New LWP 14679]
[New LWP 14680]
[New LWP 14681]
[New LWP 14682]
[New LWP 14683]
[New LWP 14684]
[New LWP 14685]
[New LWP 14686]
[New LWP 14687]
[New LWP 14688]
[New LWP 14689]
[New LWP 14691]
[New LWP 14692]
[New LWP 14693]
[New LWP 14694]
[New LWP 14695]
[New LWP 14696]
[New LWP 14697]
[New LWP 14698]
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
0x00007f189f232d50 in ?? ()
Id Target Id Frame
* 1 LWP 14565 "kudu" 0x00007f189f232d50 in ?? ()
2 LWP 14567 "kudu" 0x00007f189f22efb9 in ?? ()
3 LWP 14568 "kudu" 0x00007f189f22efb9 in ?? ()
4 LWP 14569 "kudu" 0x00007f189f22efb9 in ?? ()
5 LWP 14570 "kernel-watcher-" 0x00007f189f22efb9 in ?? ()
6 LWP 14576 "ntp client-1457" 0x00007f189f2329e2 in ?? ()
7 LWP 14577 "file cache-evic" 0x00007f189f22efb9 in ?? ()
8 LWP 14578 "sq_acceptor" 0x00007f1898059cb9 in ?? ()
9 LWP 14582 "rpc reactor-145" 0x00007f1898066a47 in ?? ()
10 LWP 14583 "rpc reactor-145" 0x00007f1898066a47 in ?? ()
11 LWP 14584 "rpc reactor-145" 0x00007f1898066a47 in ?? ()
12 LWP 14585 "rpc reactor-145" 0x00007f1898066a47 in ?? ()
13 LWP 14586 "MaintenanceMgr " 0x00007f189f22ead3 in ?? ()
14 LWP 14587 "txn-status-mana" 0x00007f189f22efb9 in ?? ()
15 LWP 14588 "collect_and_rem" 0x00007f189f22efb9 in ?? ()
16 LWP 14589 "tc-session-exp-" 0x00007f189f22efb9 in ?? ()
17 LWP 14590 "rpc worker-1459" 0x00007f189f22ead3 in ?? ()
18 LWP 14591 "rpc worker-1459" 0x00007f189f22ead3 in ?? ()
19 LWP 14592 "rpc worker-1459" 0x00007f189f22ead3 in ?? ()
20 LWP 14593 "rpc worker-1459" 0x00007f189f22ead3 in ?? ()
21 LWP 14594 "rpc worker-1459" 0x00007f189f22ead3 in ?? ()
22 LWP 14595 "rpc worker-1459" 0x00007f189f22ead3 in ?? ()
23 LWP 14596 "rpc worker-1459" 0x00007f189f22ead3 in ?? ()
24 LWP 14597 "rpc worker-1459" 0x00007f189f22ead3 in ?? ()
25 LWP 14598 "rpc worker-1459" 0x00007f189f22ead3 in ?? ()
26 LWP 14599 "rpc worker-1459" 0x00007f189f22ead3 in ?? ()
27 LWP 14600 "rpc worker-1460" 0x00007f189f22ead3 in ?? ()
28 LWP 14601 "rpc worker-1460" 0x00007f189f22ead3 in ?? ()
29 LWP 14602 "rpc worker-1460" 0x00007f189f22ead3 in ?? ()
30 LWP 14603 "rpc worker-1460" 0x00007f189f22ead3 in ?? ()
31 LWP 14604 "rpc worker-1460" 0x00007f189f22ead3 in ?? ()
32 LWP 14605 "rpc worker-1460" 0x00007f189f22ead3 in ?? ()
33 LWP 14606 "rpc worker-1460" 0x00007f189f22ead3 in ?? ()
34 LWP 14607 "rpc worker-1460" 0x00007f189f22ead3 in ?? ()
35 LWP 14608 "rpc worker-1460" 0x00007f189f22ead3 in ?? ()
36 LWP 14609 "rpc worker-1460" 0x00007f189f22ead3 in ?? ()
37 LWP 14610 "rpc worker-1461" 0x00007f189f22ead3 in ?? ()
38 LWP 14611 "rpc worker-1461" 0x00007f189f22ead3 in ?? ()
39 LWP 14612 "rpc worker-1461" 0x00007f189f22ead3 in ?? ()
40 LWP 14613 "rpc worker-1461" 0x00007f189f22ead3 in ?? ()
41 LWP 14614 "rpc worker-1461" 0x00007f189f22ead3 in ?? ()
42 LWP 14615 "rpc worker-1461" 0x00007f189f22ead3 in ?? ()
43 LWP 14616 "rpc worker-1461" 0x00007f189f22ead3 in ?? ()
44 LWP 14617 "rpc worker-1461" 0x00007f189f22ead3 in ?? ()
45 LWP 14618 "rpc worker-1461" 0x00007f189f22ead3 in ?? ()
46 LWP 14619 "rpc worker-1461" 0x00007f189f22ead3 in ?? ()
47 LWP 14620 "rpc worker-1462" 0x00007f189f22ead3 in ?? ()
48 LWP 14621 "rpc worker-1462" 0x00007f189f22ead3 in ?? ()
49 LWP 14622 "rpc worker-1462" 0x00007f189f22ead3 in ?? ()
50 LWP 14623 "rpc worker-1462" 0x00007f189f22ead3 in ?? ()
51 LWP 14625 "rpc worker-1462" 0x00007f189f22ead3 in ?? ()
52 LWP 14626 "rpc worker-1462" 0x00007f189f22ead3 in ?? ()
53 LWP 14627 "rpc worker-1462" 0x00007f189f22ead3 in ?? ()
54 LWP 14628 "rpc worker-1462" 0x00007f189f22ead3 in ?? ()
55 LWP 14629 "rpc worker-1462" 0x00007f189f22ead3 in ?? ()
56 LWP 14630 "rpc worker-1463" 0x00007f189f22ead3 in ?? ()
57 LWP 14631 "rpc worker-1463" 0x00007f189f22ead3 in ?? ()
58 LWP 14632 "rpc worker-1463" 0x00007f189f22ead3 in ?? ()
59 LWP 14633 "rpc worker-1463" 0x00007f189f22ead3 in ?? ()
60 LWP 14634 "rpc worker-1463" 0x00007f189f22ead3 in ?? ()
61 LWP 14635 "rpc worker-1463" 0x00007f189f22ead3 in ?? ()
62 LWP 14636 "rpc worker-1463" 0x00007f189f22ead3 in ?? ()
63 LWP 14637 "rpc worker-1463" 0x00007f189f22ead3 in ?? ()
64 LWP 14638 "rpc worker-1463" 0x00007f189f22ead3 in ?? ()
65 LWP 14639 "rpc worker-1463" 0x00007f189f22ead3 in ?? ()
66 LWP 14640 "rpc worker-1464" 0x00007f189f22ead3 in ?? ()
67 LWP 14641 "rpc worker-1464" 0x00007f189f22ead3 in ?? ()
68 LWP 14642 "rpc worker-1464" 0x00007f189f22ead3 in ?? ()
69 LWP 14643 "rpc worker-1464" 0x00007f189f22ead3 in ?? ()
70 LWP 14644 "rpc worker-1464" 0x00007f189f22ead3 in ?? ()
71 LWP 14645 "rpc worker-1464" 0x00007f189f22ead3 in ?? ()
72 LWP 14646 "rpc worker-1464" 0x00007f189f22ead3 in ?? ()
73 LWP 14647 "rpc worker-1464" 0x00007f189f22ead3 in ?? ()
74 LWP 14648 "rpc worker-1464" 0x00007f189f22ead3 in ?? ()
75 LWP 14649 "rpc worker-1464" 0x00007f189f22ead3 in ?? ()
76 LWP 14650 "rpc worker-1465" 0x00007f189f22ead3 in ?? ()
77 LWP 14651 "rpc worker-1465" 0x00007f189f22ead3 in ?? ()
78 LWP 14652 "rpc worker-1465" 0x00007f189f22ead3 in ?? ()
79 LWP 14653 "rpc worker-1465" 0x00007f189f22ead3 in ?? ()
80 LWP 14654 "rpc worker-1465" 0x00007f189f22ead3 in ?? ()
81 LWP 14655 "rpc worker-1465" 0x00007f189f22ead3 in ?? ()
82 LWP 14656 "rpc worker-1465" 0x00007f189f22ead3 in ?? ()
83 LWP 14657 "rpc worker-1465" 0x00007f189f22ead3 in ?? ()
84 LWP 14658 "rpc worker-1465" 0x00007f189f22ead3 in ?? ()
85 LWP 14659 "rpc worker-1465" 0x00007f189f22ead3 in ?? ()
86 LWP 14660 "rpc worker-1466" 0x00007f189f22ead3 in ?? ()
87 LWP 14661 "rpc worker-1466" 0x00007f189f22ead3 in ?? ()
88 LWP 14662 "rpc worker-1466" 0x00007f189f22ead3 in ?? ()
89 LWP 14663 "rpc worker-1466" 0x00007f189f22ead3 in ?? ()
90 LWP 14664 "rpc worker-1466" 0x00007f189f22ead3 in ?? ()
91 LWP 14665 "rpc worker-1466" 0x00007f189f22ead3 in ?? ()
92 LWP 14666 "rpc worker-1466" 0x00007f189f22ead3 in ?? ()
93 LWP 14667 "rpc worker-1466" 0x00007f189f22ead3 in ?? ()
94 LWP 14668 "rpc worker-1466" 0x00007f189f22ead3 in ?? ()
95 LWP 14669 "rpc worker-1466" 0x00007f189f22ead3 in ?? ()
96 LWP 14670 "rpc worker-1467" 0x00007f189f22ead3 in ?? ()
97 LWP 14671 "rpc worker-1467" 0x00007f189f22ead3 in ?? ()
98 LWP 14672 "rpc worker-1467" 0x00007f189f22ead3 in ?? ()
99 LWP 14673 "rpc worker-1467" 0x00007f189f22ead3 in ?? ()
100 LWP 14674 "rpc worker-1467" 0x00007f189f22ead3 in ?? ()
101 LWP 14675 "rpc worker-1467" 0x00007f189f22ead3 in ?? ()
102 LWP 14676 "rpc worker-1467" 0x00007f189f22ead3 in ?? ()
103 LWP 14678 "rpc worker-1467" 0x00007f189f22ead3 in ?? ()
104 LWP 14679 "rpc worker-1467" 0x00007f189f22ead3 in ?? ()
105 LWP 14680 "rpc worker-1468" 0x00007f189f22ead3 in ?? ()
106 LWP 14681 "rpc worker-1468" 0x00007f189f22ead3 in ?? ()
107 LWP 14682 "rpc worker-1468" 0x00007f189f22ead3 in ?? ()
108 LWP 14683 "rpc worker-1468" 0x00007f189f22ead3 in ?? ()
109 LWP 14684 "rpc worker-1468" 0x00007f189f22ead3 in ?? ()
110 LWP 14685 "rpc worker-1468" 0x00007f189f22ead3 in ?? ()
111 LWP 14686 "rpc worker-1468" 0x00007f189f22ead3 in ?? ()
112 LWP 14687 "rpc worker-1468" 0x00007f189f22ead3 in ?? ()
113 LWP 14688 "rpc worker-1468" 0x00007f189f22ead3 in ?? ()
114 LWP 14689 "rpc worker-1468" 0x00007f189f22ead3 in ?? ()
115 LWP 14691 "rpc worker-1469" 0x00007f189f22ead3 in ?? ()
116 LWP 14692 "rpc worker-1469" 0x00007f189f22ead3 in ?? ()
117 LWP 14693 "diag-logger-146" 0x00007f189f22efb9 in ?? ()
118 LWP 14694 "result-tracker-" 0x00007f189f22efb9 in ?? ()
119 LWP 14695 "excess-log-dele" 0x00007f189f22efb9 in ?? ()
120 LWP 14696 "acceptor-14696" 0x00007f18980680c7 in ?? ()
121 LWP 14697 "heartbeat-14697" 0x00007f189f22efb9 in ?? ()
122 LWP 14698 "maintenance_sch" 0x00007f189f22efb9 in ?? ()
Thread 122 (LWP 14698):
#0 0x00007f189f22efb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000030b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017b80 in ?? ()
#5 0x00007f1854bfb550 in ?? ()
#6 0x0000000000000616 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 14697):
#0 0x00007f189f22efb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000000c2 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000018b40 in ?? ()
#5 0x00007f18554135b0 in ?? ()
#6 0x0000000000000184 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 14696):
#0 0x00007f18980680c7 in ?? ()
#1 0x00006140000416a8 in ?? ()
#2 0x00007f1855c3ad50 in ?? ()
#3 0x00007f1855c3ad80 in ?? ()
#4 0x00007f1855c3ae80 in ?? ()
#5 0x00007f1855c3ad70 in ?? ()
#6 0x00007f1855c3ade0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x0000000000894fab in __sanitizer::theDepot ()
#9 0x00007f189be82da0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 14695):
#0 0x00007f189f22efb9 in ?? ()
#1 0x00007f1856454f00 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffddf066ab0 in ?? ()
#5 0x00007f1856454ef0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 14694):
#0 0x00007f189f22efb9 in ?? ()
#1 0x00007f1856c6d0c0 in ?? ()
#2 0x00000000000000c3 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000861f0 in ?? ()
#5 0x00007f1856c6d0b0 in ?? ()
#6 0x0000000000000186 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 14693):
#0 0x00007f189f22efb9 in ?? ()
#1 0x00007f1857484e00 in ?? ()
#2 0x00000000000000c5 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200004f160 in ?? ()
#5 0x00007f1857484e50 in ?? ()
#6 0x000000000000018a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 116 (LWP 14692):
#0 0x00007f189f22ead3 in ?? ()
#1 0x000000000000001b in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007f1857c9ce50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f1857c9ce70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007f189f22e770 in ?? ()
#10 0x00007f1857c9ce70 in ?? ()
#11 0x00000fe30af939d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 14691):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000029 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007f18584b4e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f18584b4e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007f189f22e770 in ?? ()
#10 0x00007f18584b4e70 in ?? ()
#11 0x00000fe30b0969d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 14689):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 14688):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 14687):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 14686):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 14685):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 14684):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 14683):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 14682):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 14681):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 14680):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 14679):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 14678):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 14676):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 14675):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 14674):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 14673):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 14672):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 14671):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 14670):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 14669):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 14668):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 14667):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 14666):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 14665):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 14664):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 14663):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 14662):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 14661):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 14660):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 14659):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 14658):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 14657):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 14656):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 14655):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 14654):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 14653):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 14652):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 14651):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 14650):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007f186c05de50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f186c05de70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007f189f22e770 in ?? ()
#10 0x00007f186c05de70 in ?? ()
#11 0x00000fe30d80bbd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 14649):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 14648):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 14647):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 14646):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 14645):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 14644):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 14643):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 14642):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 14641):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 14640):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 14639):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 14638):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 14637):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 14636):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 14635):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 14634):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 14633):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 14632):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 14631):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 14630):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007f187623ee50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f187623ee70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007f189f22e770 in ?? ()
#10 0x00007f187623ee70 in ?? ()
#11 0x00000fe30ec47dd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 14629):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 14628):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 14627):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 14626):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 14625):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 14623):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 14622):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 14621):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 14620):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 14619):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 14618):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 14617):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 14616):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 14615):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 14614):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 14613):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 14612):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 14611):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 14610):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 14609):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007f187fc07e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f187fc07e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007f189f22e770 in ?? ()
#10 0x00007f187fc07e70 in ?? ()
#11 0x00000fe30ff80fd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 14608):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 14607):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 14606):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 14605):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 14604):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 14603):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 14602):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 14601):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 14600):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 14599):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 14598):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 14597):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 14596):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 14595):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 14594):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 14593):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 14592):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 14591):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 14590):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 14589):
#0 0x00007f189f22efb9 in ?? ()
#1 0x00007f1889e1cc80 in ?? ()
#2 0x0000000000000013 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006130000193e0 in ?? ()
#5 0x00007f1889e1cc70 in ?? ()
#6 0x0000000000000026 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 14588):
#0 0x00007f189f22efb9 in ?? ()
#1 0x00007f188a6451c0 in ?? ()
#2 0x0000000000000027 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000cc98 in ?? ()
#5 0x00007f188a645210 in ?? ()
#6 0x000000000000004e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 14587):
#0 0x00007f189f22efb9 in ?? ()
#1 0x00007f188ae5c1c0 in ?? ()
#2 0x0000000000000013 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000f9a8 in ?? ()
#5 0x00007f188ae5c1b0 in ?? ()
#6 0x0000000000000026 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 14586):
#0 0x00007f189f22ead3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 14585):
#0 0x00007f1898066a47 in ?? ()
#1 0x00007f188be9c2e0 in ?? ()
#2 0x0000619000059180 in ?? ()
#3 0x00007f188be9c2d0 in ?? ()
#4 0x00007f188be9c4e0 in ?? ()
#5 0x00007f188be9c320 in ?? ()
#6 0x0000614000041898 in ?? ()
#7 0x00007f188be9c3a0 in ?? ()
#8 0x00007f189a6adde0 in ?? ()
#9 0x3fb96e92104a0000 in ?? ()
#10 0x0000619000059180 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059180 in ?? ()
#13 0x4222e3f697800000 in ?? ()
#14 0x00007f1800000000 in ?? ()
#15 0x41d98e3f99a4af49 in ?? ()
#16 0x00000fe3917cb874 in ?? ()
#17 0x00007f188be9c380 in ?? ()
#18 0x00007f189a6b149e in ?? ()
#19 0x00007f188be9c350 in ?? ()
#20 0x3fb96e92104a0000 in ?? ()
#21 0x000000008be9c3a0 in ?? ()
#22 0x0000619000059180 in ?? ()
#23 0x0000614000041898 in ?? ()
#24 0x3fb96e92104a0000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 14584):
#0 0x00007f1898066a47 in ?? ()
#1 0x00007f188c6b32e0 in ?? ()
#2 0x0000619000058c80 in ?? ()
#3 0x00007f188c6b32d0 in ?? ()
#4 0x00007f188c6b34e0 in ?? ()
#5 0x00007f188c6b3320 in ?? ()
#6 0x0000614000041698 in ?? ()
#7 0x00007f188c6b33a0 in ?? ()
#8 0x00007f189a6adde0 in ?? ()
#9 0x3fb95d1e4ecb0000 in ?? ()
#10 0x0000619000058c80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058c80 in ?? ()
#13 0x4213d1120e000000 in ?? ()
#14 0x00007f1800000000 in ?? ()
#15 0x41d98e3f99a4af48 in ?? ()
#16 0x00000fe3918ce674 in ?? ()
#17 0x00007f188c6b3380 in ?? ()
#18 0x00007f189a6b149e in ?? ()
#19 0x00007f188c6b3350 in ?? ()
#20 0x3fb95d1e4ecb0000 in ?? ()
#21 0x000000008c6b33a0 in ?? ()
#22 0x0000619000058c80 in ?? ()
#23 0x0000614000041698 in ?? ()
#24 0x3fb95d1e4ecb0000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 14583):
#0 0x00007f1898066a47 in ?? ()
#1 0x00007f188ceb42e0 in ?? ()
#2 0x0000619000058780 in ?? ()
#3 0x00007f188ceb42d0 in ?? ()
#4 0x00007f188ceb44e0 in ?? ()
#5 0x00007f188ceb4320 in ?? ()
#6 0x0000614000041498 in ?? ()
#7 0x00007f188ceb43a0 in ?? ()
#8 0x00007f189a6adde0 in ?? ()
#9 0x3fb99253cbeb8000 in ?? ()
#10 0x0000619000058780 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058780 in ?? ()
#13 0x422493fa4b800000 in ?? ()
#14 0x00007f1800000000 in ?? ()
#15 0x41d98e3f99a4af46 in ?? ()
#16 0x00000fe3919ce874 in ?? ()
#17 0x00007f188ceb4380 in ?? ()
#18 0x00007f189a6b149e in ?? ()
#19 0x00007f188ceb4350 in ?? ()
#20 0x3fb99253cbeb8000 in ?? ()
#21 0x000000008ceb43a0 in ?? ()
#22 0x0000619000058780 in ?? ()
#23 0x0000614000041498 in ?? ()
#24 0x3fb99253cbeb8000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 14582):
#0 0x00007f1898066a47 in ?? ()
#1 0x00007f188eead2e0 in ?? ()
#2 0x0000619000058280 in ?? ()
#3 0x00007f188eead2d0 in ?? ()
#4 0x00007f188eead4e0 in ?? ()
#5 0x00007f188eead320 in ?? ()
#6 0x0000614000041298 in ?? ()
#7 0x00007f188eead3a0 in ?? ()
#8 0x00007f189a6adde0 in ?? ()
#9 0x3fb967c6f4460000 in ?? ()
#10 0x0000619000058280 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058280 in ?? ()
#13 0x420cfaa1ec000000 in ?? ()
#14 0x00007f1800000000 in ?? ()
#15 0x41d98e3f99a4af47 in ?? ()
#16 0x00000fe391dcda74 in ?? ()
#17 0x00007f188eead380 in ?? ()
#18 0x00007f189a6b149e in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 14578):
#0 0x00007f1898059cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f18908b7788 in ?? ()
#3 0x0000602000017cb0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x000000000088cbf1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 14577):
#0 0x00007f189f22efb9 in ?? ()
#1 0x00007f18900b6080 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000c2c0 in ?? ()
#5 0x00007f18900b6070 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 14576):
#0 0x00007f189f2329e2 in ?? ()
#1 0x00007f188f8b3b60 in ?? ()
#2 0x00007f188f8b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f188f8b3bb0 in ?? ()
#6 0x0000000001760060 in ?? ()
#7 0x00007f188f8b3c10 in ?? ()
#8 0x0000611000087690 in ?? ()
#9 0x00007f188f8b3c00 in ?? ()
#10 0x00000000008870e7 in __sanitizer::theDepot ()
#11 0x00007f18a4b9238c in ?? ()
#12 0x00007f18a4b81c19 in ?? ()
#13 0x00007f18a4b86206 in ?? ()
#14 0x00007f18a4b8ac30 in ?? ()
#15 0x00007f18a4b8aa59 in ?? ()
#16 0x0000000000a60bcd in __sanitizer::theDepot ()
#17 0x00007f189bea301e in ?? ()
#18 0x00007f189f2286db in ?? ()
#19 0x00000fe391f0e67c in ?? ()
#20 0x00007f188f8b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 14570):
#0 0x00007f189f22efb9 in ?? ()
#1 0x00007f18918b8c40 in ?? ()
#2 0x00000000000003d2 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000d1a8 in ?? ()
#5 0x00007f18918b8c30 in ?? ()
#6 0x00000000000007a4 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 14569):
#0 0x00007f189f22efb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000171b8 in ?? ()
#5 0x00007f18920ba410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 3 (LWP 14568):
#0 0x00007f189f22efb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000017038 in ?? ()
#5 0x00007f18928bb410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (LWP 14567):
#0 0x00007f189f22efb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f18930bc410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 14565):
#0 0x00007f189f232d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20240506 16:19:14.449347 12982 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 5 with UUID c0471a9ec7ba4ea38e8df76c78d8b38f and pid 14425
************************ BEGIN STACKS **************************
[New LWP 14427]
[New LWP 14428]
[New LWP 14429]
[New LWP 14430]
[New LWP 14436]
[New LWP 14437]
[New LWP 14438]
[New LWP 14441]
[New LWP 14442]
[New LWP 14443]
[New LWP 14444]
[New LWP 14445]
[New LWP 14446]
[New LWP 14448]
[New LWP 14449]
[New LWP 14450]
[New LWP 14451]
[New LWP 14452]
[New LWP 14453]
[New LWP 14454]
[New LWP 14455]
[New LWP 14456]
[New LWP 14457]
[New LWP 14458]
[New LWP 14459]
[New LWP 14460]
[New LWP 14461]
[New LWP 14462]
[New LWP 14463]
[New LWP 14464]
[New LWP 14465]
[New LWP 14466]
[New LWP 14467]
[New LWP 14468]
[New LWP 14469]
[New LWP 14470]
[New LWP 14471]
[New LWP 14472]
[New LWP 14473]
[New LWP 14474]
[New LWP 14475]
[New LWP 14476]
[New LWP 14477]
[New LWP 14479]
[New LWP 14480]
[New LWP 14481]
[New LWP 14482]
[New LWP 14483]
[New LWP 14484]
[New LWP 14485]
[New LWP 14486]
[New LWP 14487]
[New LWP 14489]
[New LWP 14490]
[New LWP 14491]
[New LWP 14492]
[New LWP 14493]
[New LWP 14494]
[New LWP 14495]
[New LWP 14496]
[New LWP 14497]
[New LWP 14498]
[New LWP 14499]
[New LWP 14500]
[New LWP 14501]
[New LWP 14502]
[New LWP 14503]
[New LWP 14504]
[New LWP 14505]
[New LWP 14506]
[New LWP 14508]
[New LWP 14509]
[New LWP 14511]
[New LWP 14512]
[New LWP 14513]
[New LWP 14514]
[New LWP 14515]
[New LWP 14516]
[New LWP 14517]
[New LWP 14518]
[New LWP 14519]
[New LWP 14520]
[New LWP 14521]
[New LWP 14522]
[New LWP 14523]
[New LWP 14524]
[New LWP 14525]
[New LWP 14526]
[New LWP 14527]
[New LWP 14528]
[New LWP 14529]
[New LWP 14530]
[New LWP 14531]
[New LWP 14532]
[New LWP 14533]
[New LWP 14534]
[New LWP 14535]
[New LWP 14536]
[New LWP 14537]
[New LWP 14538]
[New LWP 14539]
[New LWP 14540]
[New LWP 14541]
[New LWP 14542]
[New LWP 14543]
[New LWP 14544]
[New LWP 14545]
[New LWP 14546]
[New LWP 14547]
[New LWP 14548]
[New LWP 14549]
[New LWP 14550]
[New LWP 14551]
[New LWP 14552]
[New LWP 14553]
[New LWP 14554]
[New LWP 14555]
[New LWP 14556]
[New LWP 14557]
[New LWP 14558]
[New LWP 14564]
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
0x00007fa7747bdd50 in ?? ()
Id Target Id Frame
* 1 LWP 14425 "kudu" 0x00007fa7747bdd50 in ?? ()
2 LWP 14427 "kudu" 0x00007fa7747b9fb9 in ?? ()
3 LWP 14428 "kudu" 0x00007fa7747b9fb9 in ?? ()
4 LWP 14429 "kudu" 0x00007fa7747b9fb9 in ?? ()
5 LWP 14430 "kernel-watcher-" 0x00007fa7747b9fb9 in ?? ()
6 LWP 14436 "ntp client-1443" 0x00007fa7747bd9e2 in ?? ()
7 LWP 14437 "file cache-evic" 0x00007fa7747b9fb9 in ?? ()
8 LWP 14438 "sq_acceptor" 0x00007fa76d5e4cb9 in ?? ()
9 LWP 14441 "rpc reactor-144" 0x00007fa76d5f1a47 in ?? ()
10 LWP 14442 "rpc reactor-144" 0x00007fa76d5f1a47 in ?? ()
11 LWP 14443 "rpc reactor-144" 0x00007fa76d5f1a47 in ?? ()
12 LWP 14444 "rpc reactor-144" 0x00007fa76d5f1a47 in ?? ()
13 LWP 14445 "MaintenanceMgr " 0x00007fa7747b9ad3 in ?? ()
14 LWP 14446 "txn-status-mana" 0x00007fa7747b9fb9 in ?? ()
15 LWP 14448 "collect_and_rem" 0x00007fa7747b9fb9 in ?? ()
16 LWP 14449 "tc-session-exp-" 0x00007fa7747b9fb9 in ?? ()
17 LWP 14450 "rpc worker-1445" 0x00007fa7747b9ad3 in ?? ()
18 LWP 14451 "rpc worker-1445" 0x00007fa7747b9ad3 in ?? ()
19 LWP 14452 "rpc worker-1445" 0x00007fa7747b9ad3 in ?? ()
20 LWP 14453 "rpc worker-1445" 0x00007fa7747b9ad3 in ?? ()
21 LWP 14454 "rpc worker-1445" 0x00007fa7747b9ad3 in ?? ()
22 LWP 14455 "rpc worker-1445" 0x00007fa7747b9ad3 in ?? ()
23 LWP 14456 "rpc worker-1445" 0x00007fa7747b9ad3 in ?? ()
24 LWP 14457 "rpc worker-1445" 0x00007fa7747b9ad3 in ?? ()
25 LWP 14458 "rpc worker-1445" 0x00007fa7747b9ad3 in ?? ()
26 LWP 14459 "rpc worker-1445" 0x00007fa7747b9ad3 in ?? ()
27 LWP 14460 "rpc worker-1446" 0x00007fa7747b9ad3 in ?? ()
28 LWP 14461 "rpc worker-1446" 0x00007fa7747b9ad3 in ?? ()
29 LWP 14462 "rpc worker-1446" 0x00007fa7747b9ad3 in ?? ()
30 LWP 14463 "rpc worker-1446" 0x00007fa7747b9ad3 in ?? ()
31 LWP 14464 "rpc worker-1446" 0x00007fa7747b9ad3 in ?? ()
32 LWP 14465 "rpc worker-1446" 0x00007fa7747b9ad3 in ?? ()
33 LWP 14466 "rpc worker-1446" 0x00007fa7747b9ad3 in ?? ()
34 LWP 14467 "rpc worker-1446" 0x00007fa7747b9ad3 in ?? ()
35 LWP 14468 "rpc worker-1446" 0x00007fa7747b9ad3 in ?? ()
36 LWP 14469 "rpc worker-1446" 0x00007fa7747b9ad3 in ?? ()
37 LWP 14470 "rpc worker-1447" 0x00007fa7747b9ad3 in ?? ()
38 LWP 14471 "rpc worker-1447" 0x00007fa7747b9ad3 in ?? ()
39 LWP 14472 "rpc worker-1447" 0x00007fa7747b9ad3 in ?? ()
40 LWP 14473 "rpc worker-1447" 0x00007fa7747b9ad3 in ?? ()
41 LWP 14474 "rpc worker-1447" 0x00007fa7747b9ad3 in ?? ()
42 LWP 14475 "rpc worker-1447" 0x00007fa7747b9ad3 in ?? ()
43 LWP 14476 "rpc worker-1447" 0x00007fa7747b9ad3 in ?? ()
44 LWP 14477 "rpc worker-1447" 0x00007fa7747b9ad3 in ?? ()
45 LWP 14479 "rpc worker-1447" 0x00007fa7747b9ad3 in ?? ()
46 LWP 14480 "rpc worker-1448" 0x00007fa7747b9ad3 in ?? ()
47 LWP 14481 "rpc worker-1448" 0x00007fa7747b9ad3 in ?? ()
48 LWP 14482 "rpc worker-1448" 0x00007fa7747b9ad3 in ?? ()
49 LWP 14483 "rpc worker-1448" 0x00007fa7747b9ad3 in ?? ()
50 LWP 14484 "rpc worker-1448" 0x00007fa7747b9ad3 in ?? ()
51 LWP 14485 "rpc worker-1448" 0x00007fa7747b9ad3 in ?? ()
52 LWP 14486 "rpc worker-1448" 0x00007fa7747b9ad3 in ?? ()
53 LWP 14487 "rpc worker-1448" 0x00007fa7747b9ad3 in ?? ()
54 LWP 14489 "rpc worker-1448" 0x00007fa7747b9ad3 in ?? ()
55 LWP 14490 "rpc worker-1449" 0x00007fa7747b9ad3 in ?? ()
56 LWP 14491 "rpc worker-1449" 0x00007fa7747b9ad3 in ?? ()
57 LWP 14492 "rpc worker-1449" 0x00007fa7747b9ad3 in ?? ()
58 LWP 14493 "rpc worker-1449" 0x00007fa7747b9ad3 in ?? ()
59 LWP 14494 "rpc worker-1449" 0x00007fa7747b9ad3 in ?? ()
60 LWP 14495 "rpc worker-1449" 0x00007fa7747b9ad3 in ?? ()
61 LWP 14496 "rpc worker-1449" 0x00007fa7747b9ad3 in ?? ()
62 LWP 14497 "rpc worker-1449" 0x00007fa7747b9ad3 in ?? ()
63 LWP 14498 "rpc worker-1449" 0x00007fa7747b9ad3 in ?? ()
64 LWP 14499 "rpc worker-1449" 0x00007fa7747b9ad3 in ?? ()
65 LWP 14500 "rpc worker-1450" 0x00007fa7747b9ad3 in ?? ()
66 LWP 14501 "rpc worker-1450" 0x00007fa7747b9ad3 in ?? ()
67 LWP 14502 "rpc worker-1450" 0x00007fa7747b9ad3 in ?? ()
68 LWP 14503 "rpc worker-1450" 0x00007fa7747b9ad3 in ?? ()
69 LWP 14504 "rpc worker-1450" 0x00007fa7747b9ad3 in ?? ()
70 LWP 14505 "rpc worker-1450" 0x00007fa7747b9ad3 in ?? ()
71 LWP 14506 "rpc worker-1450" 0x00007fa7747b9ad3 in ?? ()
72 LWP 14508 "rpc worker-1450" 0x00007fa7747b9ad3 in ?? ()
73 LWP 14509 "rpc worker-1450" 0x00007fa7747b9ad3 in ?? ()
74 LWP 14511 "rpc worker-1451" 0x00007fa7747b9ad3 in ?? ()
75 LWP 14512 "rpc worker-1451" 0x00007fa7747b9ad3 in ?? ()
76 LWP 14513 "rpc worker-1451" 0x00007fa7747b9ad3 in ?? ()
77 LWP 14514 "rpc worker-1451" 0x00007fa7747b9ad3 in ?? ()
78 LWP 14515 "rpc worker-1451" 0x00007fa7747b9ad3 in ?? ()
79 LWP 14516 "rpc worker-1451" 0x00007fa7747b9ad3 in ?? ()
80 LWP 14517 "rpc worker-1451" 0x00007fa7747b9ad3 in ?? ()
81 LWP 14518 "rpc worker-1451" 0x00007fa7747b9ad3 in ?? ()
82 LWP 14519 "rpc worker-1451" 0x00007fa7747b9ad3 in ?? ()
83 LWP 14520 "rpc worker-1452" 0x00007fa7747b9ad3 in ?? ()
84 LWP 14521 "rpc worker-1452" 0x00007fa7747b9ad3 in ?? ()
85 LWP 14522 "rpc worker-1452" 0x00007fa7747b9ad3 in ?? ()
86 LWP 14523 "rpc worker-1452" 0x00007fa7747b9ad3 in ?? ()
87 LWP 14524 "rpc worker-1452" 0x00007fa7747b9ad3 in ?? ()
88 LWP 14525 "rpc worker-1452" 0x00007fa7747b9ad3 in ?? ()
89 LWP 14526 "rpc worker-1452" 0x00007fa7747b9ad3 in ?? ()
90 LWP 14527 "rpc worker-1452" 0x00007fa7747b9ad3 in ?? ()
91 LWP 14528 "rpc worker-1452" 0x00007fa7747b9ad3 in ?? ()
92 LWP 14529 "rpc worker-1452" 0x00007fa7747b9ad3 in ?? ()
93 LWP 14530 "rpc worker-1453" 0x00007fa7747b9ad3 in ?? ()
94 LWP 14531 "rpc worker-1453" 0x00007fa7747b9ad3 in ?? ()
95 LWP 14532 "rpc worker-1453" 0x00007fa7747b9ad3 in ?? ()
96 LWP 14533 "rpc worker-1453" 0x00007fa7747b9ad3 in ?? ()
97 LWP 14534 "rpc worker-1453" 0x00007fa7747b9ad3 in ?? ()
98 LWP 14535 "rpc worker-1453" 0x00007fa7747b9ad3 in ?? ()
99 LWP 14536 "rpc worker-1453" 0x00007fa7747b9ad3 in ?? ()
100 LWP 14537 "rpc worker-1453" 0x00007fa7747b9ad3 in ?? ()
101 LWP 14538 "rpc worker-1453" 0x00007fa7747b9ad3 in ?? ()
102 LWP 14539 "rpc worker-1453" 0x00007fa7747b9ad3 in ?? ()
103 LWP 14540 "rpc worker-1454" 0x00007fa7747b9ad3 in ?? ()
104 LWP 14541 "rpc worker-1454" 0x00007fa7747b9ad3 in ?? ()
105 LWP 14542 "rpc worker-1454" 0x00007fa7747b9ad3 in ?? ()
106 LWP 14543 "rpc worker-1454" 0x00007fa7747b9ad3 in ?? ()
107 LWP 14544 "rpc worker-1454" 0x00007fa7747b9ad3 in ?? ()
108 LWP 14545 "rpc worker-1454" 0x00007fa7747b9ad3 in ?? ()
109 LWP 14546 "rpc worker-1454" 0x00007fa7747b9ad3 in ?? ()
110 LWP 14547 "rpc worker-1454" 0x00007fa7747b9ad3 in ?? ()
111 LWP 14548 "rpc worker-1454" 0x00007fa7747b9ad3 in ?? ()
112 LWP 14549 "rpc worker-1454" 0x00007fa7747b9ad3 in ?? ()
113 LWP 14550 "rpc worker-1455" 0x00007fa7747b9ad3 in ?? ()
114 LWP 14551 "rpc worker-1455" 0x00007fa7747b9ad3 in ?? ()
115 LWP 14552 "rpc worker-1455" 0x00007fa7747b9ad3 in ?? ()
116 LWP 14553 "rpc worker-1455" 0x00007fa7747b9ad3 in ?? ()
117 LWP 14554 "diag-logger-145" 0x00007fa7747b9fb9 in ?? ()
118 LWP 14555 "result-tracker-" 0x00007fa7747b9fb9 in ?? ()
119 LWP 14556 "excess-log-dele" 0x00007fa7747b9fb9 in ?? ()
120 LWP 14557 "acceptor-14557" 0x00007fa76d5f30c7 in ?? ()
121 LWP 14558 "heartbeat-14558" 0x00007fa7747b9fb9 in ?? ()
122 LWP 14564 "maintenance_sch" 0x00007fa7747b9fb9 in ?? ()
Thread 122 (LWP 14564):
#0 0x00007fa7747b9fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000317 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017b80 in ?? ()
#5 0x00007fa729977550 in ?? ()
#6 0x000000000000062e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 14558):
#0 0x00007fa7747b9fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000000c5 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000018b40 in ?? ()
#5 0x00007fa72a99e5b0 in ?? ()
#6 0x000000000000018a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 14557):
#0 0x00007fa76d5f30c7 in ?? ()
#1 0x0000614000041aa8 in ?? ()
#2 0x00007fa72b1c5d50 in ?? ()
#3 0x00007fa72b1c5d80 in ?? ()
#4 0x00007fa72b1c5e80 in ?? ()
#5 0x00007fa72b1c5d70 in ?? ()
#6 0x00007fa72b1c5de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x0000000000894fab in __sanitizer::theDepot ()
#9 0x00007fa77140dda0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 14556):
#0 0x00007fa7747b9fb9 in ?? ()
#1 0x00007fa72b9dff00 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff192647b0 in ?? ()
#5 0x00007fa72b9dfef0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 14555):
#0 0x00007fa7747b9fb9 in ?? ()
#1 0x00007fa72c1f80c0 in ?? ()
#2 0x00000000000000c6 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000861f0 in ?? ()
#5 0x00007fa72c1f80b0 in ?? ()
#6 0x000000000000018c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 14554):
#0 0x00007fa7747b9fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 14553):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000020 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007fa72d227e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa72d227e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 14552):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 14551):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000024 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002a6858 in ?? ()
#4 0x00007fa72e257e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa72e257e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 113 (LWP 14550):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 14549):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 14548):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 14547):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 14546):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 14545):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 14544):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 14543):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 14542):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 14541):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 14540):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 14539):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 14538):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 14537):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 14536):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 14535):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 14534):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 14533):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 14532):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 14531):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 14530):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 14529):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 14528):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 14527):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 14526):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 14525):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 14524):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 14523):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 14522):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 14521):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 14520):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 14519):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 14518):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 14517):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 14516):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 14515):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 14514):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 14513):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007fa7415e8e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa7415e8e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007fa7747b9770 in ?? ()
#10 0x00007fa7415e8e70 in ?? ()
#11 0x00000ff4e82bd1d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 14512):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 14511):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 14509):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 14508):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 14506):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 14505):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 14504):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 14503):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 14502):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 14501):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 14500):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 14499):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 14498):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 14497):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 14496):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 14495):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 14494):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 14493):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 14492):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 14491):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007fa74b7c9e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa74b7c9e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007fa7747b9770 in ?? ()
#10 0x00007fa74b7c9e70 in ?? ()
#11 0x00000ff4e96f93d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 14490):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 14489):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 14487):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 14486):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 14485):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 14484):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 14483):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 14482):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 14481):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 14480):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 14479):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 14477):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 14476):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 14475):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 14474):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 14473):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 14472):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 14471):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 14470):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 14469):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007fa755192e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa755192e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007fa7747b9770 in ?? ()
#10 0x00007fa755192e70 in ?? ()
#11 0x00000ff4eaa325d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 14468):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 14467):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 14466):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 14465):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 14464):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 14463):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 14462):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 14461):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 14460):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 14459):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 14458):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 14457):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 14456):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 14455):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 14454):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 14453):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 14452):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 14451):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 14450):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 14449):
#0 0x00007fa7747b9fb9 in ?? ()
#1 0x00007fa75f39dc80 in ?? ()
#2 0x0000000000000013 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006130000193e0 in ?? ()
#5 0x00007fa75f39dc70 in ?? ()
#6 0x0000000000000026 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 14448):
#0 0x00007fa7747b9fb9 in ?? ()
#1 0x00007fa75fbaf1c0 in ?? ()
#2 0x0000000000000027 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000cc98 in ?? ()
#5 0x00007fa75fbaf210 in ?? ()
#6 0x000000000000004e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 14446):
#0 0x00007fa7747b9fb9 in ?? ()
#1 0x00007fa7603af1c0 in ?? ()
#2 0x0000000000000013 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000f9a8 in ?? ()
#5 0x00007fa7603af1b0 in ?? ()
#6 0x0000000000000026 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 14445):
#0 0x00007fa7747b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 14444):
#0 0x00007fa76d5f1a47 in ?? ()
#1 0x00007fa7613b22e0 in ?? ()
#2 0x0000619000059180 in ?? ()
#3 0x00007fa7613b22d0 in ?? ()
#4 0x00007fa7613b24e0 in ?? ()
#5 0x00007fa7613b2320 in ?? ()
#6 0x0000614000041a98 in ?? ()
#7 0x00007fa7613b23a0 in ?? ()
#8 0x00007fa76fc38de0 in ?? ()
#9 0x3fb95c79859fc000 in ?? ()
#10 0x0000619000059180 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059180 in ?? ()
#13 0x420401b53a000000 in ?? ()
#14 0x00007fa700000000 in ?? ()
#15 0x41d98e3f99a4af46 in ?? ()
#16 0x00000ff56c26e474 in ?? ()
#17 0x00007fa7613b2380 in ?? ()
#18 0x00007fa76fc3c49e in ?? ()
#19 0x00007fa7613b2350 in ?? ()
#20 0x3fb95c79859fc000 in ?? ()
#21 0x00000000613b23a0 in ?? ()
#22 0x0000619000059180 in ?? ()
#23 0x0000614000041a98 in ?? ()
#24 0x3fb95c79859fc000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 14443):
#0 0x00007fa76d5f1a47 in ?? ()
#1 0x00007fa761bb32e0 in ?? ()
#2 0x0000619000058c80 in ?? ()
#3 0x00007fa761bb32d0 in ?? ()
#4 0x00007fa761bb34e0 in ?? ()
#5 0x00007fa761bb3320 in ?? ()
#6 0x0000614000041898 in ?? ()
#7 0x00007fa761bb33a0 in ?? ()
#8 0x00007fa76fc38de0 in ?? ()
#9 0x3fb97c52b64a4000 in ?? ()
#10 0x0000619000058c80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058c80 in ?? ()
#13 0x42232da274800000 in ?? ()
#14 0x00007fa700000000 in ?? ()
#15 0x41d98e3f99a4af47 in ?? ()
#16 0x00000ff56c36e674 in ?? ()
#17 0x00007fa761bb3380 in ?? ()
#18 0x00007fa76fc3c49e in ?? ()
#19 0x00007fa761bb3350 in ?? ()
#20 0x3fb97c52b64a4000 in ?? ()
#21 0x0000000061bb33a0 in ?? ()
#22 0x0000619000058c80 in ?? ()
#23 0x0000614000041898 in ?? ()
#24 0x3fb97c52b64a4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 14442):
#0 0x00007fa76d5f1a47 in ?? ()
#1 0x00007fa7623b42e0 in ?? ()
#2 0x0000619000058780 in ?? ()
#3 0x00007fa7623b42d0 in ?? ()
#4 0x00007fa7623b44e0 in ?? ()
#5 0x00007fa7623b4320 in ?? ()
#6 0x0000614000041698 in ?? ()
#7 0x00007fa7623b43a0 in ?? ()
#8 0x00007fa76fc38de0 in ?? ()
#9 0x3fb954ea1a320000 in ?? ()
#10 0x0000619000058780 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058780 in ?? ()
#13 0x421b4a9e38000000 in ?? ()
#14 0x00007fa700000000 in ?? ()
#15 0x41d98e3f99a4af45 in ?? ()
#16 0x00000ff56c46e874 in ?? ()
#17 0x00007fa7623b4380 in ?? ()
#18 0x00007fa76fc3c49e in ?? ()
#19 0x00007fa7623b4350 in ?? ()
#20 0x3fb954ea1a320000 in ?? ()
#21 0x00000000623b43a0 in ?? ()
#22 0x0000619000058780 in ?? ()
#23 0x0000614000041698 in ?? ()
#24 0x3fb954ea1a320000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 14441):
#0 0x00007fa76d5f1a47 in ?? ()
#1 0x00007fa7643ad2e0 in ?? ()
#2 0x0000619000058280 in ?? ()
#3 0x00007fa7643ad2d0 in ?? ()
#4 0x00007fa7643ad4e0 in ?? ()
#5 0x00007fa7643ad320 in ?? ()
#6 0x0000614000041498 in ?? ()
#7 0x00007fa7643ad3a0 in ?? ()
#8 0x00007fa76fc38de0 in ?? ()
#9 0x3fb9711f267b0000 in ?? ()
#10 0x0000619000058280 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058280 in ?? ()
#13 0x4222d3c033800000 in ?? ()
#14 0x00007fa700000000 in ?? ()
#15 0x41d98e3f99a4af47 in ?? ()
#16 0x00000ff56c86da74 in ?? ()
#17 0x00007fa7643ad380 in ?? ()
#18 0x00007fa76fc3c49e in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 14438):
#0 0x00007fa76d5e4cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fa765db7788 in ?? ()
#3 0x0000602000017cb0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x000000000088cbf1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 14437):
#0 0x00007fa7747b9fb9 in ?? ()
#1 0x00007fa7655b6080 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000c2c0 in ?? ()
#5 0x00007fa7655b6070 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 14436):
#0 0x00007fa7747bd9e2 in ?? ()
#1 0x00007fa764db3b60 in ?? ()
#2 0x00007fa764db3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fa764db3bb0 in ?? ()
#6 0x0000000001760060 in ?? ()
#7 0x00007fa764db3c10 in ?? ()
#8 0x0000611000087690 in ?? ()
#9 0x00007fa764db3c00 in ?? ()
#10 0x00000000008870e7 in __sanitizer::theDepot ()
#11 0x00007fa77a11d38c in ?? ()
#12 0x00007fa77a10cc19 in ?? ()
#13 0x00007fa77a111206 in ?? ()
#14 0x00007fa77a115c30 in ?? ()
#15 0x00007fa77a115a59 in ?? ()
#16 0x0000000000a60bcd in __sanitizer::theDepot ()
#17 0x00007fa77142e01e in ?? ()
#18 0x00007fa7747b36db in ?? ()
#19 0x00000ff56c9ae67c in ?? ()
#20 0x00007fa764db3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 14430):
#0 0x00007fa7747b9fb9 in ?? ()
#1 0x00007fa766db8c40 in ?? ()
#2 0x00000000000003e2 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000d1a8 in ?? ()
#5 0x00007fa766db8c30 in ?? ()
#6 0x00000000000007c4 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 14429):
#0 0x00007fa7747b9fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000171b8 in ?? ()
#5 0x00007fa7675ba410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 3 (LWP 14428):
#0 0x00007fa7747b9fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000017038 in ?? ()
#5 0x00007fa767dbb410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (LWP 14427):
#0 0x00007fa7747b9fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fa7685bc410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 14425):
#0 0x00007fa7747bdd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20240506 16:19:15.236790 12982 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 6 with UUID 18c7d19aee15430b8c3a0ef90b23fee5 and pid 14704
************************ BEGIN STACKS **************************
[New LWP 14708]
[New LWP 14709]
[New LWP 14710]
[New LWP 14711]
[New LWP 14718]
[New LWP 14719]
[New LWP 14720]
[New LWP 14723]
[New LWP 14726]
[New LWP 14727]
[New LWP 14728]
[New LWP 14729]
[New LWP 14730]
[New LWP 14731]
[New LWP 14732]
[New LWP 14733]
[New LWP 14734]
[New LWP 14735]
[New LWP 14736]
[New LWP 14737]
[New LWP 14738]
[New LWP 14739]
[New LWP 14740]
[New LWP 14741]
[New LWP 14742]
[New LWP 14743]
[New LWP 14744]
[New LWP 14745]
[New LWP 14746]
[New LWP 14747]
[New LWP 14748]
[New LWP 14749]
[New LWP 14750]
[New LWP 14751]
[New LWP 14752]
[New LWP 14753]
[New LWP 14754]
[New LWP 14755]
[New LWP 14756]
[New LWP 14757]
[New LWP 14758]
[New LWP 14759]
[New LWP 14760]
[New LWP 14761]
[New LWP 14762]
[New LWP 14763]
[New LWP 14764]
[New LWP 14765]
[New LWP 14766]
[New LWP 14767]
[New LWP 14768]
[New LWP 14769]
[New LWP 14770]
[New LWP 14771]
[New LWP 14772]
[New LWP 14773]
[New LWP 14774]
[New LWP 14775]
[New LWP 14776]
[New LWP 14777]
[New LWP 14778]
[New LWP 14779]
[New LWP 14780]
[New LWP 14781]
[New LWP 14782]
[New LWP 14783]
[New LWP 14784]
[New LWP 14785]
[New LWP 14786]
[New LWP 14787]
[New LWP 14788]
[New LWP 14789]
[New LWP 14790]
[New LWP 14791]
[New LWP 14792]
[New LWP 14793]
[New LWP 14794]
[New LWP 14795]
[New LWP 14796]
[New LWP 14797]
[New LWP 14798]
[New LWP 14799]
[New LWP 14800]
[New LWP 14801]
[New LWP 14802]
[New LWP 14803]
[New LWP 14804]
[New LWP 14805]
[New LWP 14806]
[New LWP 14807]
[New LWP 14808]
[New LWP 14809]
[New LWP 14810]
[New LWP 14811]
[New LWP 14812]
[New LWP 14813]
[New LWP 14814]
[New LWP 14815]
[New LWP 14816]
[New LWP 14817]
[New LWP 14818]
[New LWP 14819]
[New LWP 14820]
[New LWP 14821]
[New LWP 14822]
[New LWP 14823]
[New LWP 14824]
[New LWP 14825]
[New LWP 14826]
[New LWP 14827]
[New LWP 14828]
[New LWP 14829]
[New LWP 14830]
[New LWP 14831]
[New LWP 14832]
[New LWP 14833]
[New LWP 14834]
[New LWP 14835]
[New LWP 14836]
[New LWP 14837]
[New LWP 14839]
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
0x00007efda4b05d50 in ?? ()
Id Target Id Frame
* 1 LWP 14704 "kudu" 0x00007efda4b05d50 in ?? ()
2 LWP 14708 "kudu" 0x00007efda4b01fb9 in ?? ()
3 LWP 14709 "kudu" 0x00007efda4b01fb9 in ?? ()
4 LWP 14710 "kudu" 0x00007efda4b01fb9 in ?? ()
5 LWP 14711 "kernel-watcher-" 0x00007efda4b01fb9 in ?? ()
6 LWP 14718 "ntp client-1471" 0x00007efda4b059e2 in ?? ()
7 LWP 14719 "file cache-evic" 0x00007efda4b01fb9 in ?? ()
8 LWP 14720 "sq_acceptor" 0x00007efd9d92ccb9 in ?? ()
9 LWP 14723 "rpc reactor-147" 0x00007efd9d939a47 in ?? ()
10 LWP 14726 "rpc reactor-147" 0x00007efd9d939a47 in ?? ()
11 LWP 14727 "rpc reactor-147" 0x00007efd9d939a47 in ?? ()
12 LWP 14728 "rpc reactor-147" 0x00007efd9d939a47 in ?? ()
13 LWP 14729 "MaintenanceMgr " 0x00007efda4b01ad3 in ?? ()
14 LWP 14730 "txn-status-mana" 0x00007efda4b01fb9 in ?? ()
15 LWP 14731 "collect_and_rem" 0x00007efda4b01fb9 in ?? ()
16 LWP 14732 "tc-session-exp-" 0x00007efda4b01fb9 in ?? ()
17 LWP 14733 "rpc worker-1473" 0x00007efda4b01ad3 in ?? ()
18 LWP 14734 "rpc worker-1473" 0x00007efda4b01ad3 in ?? ()
19 LWP 14735 "rpc worker-1473" 0x00007efda4b01ad3 in ?? ()
20 LWP 14736 "rpc worker-1473" 0x00007efda4b01ad3 in ?? ()
21 LWP 14737 "rpc worker-1473" 0x00007efda4b01ad3 in ?? ()
22 LWP 14738 "rpc worker-1473" 0x00007efda4b01ad3 in ?? ()
23 LWP 14739 "rpc worker-1473" 0x00007efda4b01ad3 in ?? ()
24 LWP 14740 "rpc worker-1474" 0x00007efda4b01ad3 in ?? ()
25 LWP 14741 "rpc worker-1474" 0x00007efda4b01ad3 in ?? ()
26 LWP 14742 "rpc worker-1474" 0x00007efda4b01ad3 in ?? ()
27 LWP 14743 "rpc worker-1474" 0x00007efda4b01ad3 in ?? ()
28 LWP 14744 "rpc worker-1474" 0x00007efda4b01ad3 in ?? ()
29 LWP 14745 "rpc worker-1474" 0x00007efda4b01ad3 in ?? ()
30 LWP 14746 "rpc worker-1474" 0x00007efda4b01ad3 in ?? ()
31 LWP 14747 "rpc worker-1474" 0x00007efda4b01ad3 in ?? ()
32 LWP 14748 "rpc worker-1474" 0x00007efda4b01ad3 in ?? ()
33 LWP 14749 "rpc worker-1474" 0x00007efda4b01ad3 in ?? ()
34 LWP 14750 "rpc worker-1475" 0x00007efda4b01ad3 in ?? ()
35 LWP 14751 "rpc worker-1475" 0x00007efda4b01ad3 in ?? ()
36 LWP 14752 "rpc worker-1475" 0x00007efda4b01ad3 in ?? ()
37 LWP 14753 "rpc worker-1475" 0x00007efda4b01ad3 in ?? ()
38 LWP 14754 "rpc worker-1475" 0x00007efda4b01ad3 in ?? ()
39 LWP 14755 "rpc worker-1475" 0x00007efda4b01ad3 in ?? ()
40 LWP 14756 "rpc worker-1475" 0x00007efda4b01ad3 in ?? ()
41 LWP 14757 "rpc worker-1475" 0x00007efda4b01ad3 in ?? ()
42 LWP 14758 "rpc worker-1475" 0x00007efda4b01ad3 in ?? ()
43 LWP 14759 "rpc worker-1475" 0x00007efda4b01ad3 in ?? ()
44 LWP 14760 "rpc worker-1476" 0x00007efda4b01ad3 in ?? ()
45 LWP 14761 "rpc worker-1476" 0x00007efda4b01ad3 in ?? ()
46 LWP 14762 "rpc worker-1476" 0x00007efda4b01ad3 in ?? ()
47 LWP 14763 "rpc worker-1476" 0x00007efda4b01ad3 in ?? ()
48 LWP 14764 "rpc worker-1476" 0x00007efda4b01ad3 in ?? ()
49 LWP 14765 "rpc worker-1476" 0x00007efda4b01ad3 in ?? ()
50 LWP 14766 "rpc worker-1476" 0x00007efda4b01ad3 in ?? ()
51 LWP 14767 "rpc worker-1476" 0x00007efda4b01ad3 in ?? ()
52 LWP 14768 "rpc worker-1476" 0x00007efda4b01ad3 in ?? ()
53 LWP 14769 "rpc worker-1476" 0x00007efda4b01ad3 in ?? ()
54 LWP 14770 "rpc worker-1477" 0x00007efda4b01ad3 in ?? ()
55 LWP 14771 "rpc worker-1477" 0x00007efda4b01ad3 in ?? ()
56 LWP 14772 "rpc worker-1477" 0x00007efda4b01ad3 in ?? ()
57 LWP 14773 "rpc worker-1477" 0x00007efda4b01ad3 in ?? ()
58 LWP 14774 "rpc worker-1477" 0x00007efda4b01ad3 in ?? ()
59 LWP 14775 "rpc worker-1477" 0x00007efda4b01ad3 in ?? ()
60 LWP 14776 "rpc worker-1477" 0x00007efda4b01ad3 in ?? ()
61 LWP 14777 "rpc worker-1477" 0x00007efda4b01ad3 in ?? ()
62 LWP 14778 "rpc worker-1477" 0x00007efda4b01ad3 in ?? ()
63 LWP 14779 "rpc worker-1477" 0x00007efda4b01ad3 in ?? ()
64 LWP 14780 "rpc worker-1478" 0x00007efda4b01ad3 in ?? ()
65 LWP 14781 "rpc worker-1478" 0x00007efda4b01ad3 in ?? ()
66 LWP 14782 "rpc worker-1478" 0x00007efda4b01ad3 in ?? ()
67 LWP 14783 "rpc worker-1478" 0x00007efda4b01ad3 in ?? ()
68 LWP 14784 "rpc worker-1478" 0x00007efda4b01ad3 in ?? ()
69 LWP 14785 "rpc worker-1478" 0x00007efda4b01ad3 in ?? ()
70 LWP 14786 "rpc worker-1478" 0x00007efda4b01ad3 in ?? ()
71 LWP 14787 "rpc worker-1478" 0x00007efda4b01ad3 in ?? ()
72 LWP 14788 "rpc worker-1478" 0x00007efda4b01ad3 in ?? ()
73 LWP 14789 "rpc worker-1478" 0x00007efda4b01ad3 in ?? ()
74 LWP 14790 "rpc worker-1479" 0x00007efda4b01ad3 in ?? ()
75 LWP 14791 "rpc worker-1479" 0x00007efda4b01ad3 in ?? ()
76 LWP 14792 "rpc worker-1479" 0x00007efda4b01ad3 in ?? ()
77 LWP 14793 "rpc worker-1479" 0x00007efda4b01ad3 in ?? ()
78 LWP 14794 "rpc worker-1479" 0x00007efda4b01ad3 in ?? ()
79 LWP 14795 "rpc worker-1479" 0x00007efda4b01ad3 in ?? ()
80 LWP 14796 "rpc worker-1479" 0x00007efda4b01ad3 in ?? ()
81 LWP 14797 "rpc worker-1479" 0x00007efda4b01ad3 in ?? ()
82 LWP 14798 "rpc worker-1479" 0x00007efda4b01ad3 in ?? ()
83 LWP 14799 "rpc worker-1479" 0x00007efda4b01ad3 in ?? ()
84 LWP 14800 "rpc worker-1480" 0x00007efda4b01ad3 in ?? ()
85 LWP 14801 "rpc worker-1480" 0x00007efda4b01ad3 in ?? ()
86 LWP 14802 "rpc worker-1480" 0x00007efda4b01ad3 in ?? ()
87 LWP 14803 "rpc worker-1480" 0x00007efda4b01ad3 in ?? ()
88 LWP 14804 "rpc worker-1480" 0x00007efda4b01ad3 in ?? ()
89 LWP 14805 "rpc worker-1480" 0x00007efda4b01ad3 in ?? ()
90 LWP 14806 "rpc worker-1480" 0x00007efda4b01ad3 in ?? ()
91 LWP 14807 "rpc worker-1480" 0x00007efda4b01ad3 in ?? ()
92 LWP 14808 "rpc worker-1480" 0x00007efda4b01ad3 in ?? ()
93 LWP 14809 "rpc worker-1480" 0x00007efda4b01ad3 in ?? ()
94 LWP 14810 "rpc worker-1481" 0x00007efda4b01ad3 in ?? ()
95 LWP 14811 "rpc worker-1481" 0x00007efda4b01ad3 in ?? ()
96 LWP 14812 "rpc worker-1481" 0x00007efda4b01ad3 in ?? ()
97 LWP 14813 "rpc worker-1481" 0x00007efda4b01ad3 in ?? ()
98 LWP 14814 "rpc worker-1481" 0x00007efda4b01ad3 in ?? ()
99 LWP 14815 "rpc worker-1481" 0x00007efda4b01ad3 in ?? ()
100 LWP 14816 "rpc worker-1481" 0x00007efda4b01ad3 in ?? ()
101 LWP 14817 "rpc worker-1481" 0x00007efda4b01ad3 in ?? ()
102 LWP 14818 "rpc worker-1481" 0x00007efda4b01ad3 in ?? ()
103 LWP 14819 "rpc worker-1481" 0x00007efda4b01ad3 in ?? ()
104 LWP 14820 "rpc worker-1482" 0x00007efda4b01ad3 in ?? ()
105 LWP 14821 "rpc worker-1482" 0x00007efda4b01ad3 in ?? ()
106 LWP 14822 "rpc worker-1482" 0x00007efda4b01ad3 in ?? ()
107 LWP 14823 "rpc worker-1482" 0x00007efda4b01ad3 in ?? ()
108 LWP 14824 "rpc worker-1482" 0x00007efda4b01ad3 in ?? ()
109 LWP 14825 "rpc worker-1482" 0x00007efda4b01ad3 in ?? ()
110 LWP 14826 "rpc worker-1482" 0x00007efda4b01ad3 in ?? ()
111 LWP 14827 "rpc worker-1482" 0x00007efda4b01ad3 in ?? ()
112 LWP 14828 "rpc worker-1482" 0x00007efda4b01ad3 in ?? ()
113 LWP 14829 "rpc worker-1482" 0x00007efda4b01ad3 in ?? ()
114 LWP 14830 "rpc worker-1483" 0x00007efda4b01ad3 in ?? ()
115 LWP 14831 "rpc worker-1483" 0x00007efda4b01ad3 in ?? ()
116 LWP 14832 "rpc worker-1483" 0x00007efda4b01ad3 in ?? ()
117 LWP 14833 "diag-logger-148" 0x00007efda4b01fb9 in ?? ()
118 LWP 14834 "result-tracker-" 0x00007efda4b01fb9 in ?? ()
119 LWP 14835 "excess-log-dele" 0x00007efda4b01fb9 in ?? ()
120 LWP 14836 "acceptor-14836" 0x00007efd9d93b0c7 in ?? ()
121 LWP 14837 "heartbeat-14837" 0x00007efda4b01fb9 in ?? ()
122 LWP 14839 "maintenance_sch" 0x00007efda4b01fb9 in ?? ()
Thread 122 (LWP 14839):
#0 0x00007efda4b01fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000308 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017b80 in ?? ()
#5 0x00007efd5a4ce550 in ?? ()
#6 0x0000000000000610 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 14837):
#0 0x00007efda4b01fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000000c1 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000018b40 in ?? ()
#5 0x00007efd5ace65b0 in ?? ()
#6 0x0000000000000182 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 14836):
#0 0x00007efd9d93b0c7 in ?? ()
#1 0x00006140000416a8 in ?? ()
#2 0x00007efd5b50dd50 in ?? ()
#3 0x00007efd5b50dd80 in ?? ()
#4 0x00007efd5b50de80 in ?? ()
#5 0x00007efd5b50dd70 in ?? ()
#6 0x00007efd5b50dde0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x0000000000894fab in __sanitizer::theDepot ()
#9 0x00007efda1755da0 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 14835):
#0 0x00007efda4b01fb9 in ?? ()
#1 0x00007efd5bd27f00 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff85b230d0 in ?? ()
#5 0x00007efd5bd27ef0 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 14834):
#0 0x00007efda4b01fb9 in ?? ()
#1 0x00007efd5c5400c0 in ?? ()
#2 0x00000000000000c2 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000861f0 in ?? ()
#5 0x00007efd5c5400b0 in ?? ()
#6 0x0000000000000184 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 14833):
#0 0x00007efda4b01fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 14832):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x000000000000001f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002b004c in ?? ()
#4 0x00007efd5d56fe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007efd5d56fe70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002b0000 in ?? ()
#9 0x00007efda4b01770 in ?? ()
#10 0x00007efd5d56fe70 in ?? ()
#11 0x00000fdfabaadfd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 14831):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000025 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002a685c in ?? ()
#4 0x00007efd5dd87e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007efd5dd87e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0002a6810 in ?? ()
#9 0x00007efda4b01770 in ?? ()
#10 0x00007efd5dd87e70 in ?? ()
#11 0x00000fdfabbb0fd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 14830):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 14829):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 14828):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 14827):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 14826):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 14825):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 14824):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 14823):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 14822):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 14821):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 14820):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 14819):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 14818):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 14817):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 14816):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 14815):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 14814):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 14813):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 14812):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 14811):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 14810):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 14809):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 14808):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 14807):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 14806):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 14805):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 14804):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 14803):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 14802):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 14801):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 14800):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 14799):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 14798):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 14797):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 14796):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 14795):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 14794):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 14793):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 14792):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007efd71930e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007efd71930e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007efda4b01770 in ?? ()
#10 0x00007efd71930e70 in ?? ()
#11 0x00000fdfae3261d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 14791):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 14790):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 14789):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 14788):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 14787):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 14786):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 14785):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 14784):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 14783):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 14782):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 14781):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 14780):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 14779):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 14778):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 14777):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 14776):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 14775):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 14774):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 14773):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 14772):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007efd7b2f9e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007efd7b2f9e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007efda4b01770 in ?? ()
#10 0x00007efd7b2f9e70 in ?? ()
#11 0x00000fdfaf65f3d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 14771):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 14770):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 14769):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 14768):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 14767):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 14766):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 14765):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 14764):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 14763):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 14762):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 14761):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 14760):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 14759):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 14758):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 14757):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 14756):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 14755):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 14754):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 14753):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 14752):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007efd854dae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007efd854dae70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007efda4b01770 in ?? ()
#10 0x00007efd854dae70 in ?? ()
#11 0x00000fdfb0a9b5d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 14751):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 14750):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 14749):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 14748):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 14747):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 14746):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 14745):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 14744):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 14743):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 14742):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 14741):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 14740):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 14739):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 14738):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 14737):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 14736):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 14735):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 14734):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 14733):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 14732):
#0 0x00007efda4b01fb9 in ?? ()
#1 0x00007efd8f5c1c80 in ?? ()
#2 0x0000000000000013 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006130000193e0 in ?? ()
#5 0x00007efd8f5c1c70 in ?? ()
#6 0x0000000000000026 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 14731):
#0 0x00007efda4b01fb9 in ?? ()
#1 0x00007efd8fdc31c0 in ?? ()
#2 0x0000000000000027 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000cc98 in ?? ()
#5 0x00007efd8fdc3210 in ?? ()
#6 0x000000000000004e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 14730):
#0 0x00007efda4b01fb9 in ?? ()
#1 0x00007efd905c31c0 in ?? ()
#2 0x0000000000000013 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000f9a8 in ?? ()
#5 0x00007efd905c31b0 in ?? ()
#6 0x0000000000000026 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 14729):
#0 0x00007efda4b01ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 14728):
#0 0x00007efd9d939a47 in ?? ()
#1 0x00007efd915c62e0 in ?? ()
#2 0x0000619000059180 in ?? ()
#3 0x00007efd915c62d0 in ?? ()
#4 0x00007efd915c64e0 in ?? ()
#5 0x00007efd915c6320 in ?? ()
#6 0x0000614000041a98 in ?? ()
#7 0x00007efd915c63a0 in ?? ()
#8 0x00007efd9ff80de0 in ?? ()
#9 0x3fb952d8ee9fc000 in ?? ()
#10 0x0000619000059180 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059180 in ?? ()
#13 0x421df5ecfa000000 in ?? ()
#14 0x00007efd00000000 in ?? ()
#15 0x41d98e3f99a4af44 in ?? ()
#16 0x00000fe0322b0c74 in ?? ()
#17 0x00007efd915c6380 in ?? ()
#18 0x00007efd9ff8449e in ?? ()
#19 0x00007efd915c6350 in ?? ()
#20 0x3fb952d8ee9fc000 in ?? ()
#21 0x00000000915c63a0 in ?? ()
#22 0x0000619000059180 in ?? ()
#23 0x0000614000041a98 in ?? ()
#24 0x3fb952d8ee9fc000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 14727):
#0 0x00007efd9d939a47 in ?? ()
#1 0x00007efd91dc72e0 in ?? ()
#2 0x0000619000058c80 in ?? ()
#3 0x00007efd91dc72d0 in ?? ()
#4 0x00007efd91dc74e0 in ?? ()
#5 0x00007efd91dc7320 in ?? ()
#6 0x0000614000041898 in ?? ()
#7 0x00007efd91dc73a0 in ?? ()
#8 0x00007efd9ff80de0 in ?? ()
#9 0x3fb974cd4210c000 in ?? ()
#10 0x0000619000058c80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058c80 in ?? ()
#13 0x420cab002e000000 in ?? ()
#14 0x00007efd00000000 in ?? ()
#15 0x41d98e3f99a4af47 in ?? ()
#16 0x00000fe0323b0e74 in ?? ()
#17 0x00007efd91dc7380 in ?? ()
#18 0x00007efd9ff8449e in ?? ()
#19 0x00007efd91dc7350 in ?? ()
#20 0x3fb974cd4210c000 in ?? ()
#21 0x0000000091dc73a0 in ?? ()
#22 0x0000619000058c80 in ?? ()
#23 0x0000614000041898 in ?? ()
#24 0x3fb974cd4210c000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 14726):
#0 0x00007efd9d939a47 in ?? ()
#1 0x00007efd925c82e0 in ?? ()
#2 0x0000619000058780 in ?? ()
#3 0x00007efd925c82d0 in ?? ()
#4 0x00007efd925c84e0 in ?? ()
#5 0x00007efd925c8320 in ?? ()
#6 0x0000614000041698 in ?? ()
#7 0x00007efd925c83a0 in ?? ()
#8 0x00007efd9ff80de0 in ?? ()
#9 0x3fb952d9ab9a4000 in ?? ()
#10 0x0000619000058780 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058780 in ?? ()
#13 0x42111a8ea5000000 in ?? ()
#14 0x00007efd00000000 in ?? ()
#15 0x41d98e3f99a4af45 in ?? ()
#16 0x00000fe0324b1074 in ?? ()
#17 0x00007efd925c8380 in ?? ()
#18 0x00007efd9ff8449e in ?? ()
#19 0x00007efd925c8350 in ?? ()
#20 0x3fb952d9ab9a4000 in ?? ()
#21 0x00000000925c83a0 in ?? ()
#22 0x0000619000058780 in ?? ()
#23 0x0000614000041698 in ?? ()
#24 0x3fb952d9ab9a4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 14723):
#0 0x00007efd9d939a47 in ?? ()
#1 0x00007efd941b32e0 in ?? ()
#2 0x0000619000058280 in ?? ()
#3 0x00007efd941b32d0 in ?? ()
#4 0x00007efd941b34e0 in ?? ()
#5 0x00007efd941b3320 in ?? ()
#6 0x0000614000041498 in ?? ()
#7 0x00007efd941b33a0 in ?? ()
#8 0x00007efd9ff80de0 in ?? ()
#9 0x3fb972e8a5ef4000 in ?? ()
#10 0x0000619000058280 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058280 in ?? ()
#13 0x421c88ab1e000000 in ?? ()
#14 0x00007efd00000000 in ?? ()
#15 0x41d98e3f99a4af45 in ?? ()
#16 0x00000fe03282e674 in ?? ()
#17 0x00007efd941b3380 in ?? ()
#18 0x00007efd9ff8449e in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 14720):
#0 0x00007efd9d92ccb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007efd961b7788 in ?? ()
#3 0x0000602000017cb0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x000000000088cbf1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 14719):
#0 0x00007efda4b01fb9 in ?? ()
#1 0x00007efd959b6080 in ?? ()
#2 0x0000000000000003 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000c2c0 in ?? ()
#5 0x00007efd959b6070 in ?? ()
#6 0x0000000000000006 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 14718):
#0 0x00007efda4b059e2 in ?? ()
#1 0x00007efd949b2b60 in ?? ()
#2 0x00007efd949b2c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007efd949b2bb0 in ?? ()
#6 0x0000000001760060 in ?? ()
#7 0x00007efd949b2c10 in ?? ()
#8 0x0000611000087690 in ?? ()
#9 0x00007efd949b2c00 in ?? ()
#10 0x00000000008870e7 in __sanitizer::theDepot ()
#11 0x00007efdaa46538c in ?? ()
#12 0x00007efdaa454c19 in ?? ()
#13 0x00007efdaa459206 in ?? ()
#14 0x00007efdaa45dc30 in ?? ()
#15 0x00007efdaa45da59 in ?? ()
#16 0x0000000000a60bcd in __sanitizer::theDepot ()
#17 0x00007efda177601e in ?? ()
#18 0x00007efda4afb6db in ?? ()
#19 0x00000fe03292e47c in ?? ()
#20 0x00007efd949b2400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 14711):
#0 0x00007efda4b01fb9 in ?? ()
#1 0x00007efd971b8c40 in ?? ()
#2 0x00000000000003d0 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000d1a8 in ?? ()
#5 0x00007efd971b8c30 in ?? ()
#6 0x00000000000007a0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 14710):
#0 0x00007efda4b01fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000171b8 in ?? ()
#5 0x00007efd979ba410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 3 (LWP 14709):
#0 0x00007efda4b01fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000017038 in ?? ()
#5 0x00007efd981bb410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (LWP 14708):
#0 0x00007efda4b01fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000008 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007efd989bc410 in ?? ()
#6 0x0000000000000010 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 14704):
#0 0x00007efda4b05d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20240506 16:19:16.030762 12982 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 7 with UUID dc9fadb1e71e44c18dbade6135728983 and pid 16492
************************ BEGIN STACKS **************************
I20240506 16:19:16.266680 15899 maintenance_manager.cc:419] P e6d1cddba8c94532b40c0ae9c06740ef: Scheduling FlushMRSOp(5ea58089cc7849fbabaa4bde5c75d45f): perf score=0.078445
[New LWP 16493]
[New LWP 16494]
[New LWP 16495]
[New LWP 16496]
[New LWP 16502]
[New LWP 16503]
[New LWP 16507]
[New LWP 16512]
[New LWP 16513]
[New LWP 16514]
[New LWP 16515]
[New LWP 16516]
[New LWP 16517]
[New LWP 16518]
[New LWP 16519]
[New LWP 16520]
[New LWP 16521]
[New LWP 16522]
[New LWP 16523]
[New LWP 16524]
[New LWP 16525]
[New LWP 16526]
[New LWP 16527]
[New LWP 16528]
[New LWP 16529]
[New LWP 16530]
[New LWP 16531]
[New LWP 16532]
[New LWP 16533]
[New LWP 16534]
[New LWP 16535]
[New LWP 16537]
[New LWP 16538]
[New LWP 16539]
[New LWP 16540]
[New LWP 16541]
[New LWP 16542]
[New LWP 16543]
[New LWP 16544]
[New LWP 16545]
[New LWP 16546]
[New LWP 16547]
[New LWP 16548]
[New LWP 16549]
[New LWP 16550]
[New LWP 16551]
[New LWP 16552]
[New LWP 16553]
[New LWP 16554]
[New LWP 16555]
[New LWP 16556]
[New LWP 16557]
[New LWP 16558]
[New LWP 16559]
[New LWP 16560]
[New LWP 16561]
[New LWP 16562]
[New LWP 16563]
[New LWP 16564]
[New LWP 16565]
[New LWP 16566]
[New LWP 16567]
[New LWP 16568]
[New LWP 16569]
[New LWP 16570]
[New LWP 16571]
[New LWP 16572]
[New LWP 16573]
[New LWP 16574]
[New LWP 16575]
[New LWP 16576]
[New LWP 16578]
[New LWP 16579]
[New LWP 16580]
[New LWP 16581]
[New LWP 16582]
[New LWP 16583]
[New LWP 16584]
[New LWP 16585]
[New LWP 16586]
[New LWP 16587]
[New LWP 16589]
[New LWP 16590]
[New LWP 16591]
[New LWP 16592]
[New LWP 16593]
[New LWP 16594]
[New LWP 16595]
[New LWP 16596]
[New LWP 16597]
[New LWP 16598]
[New LWP 16599]
[New LWP 16600]
[New LWP 16601]
[New LWP 16602]
[New LWP 16603]
[New LWP 16604]
[New LWP 16605]
[New LWP 16606]
[New LWP 16607]
[New LWP 16608]
[New LWP 16609]
[New LWP 16610]
[New LWP 16611]
[New LWP 16612]
[New LWP 16613]
[New LWP 16614]
[New LWP 16615]
[New LWP 16616]
[New LWP 16617]
[New LWP 16618]
[New LWP 16619]
[New LWP 16620]
[New LWP 16621]
[New LWP 16622]
[New LWP 16623]
[New LWP 16624]
[New LWP 16625]
[New LWP 16626]
[New LWP 16627]
[New LWP 16632]
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
0x00007fb5fe08fd50 in ?? ()
Id Target Id Frame
* 1 LWP 16492 "kudu" 0x00007fb5fe08fd50 in ?? ()
2 LWP 16493 "kudu" 0x00007fb5fe08bfb9 in ?? ()
3 LWP 16494 "kudu" 0x00007fb5fe08bfb9 in ?? ()
4 LWP 16495 "kudu" 0x00007fb5fe08bfb9 in ?? ()
5 LWP 16496 "kernel-watcher-" 0x00007fb5fe08bfb9 in ?? ()
6 LWP 16502 "ntp client-1650" 0x00007fb5fe08f9e2 in ?? ()
7 LWP 16503 "file cache-evic" 0x00007fb5fe08bfb9 in ?? ()
8 LWP 16507 "sq_acceptor" 0x00007fb5f6eb6cb9 in ?? ()
9 LWP 16512 "rpc reactor-165" 0x00007fb5f6ec3a47 in ?? ()
10 LWP 16513 "rpc reactor-165" 0x00007fb5f6ec3a47 in ?? ()
11 LWP 16514 "rpc reactor-165" 0x00007fb5f6ec3a47 in ?? ()
12 LWP 16515 "rpc reactor-165" 0x00007fb5f6ec3a47 in ?? ()
13 LWP 16516 "MaintenanceMgr " 0x00007fb5fe08bad3 in ?? ()
14 LWP 16517 "txn-status-mana" 0x00007fb5fe08bfb9 in ?? ()
15 LWP 16518 "collect_and_rem" 0x00007fb5fe08bfb9 in ?? ()
16 LWP 16519 "tc-session-exp-" 0x00007fb5fe08bfb9 in ?? ()
17 LWP 16520 "rpc worker-1652" 0x00007fb5fe08bad3 in ?? ()
18 LWP 16521 "rpc worker-1652" 0x00007fb5fe08bad3 in ?? ()
19 LWP 16522 "rpc worker-1652" 0x00007fb5fe08bad3 in ?? ()
20 LWP 16523 "rpc worker-1652" 0x00007fb5fe08bad3 in ?? ()
21 LWP 16524 "rpc worker-1652" 0x00007fb5fe08bad3 in ?? ()
22 LWP 16525 "rpc worker-1652" 0x00007fb5fe08bad3 in ?? ()
23 LWP 16526 "rpc worker-1652" 0x00007fb5fe08bad3 in ?? ()
24 LWP 16527 "rpc worker-1652" 0x00007fb5fe08bad3 in ?? ()
25 LWP 16528 "rpc worker-1652" 0x00007fb5fe08bad3 in ?? ()
26 LWP 16529 "rpc worker-1652" 0x00007fb5fe08bad3 in ?? ()
27 LWP 16530 "rpc worker-1653" 0x00007fb5fe08bad3 in ?? ()
28 LWP 16531 "rpc worker-1653" 0x00007fb5fe08bad3 in ?? ()
29 LWP 16532 "rpc worker-1653" 0x00007fb5fe08bad3 in ?? ()
30 LWP 16533 "rpc worker-1653" 0x00007fb5fe08bad3 in ?? ()
31 LWP 16534 "rpc worker-1653" 0x00007fb5fe08bad3 in ?? ()
32 LWP 16535 "rpc worker-1653" 0x00007fb5fe08bad3 in ?? ()
33 LWP 16537 "rpc worker-1653" 0x00007fb5fe08bad3 in ?? ()
34 LWP 16538 "rpc worker-1653" 0x00007fb5fe08bad3 in ?? ()
35 LWP 16539 "rpc worker-1653" 0x00007fb5fe08bad3 in ?? ()
36 LWP 16540 "rpc worker-1654" 0x00007fb5fe08bad3 in ?? ()
37 LWP 16541 "rpc worker-1654" 0x00007fb5fe08bad3 in ?? ()
38 LWP 16542 "rpc worker-1654" 0x00007fb5fe08bad3 in ?? ()
39 LWP 16543 "rpc worker-1654" 0x00007fb5fe08bad3 in ?? ()
40 LWP 16544 "rpc worker-1654" 0x00007fb5fe08bad3 in ?? ()
41 LWP 16545 "rpc worker-1654" 0x00007fb5fe08bad3 in ?? ()
42 LWP 16546 "rpc worker-1654" 0x00007fb5fe08bad3 in ?? ()
43 LWP 16547 "rpc worker-1654" 0x00007fb5fe08bad3 in ?? ()
44 LWP 16548 "rpc worker-1654" 0x00007fb5fe08bad3 in ?? ()
45 LWP 16549 "rpc worker-1654" 0x00007fb5fe08bad3 in ?? ()
46 LWP 16550 "rpc worker-1655" 0x00007fb5fe08bad3 in ?? ()
47 LWP 16551 "rpc worker-1655" 0x00007fb5fe08bad3 in ?? ()
48 LWP 16552 "rpc worker-1655" 0x00007fb5fe08bad3 in ?? ()
49 LWP 16553 "rpc worker-1655" 0x00007fb5fe08bad3 in ?? ()
50 LWP 16554 "rpc worker-1655" 0x00007fb5fe08bad3 in ?? ()
51 LWP 16555 "rpc worker-1655" 0x00007fb5fe08bad3 in ?? ()
52 LWP 16556 "rpc worker-1655" 0x00007fb5fe08bad3 in ?? ()
53 LWP 16557 "rpc worker-1655" 0x00007fb5fe08bad3 in ?? ()
54 LWP 16558 "rpc worker-1655" 0x00007fb5fe08bad3 in ?? ()
55 LWP 16559 "rpc worker-1655" 0x00007fb5fe08bad3 in ?? ()
56 LWP 16560 "rpc worker-1656" 0x00007fb5fe08bad3 in ?? ()
57 LWP 16561 "rpc worker-1656" 0x00007fb5fe08bad3 in ?? ()
58 LWP 16562 "rpc worker-1656" 0x00007fb5fe08bad3 in ?? ()
59 LWP 16563 "rpc worker-1656" 0x00007fb5fe08bad3 in ?? ()
60 LWP 16564 "rpc worker-1656" 0x00007fb5fe08bad3 in ?? ()
61 LWP 16565 "rpc worker-1656" 0x00007fb5fe08bad3 in ?? ()
62 LWP 16566 "rpc worker-1656" 0x00007fb5fe08bad3 in ?? ()
63 LWP 16567 "rpc worker-1656" 0x00007fb5fe08bad3 in ?? ()
64 LWP 16568 "rpc worker-1656" 0x00007fb5fe08bad3 in ?? ()
65 LWP 16569 "rpc worker-1656" 0x00007fb5fe08bad3 in ?? ()
66 LWP 16570 "rpc worker-1657" 0x00007fb5fe08bad3 in ?? ()
67 LWP 16571 "rpc worker-1657" 0x00007fb5fe08bad3 in ?? ()
68 LWP 16572 "rpc worker-1657" 0x00007fb5fe08bad3 in ?? ()
69 LWP 16573 "rpc worker-1657" 0x00007fb5fe08bad3 in ?? ()
70 LWP 16574 "rpc worker-1657" 0x00007fb5fe08bad3 in ?? ()
71 LWP 16575 "rpc worker-1657" 0x00007fb5fe08bad3 in ?? ()
72 LWP 16576 "rpc worker-1657" 0x00007fb5fe08bad3 in ?? ()
73 LWP 16578 "rpc worker-1657" 0x00007fb5fe08bad3 in ?? ()
74 LWP 16579 "rpc worker-1657" 0x00007fb5fe08bad3 in ?? ()
75 LWP 16580 "rpc worker-1658" 0x00007fb5fe08bad3 in ?? ()
76 LWP 16581 "rpc worker-1658" 0x00007fb5fe08bad3 in ?? ()
77 LWP 16582 "rpc worker-1658" 0x00007fb5fe08bad3 in ?? ()
78 LWP 16583 "rpc worker-1658" 0x00007fb5fe08bad3 in ?? ()
79 LWP 16584 "rpc worker-1658" 0x00007fb5fe08bad3 in ?? ()
80 LWP 16585 "rpc worker-1658" 0x00007fb5fe08bad3 in ?? ()
81 LWP 16586 "rpc worker-1658" 0x00007fb5fe08bad3 in ?? ()
82 LWP 16587 "rpc worker-1658" 0x00007fb5fe08bad3 in ?? ()
83 LWP 16589 "rpc worker-1658" 0x00007fb5fe08bad3 in ?? ()
84 LWP 16590 "rpc worker-1659" 0x00007fb5fe08bad3 in ?? ()
85 LWP 16591 "rpc worker-1659" 0x00007fb5fe08bad3 in ?? ()
86 LWP 16592 "rpc worker-1659" 0x00007fb5fe08bad3 in ?? ()
87 LWP 16593 "rpc worker-1659" 0x00007fb5fe08bad3 in ?? ()
88 LWP 16594 "rpc worker-1659" 0x00007fb5fe08bad3 in ?? ()
89 LWP 16595 "rpc worker-1659" 0x00007fb5fe08bad3 in ?? ()
90 LWP 16596 "rpc worker-1659" 0x00007fb5fe08bad3 in ?? ()
91 LWP 16597 "rpc worker-1659" 0x00007fb5fe08bad3 in ?? ()
92 LWP 16598 "rpc worker-1659" 0x00007fb5fe08bad3 in ?? ()
93 LWP 16599 "rpc worker-1659" 0x00007fb5fe08bad3 in ?? ()
94 LWP 16600 "rpc worker-1660" 0x00007fb5fe08bad3 in ?? ()
95 LWP 16601 "rpc worker-1660" 0x00007fb5fe08bad3 in ?? ()
96 LWP 16602 "rpc worker-1660" 0x00007fb5fe08bad3 in ?? ()
97 LWP 16603 "rpc worker-1660" 0x00007fb5fe08bad3 in ?? ()
98 LWP 16604 "rpc worker-1660" 0x00007fb5fe08bad3 in ?? ()
99 LWP 16605 "rpc worker-1660" 0x00007fb5fe08bad3 in ?? ()
100 LWP 16606 "rpc worker-1660" 0x00007fb5fe08bad3 in ?? ()
101 LWP 16607 "rpc worker-1660" 0x00007fb5fe08bad3 in ?? ()
102 LWP 16608 "rpc worker-1660" 0x00007fb5fe08bad3 in ?? ()
103 LWP 16609 "rpc worker-1660" 0x00007fb5fe08bad3 in ?? ()
104 LWP 16610 "rpc worker-1661" 0x00007fb5fe08bad3 in ?? ()
105 LWP 16611 "rpc worker-1661" 0x00007fb5fe08bad3 in ?? ()
106 LWP 16612 "rpc worker-1661" 0x00007fb5fe08bad3 in ?? ()
107 LWP 16613 "rpc worker-1661" 0x00007fb5fe08bad3 in ?? ()
108 LWP 16614 "rpc worker-1661" 0x00007fb5fe08bad3 in ?? ()
109 LWP 16615 "rpc worker-1661" 0x00007fb5fe08bad3 in ?? ()
110 LWP 16616 "rpc worker-1661" 0x00007fb5fe08bad3 in ?? ()
111 LWP 16617 "rpc worker-1661" 0x00007fb5fe08bad3 in ?? ()
112 LWP 16618 "rpc worker-1661" 0x00007fb5fe08bad3 in ?? ()
113 LWP 16619 "rpc worker-1661" 0x00007fb5fe08bad3 in ?? ()
114 LWP 16620 "rpc worker-1662" 0x00007fb5fe08bad3 in ?? ()
115 LWP 16621 "rpc worker-1662" 0x00007fb5fe08bad3 in ?? ()
116 LWP 16622 "rpc worker-1662" 0x00007fb5fe08bad3 in ?? ()
117 LWP 16623 "diag-logger-166" 0x00007fb5fe08bfb9 in ?? ()
118 LWP 16624 "result-tracker-" 0x00007fb5fe08bfb9 in ?? ()
119 LWP 16625 "excess-log-dele" 0x00007fb5fe08bfb9 in ?? ()
120 LWP 16626 "acceptor-16626" 0x00007fb5f6ec50c7 in ?? ()
121 LWP 16627 "heartbeat-16627" 0x00007fb5fe08bfb9 in ?? ()
122 LWP 16632 "maintenance_sch" 0x00007fb5fe08bfb9 in ?? ()
Thread 122 (LWP 16632):
#0 0x00007fb5fe08bfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000137 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017b80 in ?? ()
#5 0x00007fb5b3249550 in ?? ()
#6 0x000000000000026e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 16627):
#0 0x00007fb5fe08bfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000004d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000018b40 in ?? ()
#5 0x00007fb5b42705b0 in ?? ()
#6 0x000000000000009a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 16626):
#0 0x00007fb5f6ec50c7 in ?? ()
#1 0x0000614000021ea8 in ?? ()
#2 0x00007fb5b4a97d50 in ?? ()
#3 0x00007fb5b4a97d80 in ?? ()
#4 0x00007fb5b4a97e80 in ?? ()
#5 0x00007fb5b4a97d70 in ?? ()
#6 0x00007fb5b4a97de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x0000000000894fab in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 16625):
#0 0x00007fb5fe08bfb9 in ?? ()
#1 0x00007fb5b52b1f00 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffe4b2d95d0 in ?? ()
#5 0x00007fb5b52b1ef0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 16624):
#0 0x00007fb5fe08bfb9 in ?? ()
#1 0x00007fb5b5aca0c0 in ?? ()
#2 0x000000000000004d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000861f0 in ?? ()
#5 0x00007fb5b5aca0b0 in ?? ()
#6 0x000000000000009a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 16623):
#0 0x00007fb5fe08bfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 16622):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000014 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007fb5b6af9e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb5b6af9e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 16621):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x000000000000000c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002a6858 in ?? ()
#4 0x00007fb5b7311e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb5b7311e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 16620):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 16619):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 16618):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 16617):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 16616):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 16615):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 16614):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 16613):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 16612):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 16611):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 16610):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 16609):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 16608):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 16607):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 16606):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 16605):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 16604):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 16603):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 16602):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 16601):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 16600):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 16599):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 16598):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 16597):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 16596):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 16595):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 16594):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 16593):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 16592):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 16591):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 16590):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 16589):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 16587):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 16586):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 16585):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 16584):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 16583):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 16582):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 16581):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007fb5caebae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb5caebae70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 16580):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 16579):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 16578):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 16576):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 16575):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 16574):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 16573):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 16572):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 16571):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 16570):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 16569):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 16568):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 16567):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 16566):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 16565):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 16564):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 16563):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 16562):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 16561):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 16560):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007fb5d509be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb5d509be70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 16559):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 16558):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 16557):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 16556):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 16555):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 16554):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 16553):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 16552):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 16551):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 16550):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 16549):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 16548):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 16547):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 16546):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 16545):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 16544):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 16543):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 16542):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 16541):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 16540):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007fb5df28ce50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fb5df28ce70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 16539):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 16538):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 16537):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 16535):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 16534):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 16533):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 16532):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 16531):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 16530):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 16529):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 16528):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 16527):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 16526):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 16525):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 16524):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 16523):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 16522):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 16521):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 16520):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 16519):
#0 0x00007fb5fe08bfb9 in ?? ()
#1 0x00007fb5e8c79c80 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006130000193e0 in ?? ()
#5 0x00007fb5e8c79c70 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 16518):
#0 0x00007fb5fe08bfb9 in ?? ()
#1 0x00007fb5e94a11c0 in ?? ()
#2 0x000000000000000f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000cc98 in ?? ()
#5 0x00007fb5e94a1210 in ?? ()
#6 0x000000000000001e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 16517):
#0 0x00007fb5fe08bfb9 in ?? ()
#1 0x00007fb5e9caf1c0 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000f9a8 in ?? ()
#5 0x00007fb5e9caf1b0 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 16516):
#0 0x00007fb5fe08bad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 16515):
#0 0x00007fb5f6ec3a47 in ?? ()
#1 0x00007fb5eacb22e0 in ?? ()
#2 0x0000619000059180 in ?? ()
#3 0x00007fb5eacb22d0 in ?? ()
#4 0x00007fb5eacb24e0 in ?? ()
#5 0x00007fb5eacb2320 in ?? ()
#6 0x0000614000021e98 in ?? ()
#7 0x00007fb5eacb23a0 in ?? ()
#8 0x00007fb5f950ade0 in ?? ()
#9 0x3fb96398263f0000 in ?? ()
#10 0x0000619000059180 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059180 in ?? ()
#13 0x421a7f3f3b000000 in ?? ()
#14 0x00007fb500000000 in ?? ()
#15 0x41d98e3f99a4af44 in ?? ()
#16 0x00000ff73d58e474 in ?? ()
#17 0x00007fb5eacb2380 in ?? ()
#18 0x00007fb5f950e49e in ?? ()
#19 0x00007fb5eacb2350 in ?? ()
#20 0x3fb96398263f0000 in ?? ()
#21 0x00000000eacb23a0 in ?? ()
#22 0x0000619000059180 in ?? ()
#23 0x0000614000021e98 in ?? ()
#24 0x3fb96398263f0000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 16514):
#0 0x00007fb5f6ec3a47 in ?? ()
#1 0x00007fb5eb4b32e0 in ?? ()
#2 0x0000619000058c80 in ?? ()
#3 0x00007fb5eb4b32d0 in ?? ()
#4 0x00007fb5eb4b34e0 in ?? ()
#5 0x00007fb5eb4b3320 in ?? ()
#6 0x0000614000021c98 in ?? ()
#7 0x00007fb5eb4b33a0 in ?? ()
#8 0x00007fb5f950ade0 in ?? ()
#9 0x3fb97a3ecab94000 in ?? ()
#10 0x0000619000058c80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058c80 in ?? ()
#13 0x42288a35d9800000 in ?? ()
#14 0x00007fb500000000 in ?? ()
#15 0x41d98e3f99a4af49 in ?? ()
#16 0x00000ff73d68e674 in ?? ()
#17 0x00007fb5eb4b3380 in ?? ()
#18 0x00007fb5f950e49e in ?? ()
#19 0x00007fb5eb4b3350 in ?? ()
#20 0x3fb97a3ecab94000 in ?? ()
#21 0x00000000eb4b33a0 in ?? ()
#22 0x0000619000058c80 in ?? ()
#23 0x0000614000021c98 in ?? ()
#24 0x3fb97a3ecab94000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 16513):
#0 0x00007fb5f6ec3a47 in ?? ()
#1 0x00007fb5edcad2e0 in ?? ()
#2 0x0000619000058780 in ?? ()
#3 0x00007fb5edcad2d0 in ?? ()
#4 0x00007fb5edcad4e0 in ?? ()
#5 0x00007fb5edcad320 in ?? ()
#6 0x0000614000021a98 in ?? ()
#7 0x00007fb5edcad3a0 in ?? ()
#8 0x00007fb5f950ade0 in ?? ()
#9 0x3fb9821d25dc8000 in ?? ()
#10 0x0000619000058780 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058780 in ?? ()
#13 0x422379edb4800000 in ?? ()
#14 0x00007fb500000000 in ?? ()
#15 0x41d98e3f99a4af48 in ?? ()
#16 0x00000ff73db8da74 in ?? ()
#17 0x00007fb5edcad380 in ?? ()
#18 0x00007fb5f950e49e in ?? ()
#19 0x00007fb5edcad350 in ?? ()
#20 0x3fb9821d25dc8000 in ?? ()
#21 0x00000000edcad3a0 in ?? ()
#22 0x0000619000058780 in ?? ()
#23 0x0000614000021a98 in ?? ()
#24 0x3fb9821d25dc8000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 16512):
#0 0x00007fb5f6ec3a47 in ?? ()
#1 0x00007fb5ebcb42e0 in ?? ()
#2 0x0000619000058280 in ?? ()
#3 0x00007fb5ebcb42d0 in ?? ()
#4 0x00007fb5ebcb44e0 in ?? ()
#5 0x00007fb5ebcb4320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007fb5ebcb43a0 in ?? ()
#8 0x00007fb5f950ade0 in ?? ()
#9 0x3fb95f278f644000 in ?? ()
#10 0x0000619000058280 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058280 in ?? ()
#13 0x420462fb92000000 in ?? ()
#14 0x00007fb500000000 in ?? ()
#15 0x41d98e3f99a4af46 in ?? ()
#16 0x00000ff73d78e874 in ?? ()
#17 0x00007fb5ebcb4380 in ?? ()
#18 0x00007fb5f950e49e in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 16507):
#0 0x00007fb5f6eb6cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fb5ef6b7788 in ?? ()
#3 0x0000602000017cb0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x000000000088cbf1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 16503):
#0 0x00007fb5fe08bfb9 in ?? ()
#1 0x00007fb5eeeb6080 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000c2c0 in ?? ()
#5 0x00007fb5eeeb6070 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 16502):
#0 0x00007fb5fe08f9e2 in ?? ()
#1 0x00007fb5ee6b3b60 in ?? ()
#2 0x00007fb5ee6b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fb5ee6b3bb0 in ?? ()
#6 0x0000000001760060 in ?? ()
#7 0x00007fb5ee6b3c10 in ?? ()
#8 0x0000611000087690 in ?? ()
#9 0x00007fb5ee6b3c00 in ?? ()
#10 0x00000000008870e7 in __sanitizer::theDepot ()
#11 0x00007fb6039ef38c in ?? ()
#12 0x00007fb6039dec19 in ?? ()
#13 0x00007fb6039e3206 in ?? ()
#14 0x00007fb6039e7c30 in ?? ()
#15 0x00007fb6039e7a59 in ?? ()
#16 0x0000000000a60bcd in __sanitizer::theDepot ()
#17 0x00007fb5fad0001e in ?? ()
#18 0x00007fb5fe0856db in ?? ()
#19 0x00000ff73dcce67c in ?? ()
#20 0x00007fb5ee6b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 16496):
#0 0x00007fb5fe08bfb9 in ?? ()
#1 0x00007fb5f06b8c40 in ?? ()
#2 0x000000000000018c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000d1a8 in ?? ()
#5 0x00007fb5f06b8c30 in ?? ()
#6 0x0000000000000318 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 16495):
#0 0x00007fb5fe08bfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000004 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000171b8 in ?? ()
#5 0x00007fb5f0eba410 in ?? ()
#6 0x0000000000000008 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 3 (LWP 16494):
#0 0x00007fb5fe08bfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000004 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000017038 in ?? ()
#5 0x00007fb5f16bb410 in ?? ()
#6 0x0000000000000008 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (LWP 16493):
#0 0x00007fb5fe08bfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000004 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fb5f1ebc410 in ?? ()
#6 0x0000000000000008 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 16492):
#0 0x00007fb5fe08fd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20240506 16:19:16.876022 12982 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 8 with UUID 5beda4231be140b8959d934a86e21bb1 and pid 16640
************************ BEGIN STACKS **************************
[New LWP 16642]
[New LWP 16643]
[New LWP 16644]
[New LWP 16646]
[New LWP 16652]
[New LWP 16653]
[New LWP 16654]
[New LWP 16659]
[New LWP 16660]
[New LWP 16661]
[New LWP 16662]
[New LWP 16663]
[New LWP 16664]
[New LWP 16666]
[New LWP 16667]
[New LWP 16668]
[New LWP 16669]
[New LWP 16670]
[New LWP 16671]
[New LWP 16672]
[New LWP 16673]
[New LWP 16674]
[New LWP 16675]
[New LWP 16676]
[New LWP 16677]
[New LWP 16678]
[New LWP 16679]
[New LWP 16680]
[New LWP 16681]
[New LWP 16682]
[New LWP 16683]
[New LWP 16684]
[New LWP 16685]
[New LWP 16686]
[New LWP 16687]
[New LWP 16688]
[New LWP 16689]
[New LWP 16690]
[New LWP 16691]
[New LWP 16692]
[New LWP 16693]
[New LWP 16694]
[New LWP 16695]
[New LWP 16696]
[New LWP 16697]
[New LWP 16698]
[New LWP 16699]
[New LWP 16700]
[New LWP 16701]
[New LWP 16702]
[New LWP 16703]
[New LWP 16704]
[New LWP 16705]
[New LWP 16706]
[New LWP 16708]
[New LWP 16709]
[New LWP 16710]
[New LWP 16711]
[New LWP 16712]
[New LWP 16713]
[New LWP 16714]
[New LWP 16715]
[New LWP 16716]
[New LWP 16717]
[New LWP 16718]
[New LWP 16719]
[New LWP 16720]
[New LWP 16721]
[New LWP 16722]
[New LWP 16723]
[New LWP 16724]
[New LWP 16725]
[New LWP 16726]
[New LWP 16727]
[New LWP 16728]
[New LWP 16729]
[New LWP 16730]
[New LWP 16731]
[New LWP 16732]
[New LWP 16734]
[New LWP 16735]
[New LWP 16736]
[New LWP 16737]
[New LWP 16738]
[New LWP 16739]
[New LWP 16740]
[New LWP 16741]
[New LWP 16742]
[New LWP 16743]
[New LWP 16744]
[New LWP 16745]
[New LWP 16746]
[New LWP 16747]
[New LWP 16748]
[New LWP 16749]
[New LWP 16750]
[New LWP 16751]
[New LWP 16752]
[New LWP 16753]
[New LWP 16754]
[New LWP 16755]
[New LWP 16756]
[New LWP 16757]
[New LWP 16758]
[New LWP 16761]
[New LWP 16762]
[New LWP 16763]
[New LWP 16764]
[New LWP 16765]
[New LWP 16766]
[New LWP 16767]
[New LWP 16768]
[New LWP 16769]
[New LWP 16770]
[New LWP 16771]
[New LWP 16772]
[New LWP 16773]
[New LWP 16774]
[New LWP 16775]
[New LWP 16776]
[New LWP 16778]
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656958
Cannot access memory at address 0x202c3e6563656950
0x00007faa46de0d50 in ?? ()
Id Target Id Frame
* 1 LWP 16640 "kudu" 0x00007faa46de0d50 in ?? ()
2 LWP 16642 "kudu" 0x00007faa46ddcfb9 in ?? ()
3 LWP 16643 "kudu" 0x00007faa46ddcfb9 in ?? ()
4 LWP 16644 "kudu" 0x00007faa46ddcfb9 in ?? ()
5 LWP 16646 "kernel-watcher-" 0x00007faa46ddcfb9 in ?? ()
6 LWP 16652 "ntp client-1665" 0x00007faa46de09e2 in ?? ()
7 LWP 16653 "file cache-evic" 0x00007faa46ddcfb9 in ?? ()
8 LWP 16654 "sq_acceptor" 0x00007faa3fc07cb9 in ?? ()
9 LWP 16659 "rpc reactor-166" 0x00007faa3fc14a47 in ?? ()
10 LWP 16660 "rpc reactor-166" 0x00007faa3fc14a47 in ?? ()
11 LWP 16661 "rpc reactor-166" 0x00007faa3fc14a47 in ?? ()
12 LWP 16662 "rpc reactor-166" 0x00007faa3fc14a47 in ?? ()
13 LWP 16663 "MaintenanceMgr " 0x00007faa46ddcad3 in ?? ()
14 LWP 16664 "txn-status-mana" 0x00007faa46ddcfb9 in ?? ()
15 LWP 16666 "collect_and_rem" 0x00007faa46ddcfb9 in ?? ()
16 LWP 16667 "tc-session-exp-" 0x00007faa46ddcfb9 in ?? ()
17 LWP 16668 "rpc worker-1666" 0x00007faa46ddcad3 in ?? ()
18 LWP 16669 "rpc worker-1666" 0x00007faa46ddcad3 in ?? ()
19 LWP 16670 "rpc worker-1667" 0x00007faa46ddcad3 in ?? ()
20 LWP 16671 "rpc worker-1667" 0x00007faa46ddcad3 in ?? ()
21 LWP 16672 "rpc worker-1667" 0x00007faa46ddcad3 in ?? ()
22 LWP 16673 "rpc worker-1667" 0x00007faa46ddcad3 in ?? ()
23 LWP 16674 "rpc worker-1667" 0x00007faa46ddcad3 in ?? ()
24 LWP 16675 "rpc worker-1667" 0x00007faa46ddcad3 in ?? ()
25 LWP 16676 "rpc worker-1667" 0x00007faa46ddcad3 in ?? ()
26 LWP 16677 "rpc worker-1667" 0x00007faa46ddcad3 in ?? ()
27 LWP 16678 "rpc worker-1667" 0x00007faa46ddcad3 in ?? ()
28 LWP 16679 "rpc worker-1667" 0x00007faa46ddcad3 in ?? ()
29 LWP 16680 "rpc worker-1668" 0x00007faa46ddcad3 in ?? ()
30 LWP 16681 "rpc worker-1668" 0x00007faa46ddcad3 in ?? ()
31 LWP 16682 "rpc worker-1668" 0x00007faa46ddcad3 in ?? ()
32 LWP 16683 "rpc worker-1668" 0x00007faa46ddcad3 in ?? ()
33 LWP 16684 "rpc worker-1668" 0x00007faa46ddcad3 in ?? ()
34 LWP 16685 "rpc worker-1668" 0x00007faa46ddcad3 in ?? ()
35 LWP 16686 "rpc worker-1668" 0x00007faa46ddcad3 in ?? ()
36 LWP 16687 "rpc worker-1668" 0x00007faa46ddcad3 in ?? ()
37 LWP 16688 "rpc worker-1668" 0x00007faa46ddcad3 in ?? ()
38 LWP 16689 "rpc worker-1668" 0x00007faa46ddcad3 in ?? ()
39 LWP 16690 "rpc worker-1669" 0x00007faa46ddcad3 in ?? ()
40 LWP 16691 "rpc worker-1669" 0x00007faa46ddcad3 in ?? ()
41 LWP 16692 "rpc worker-1669" 0x00007faa46ddcad3 in ?? ()
42 LWP 16693 "rpc worker-1669" 0x00007faa46ddcad3 in ?? ()
43 LWP 16694 "rpc worker-1669" 0x00007faa46ddcad3 in ?? ()
44 LWP 16695 "rpc worker-1669" 0x00007faa46ddcad3 in ?? ()
45 LWP 16696 "rpc worker-1669" 0x00007faa46ddcad3 in ?? ()
46 LWP 16697 "rpc worker-1669" 0x00007faa46ddcad3 in ?? ()
47 LWP 16698 "rpc worker-1669" 0x00007faa46ddcad3 in ?? ()
48 LWP 16699 "rpc worker-1669" 0x00007faa46ddcad3 in ?? ()
49 LWP 16700 "rpc worker-1670" 0x00007faa46ddcad3 in ?? ()
50 LWP 16701 "rpc worker-1670" 0x00007faa46ddcad3 in ?? ()
51 LWP 16702 "rpc worker-1670" 0x00007faa46ddcad3 in ?? ()
52 LWP 16703 "rpc worker-1670" 0x00007faa46ddcad3 in ?? ()
53 LWP 16704 "rpc worker-1670" 0x00007faa46ddcad3 in ?? ()
54 LWP 16705 "rpc worker-1670" 0x00007faa46ddcad3 in ?? ()
55 LWP 16706 "rpc worker-1670" 0x00007faa46ddcad3 in ?? ()
56 LWP 16708 "rpc worker-1670" 0x00007faa46ddcad3 in ?? ()
57 LWP 16709 "rpc worker-1670" 0x00007faa46ddcad3 in ?? ()
58 LWP 16710 "rpc worker-1671" 0x00007faa46ddcad3 in ?? ()
59 LWP 16711 "rpc worker-1671" 0x00007faa46ddcad3 in ?? ()
60 LWP 16712 "rpc worker-1671" 0x00007faa46ddcad3 in ?? ()
61 LWP 16713 "rpc worker-1671" 0x00007faa46ddcad3 in ?? ()
62 LWP 16714 "rpc worker-1671" 0x00007faa46ddcad3 in ?? ()
63 LWP 16715 "rpc worker-1671" 0x00007faa46ddcad3 in ?? ()
64 LWP 16716 "rpc worker-1671" 0x00007faa46ddcad3 in ?? ()
65 LWP 16717 "rpc worker-1671" 0x00007faa46ddcad3 in ?? ()
66 LWP 16718 "rpc worker-1671" 0x00007faa46ddcad3 in ?? ()
67 LWP 16719 "rpc worker-1671" 0x00007faa46ddcad3 in ?? ()
68 LWP 16720 "rpc worker-1672" 0x00007faa46ddcad3 in ?? ()
69 LWP 16721 "rpc worker-1672" 0x00007faa46ddcad3 in ?? ()
70 LWP 16722 "rpc worker-1672" 0x00007faa46ddcad3 in ?? ()
71 LWP 16723 "rpc worker-1672" 0x00007faa46ddcad3 in ?? ()
72 LWP 16724 "rpc worker-1672" 0x00007faa46ddcad3 in ?? ()
73 LWP 16725 "rpc worker-1672" 0x00007faa46ddcad3 in ?? ()
74 LWP 16726 "rpc worker-1672" 0x00007faa46ddcad3 in ?? ()
75 LWP 16727 "rpc worker-1672" 0x00007faa46ddcad3 in ?? ()
76 LWP 16728 "rpc worker-1672" 0x00007faa46ddcad3 in ?? ()
77 LWP 16729 "rpc worker-1672" 0x00007faa46ddcad3 in ?? ()
78 LWP 16730 "rpc worker-1673" 0x00007faa46ddcad3 in ?? ()
79 LWP 16731 "rpc worker-1673" 0x00007faa46ddcad3 in ?? ()
80 LWP 16732 "rpc worker-1673" 0x00007faa46ddcad3 in ?? ()
81 LWP 16734 "rpc worker-1673" 0x00007faa46ddcad3 in ?? ()
82 LWP 16735 "rpc worker-1673" 0x00007faa46ddcad3 in ?? ()
83 LWP 16736 "rpc worker-1673" 0x00007faa46ddcad3 in ?? ()
84 LWP 16737 "rpc worker-1673" 0x00007faa46ddcad3 in ?? ()
85 LWP 16738 "rpc worker-1673" 0x00007faa46ddcad3 in ?? ()
86 LWP 16739 "rpc worker-1673" 0x00007faa46ddcad3 in ?? ()
87 LWP 16740 "rpc worker-1674" 0x00007faa46ddcad3 in ?? ()
88 LWP 16741 "rpc worker-1674" 0x00007faa46ddcad3 in ?? ()
89 LWP 16742 "rpc worker-1674" 0x00007faa46ddcad3 in ?? ()
90 LWP 16743 "rpc worker-1674" 0x00007faa46ddcad3 in ?? ()
91 LWP 16744 "rpc worker-1674" 0x00007faa46ddcad3 in ?? ()
92 LWP 16745 "rpc worker-1674" 0x00007faa46ddcad3 in ?? ()
93 LWP 16746 "rpc worker-1674" 0x00007faa46ddcad3 in ?? ()
94 LWP 16747 "rpc worker-1674" 0x00007faa46ddcad3 in ?? ()
95 LWP 16748 "rpc worker-1674" 0x00007faa46ddcad3 in ?? ()
96 LWP 16749 "rpc worker-1674" 0x00007faa46ddcad3 in ?? ()
97 LWP 16750 "rpc worker-1675" 0x00007faa46ddcad3 in ?? ()
98 LWP 16751 "rpc worker-1675" 0x00007faa46ddcad3 in ?? ()
99 LWP 16752 "rpc worker-1675" 0x00007faa46ddcad3 in ?? ()
100 LWP 16753 "rpc worker-1675" 0x00007faa46ddcad3 in ?? ()
101 LWP 16754 "rpc worker-1675" 0x00007faa46ddcad3 in ?? ()
102 LWP 16755 "rpc worker-1675" 0x00007faa46ddcad3 in ?? ()
103 LWP 16756 "rpc worker-1675" 0x00007faa46ddcad3 in ?? ()
104 LWP 16757 "rpc worker-1675" 0x00007faa46ddcad3 in ?? ()
105 LWP 16758 "rpc worker-1675" 0x00007faa46ddcad3 in ?? ()
106 LWP 16761 "rpc worker-1676" 0x00007faa46ddcad3 in ?? ()
107 LWP 16762 "rpc worker-1676" 0x00007faa46ddcad3 in ?? ()
108 LWP 16763 "rpc worker-1676" 0x00007faa46ddcad3 in ?? ()
109 LWP 16764 "rpc worker-1676" 0x00007faa46ddcad3 in ?? ()
110 LWP 16765 "rpc worker-1676" 0x00007faa46ddcad3 in ?? ()
111 LWP 16766 "rpc worker-1676" 0x00007faa46ddcad3 in ?? ()
112 LWP 16767 "rpc worker-1676" 0x00007faa46ddcad3 in ?? ()
113 LWP 16768 "rpc worker-1676" 0x00007faa46ddcad3 in ?? ()
114 LWP 16769 "rpc worker-1676" 0x00007faa46ddcad3 in ?? ()
115 LWP 16770 "rpc worker-1677" 0x00007faa46ddcad3 in ?? ()
116 LWP 16771 "rpc worker-1677" 0x00007faa46ddcad3 in ?? ()
117 LWP 16772 "diag-logger-167" 0x00007faa46ddcfb9 in ?? ()
118 LWP 16773 "result-tracker-" 0x00007faa46ddcfb9 in ?? ()
119 LWP 16774 "excess-log-dele" 0x00007faa46ddcfb9 in ?? ()
120 LWP 16775 "acceptor-16775" 0x00007faa3fc160c7 in ?? ()
121 LWP 16776 "heartbeat-16776" 0x00007faa46ddcfb9 in ?? ()
122 LWP 16778 "maintenance_sch" 0x00007faa46ddcfb9 in ?? ()
Thread 122 (LWP 16778):
#0 0x00007faa46ddcfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000012d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017b80 in ?? ()
#5 0x00007fa9fb550550 in ?? ()
#6 0x000000000000025a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 16776):
#0 0x00007faa46ddcfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000004c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000018b40 in ?? ()
#5 0x00007fa9fc5775b0 in ?? ()
#6 0x0000000000000098 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 16775):
#0 0x00007faa3fc160c7 in ?? ()
#1 0x00006140000418a8 in ?? ()
#2 0x00007fa9fcd8ed50 in ?? ()
#3 0x00007fa9fcd8ed80 in ?? ()
#4 0x00007fa9fcd8ee80 in ?? ()
#5 0x00007fa9fcd8ed70 in ?? ()
#6 0x00007fa9fcd8ede0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x0000000000894fab in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 16774):
#0 0x00007faa46ddcfb9 in ?? ()
#1 0x00007fa9fd5a7f00 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffdb6ae1910 in ?? ()
#5 0x00007fa9fd5a7ef0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 16773):
#0 0x00007faa46ddcfb9 in ?? ()
#1 0x00007fa9fddd00c0 in ?? ()
#2 0x000000000000004b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000861f0 in ?? ()
#5 0x00007fa9fddd00b0 in ?? ()
#6 0x0000000000000096 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 16772):
#0 0x00007faa46ddcfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 16771):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000014 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0003dffe8 in ?? ()
#4 0x00007fa9fee7ee50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa9fee7ee70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 16770):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x000000000000000c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0003cfff8 in ?? ()
#4 0x00007fa9ff697e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fa9ff697e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 16769):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 16768):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 16767):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 16766):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 16765):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 16764):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 16763):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 16762):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 16761):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 16758):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 16757):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 16756):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 16755):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 16754):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 16753):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 16752):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 16751):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 16750):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 16749):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 16748):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 16747):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 16746):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 16745):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 16744):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 16743):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 16742):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 16741):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 16740):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 16739):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 16738):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 16737):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 16736):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 16735):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 16734):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 16732):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 16731):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 16730):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 16729):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 16728):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000290 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001d0058 in ?? ()
#4 0x00007faa133b3e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007faa133b3e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 16727):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000004 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001cd068 in ?? ()
#4 0x00007faa13bcbe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007faa13bcbe70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 16726):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 16725):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 16724):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 16723):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 16722):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 16721):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 16720):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 16719):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 16718):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 16717):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 16716):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 16715):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 16714):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 16713):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 16712):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 16711):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 16710):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 16709):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 16708):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000146818 in ?? ()
#4 0x00007faa1bd4ce50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007faa1bd4ce70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 16706):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 16705):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 16704):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 16703):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 16702):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 16701):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 16700):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 16699):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 16698):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 16697):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 16696):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 16695):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 16694):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 16693):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 16692):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 16691):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 16690):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 16689):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 16688):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 16687):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x000000000000000c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000b67d8 in ?? ()
#4 0x00007faa25f2de50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007faa25f2de70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 16686):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 16685):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 16684):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 16683):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 16682):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 16681):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 16680):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 16679):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 16678):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 16677):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 16676):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 16675):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 16674):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 16673):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 16672):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 16671):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 16670):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 16669):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 16668):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 16667):
#0 0x00007faa46ddcfb9 in ?? ()
#1 0x00007faa3012ec80 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019ae0 in ?? ()
#5 0x00007faa3012ec70 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 16666):
#0 0x00007faa46ddcfb9 in ?? ()
#1 0x00007faa309761c0 in ?? ()
#2 0x000000000000000f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000cc98 in ?? ()
#5 0x00007faa30976210 in ?? ()
#6 0x000000000000001e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 16664):
#0 0x00007faa46ddcfb9 in ?? ()
#1 0x00007faa319ad1c0 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000f9a8 in ?? ()
#5 0x00007faa319ad1b0 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 16663):
#0 0x00007faa46ddcad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 16662):
#0 0x00007faa3fc14a47 in ?? ()
#1 0x00007faa329b02e0 in ?? ()
#2 0x0000619000059180 in ?? ()
#3 0x00007faa329b02d0 in ?? ()
#4 0x00007faa329b04e0 in ?? ()
#5 0x00007faa329b0320 in ?? ()
#6 0x0000614000041c98 in ?? ()
#7 0x00007faa329b03a0 in ?? ()
#8 0x00007faa4225bde0 in ?? ()
#9 0x3fb97d9113db8000 in ?? ()
#10 0x0000619000059180 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059180 in ?? ()
#13 0x421474dd35000000 in ?? ()
#14 0x00007faa00000000 in ?? ()
#15 0x41d98e3f99a4af47 in ?? ()
#16 0x00000ff5c652e074 in ?? ()
#17 0x00007faa329b0380 in ?? ()
#18 0x00007faa4225f49e in ?? ()
#19 0x00007faa329b0350 in ?? ()
#20 0x3fb97d9113db8000 in ?? ()
#21 0x00000000329b03a0 in ?? ()
#22 0x0000619000059180 in ?? ()
#23 0x0000614000041c98 in ?? ()
#24 0x3fb97d9113db8000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 16661):
#0 0x00007faa3fc14a47 in ?? ()
#1 0x00007faa331b12e0 in ?? ()
#2 0x0000619000058c80 in ?? ()
#3 0x00007faa331b12d0 in ?? ()
#4 0x00007faa331b14e0 in ?? ()
#5 0x00007faa331b1320 in ?? ()
#6 0x0000614000041a98 in ?? ()
#7 0x00007faa331b13a0 in ?? ()
#8 0x00007faa4225bde0 in ?? ()
#9 0x3fa9e28045b10000 in ?? ()
#10 0x0000619000058c80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058c80 in ?? ()
#13 0x425258aacd400000 in ?? ()
#14 0x00007faa00000000 in ?? ()
#15 0x41d98e3f99a4af45 in ?? ()
#16 0x00000ff5c662e274 in ?? ()
#17 0x00007faa331b1380 in ?? ()
#18 0x00007faa4225f49e in ?? ()
#19 0x00007faa331b1350 in ?? ()
#20 0x3fa9e28045b10000 in ?? ()
#21 0x00000000331b13a0 in ?? ()
#22 0x0000619000058c80 in ?? ()
#23 0x0000614000041a98 in ?? ()
#24 0x3fa9e28045b10000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 16660):
#0 0x00007faa3fc14a47 in ?? ()
#1 0x00007faa339b22e0 in ?? ()
#2 0x0000619000058780 in ?? ()
#3 0x00007faa339b22d0 in ?? ()
#4 0x00007faa339b24e0 in ?? ()
#5 0x00007faa339b2320 in ?? ()
#6 0x0000614000041898 in ?? ()
#7 0x00007faa339b23a0 in ?? ()
#8 0x00007faa4225bde0 in ?? ()
#9 0x3fb993c0915dc000 in ?? ()
#10 0x0000619000058780 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058780 in ?? ()
#13 0x4230e9fabb400000 in ?? ()
#14 0x00007faa00000000 in ?? ()
#15 0x41d98e3f99a4af45 in ?? ()
#16 0x00000ff5c672e474 in ?? ()
#17 0x00007faa339b2380 in ?? ()
#18 0x00007faa4225f49e in ?? ()
#19 0x00007faa339b2350 in ?? ()
#20 0x3fb993c0915dc000 in ?? ()
#21 0x00000000339b23a0 in ?? ()
#22 0x0000619000058780 in ?? ()
#23 0x0000614000041898 in ?? ()
#24 0x3fb993c0915dc000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 16659):
#0 0x00007faa3fc14a47 in ?? ()
#1 0x00007faa359ab2e0 in ?? ()
#2 0x0000619000058280 in ?? ()
#3 0x00007faa359ab2d0 in ?? ()
#4 0x00007faa359ab4e0 in ?? ()
#5 0x00007faa359ab320 in ?? ()
#6 0x0000614000041698 in ?? ()
#7 0x00007faa359ab3a0 in ?? ()
#8 0x00007faa4225bde0 in ?? ()
#9 0x3fb96f547aa30000 in ?? ()
#10 0x0000619000058280 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058280 in ?? ()
#13 0x4220087ebb800000 in ?? ()
#14 0x00007faa00000000 in ?? ()
#15 0x41d98e3f99a4af46 in ?? ()
#16 0x00000ff5c6b2d674 in ?? ()
#17 0x00007faa359ab380 in ?? ()
#18 0x00007faa4225f49e in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 16654):
#0 0x00007faa3fc07cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007faa383b7788 in ?? ()
#3 0x0000602000017c70 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x000000000088cbf1 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 16653):
#0 0x00007faa46ddcfb9 in ?? ()
#1 0x00007faa37bb6080 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000c2c0 in ?? ()
#5 0x00007faa37bb6070 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 16652):
#0 0x00007faa46de09e2 in ?? ()
#1 0x00007faa373b3b60 in ?? ()
#2 0x00007faa373b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007faa373b3bb0 in ?? ()
#6 0x0000000001760060 in ?? ()
#7 0x00007faa373b3c10 in ?? ()
#8 0x0000611000087690 in ?? ()
#9 0x00007faa373b3c00 in ?? ()
#10 0x00000000008870e7 in __sanitizer::theDepot ()
#11 0x00007faa4c74038c in ?? ()
#12 0x00007faa4c72fc19 in ?? ()
#13 0x00007faa4c734206 in ?? ()
#14 0x00007faa4c738c30 in ?? ()
#15 0x00007faa4c738a59 in ?? ()
#16 0x0000000000a60bcd in __sanitizer::theDepot ()
#17 0x00007faa43a5101e in ?? ()
#18 0x00007faa46dd66db in ?? ()
#19 0x00000ff5c6e6e67c in ?? ()
#20 0x00007faa373b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 16646):
#0 0x00007faa46ddcfb9 in ?? ()
#1 0x00007faa393b8c40 in ?? ()
#2 0x0000000000000181 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000d1a8 in ?? ()
#5 0x00007faa393b8c30 in ?? ()
#6 0x0000000000000302 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 16644):
#0 0x00007faa46ddcfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000004 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006120000171b8 in ?? ()
#5 0x00007faa39bba410 in ?? ()
#6 0x0000000000000008 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 3 (LWP 16643):
#0 0x00007faa46ddcfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000004 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000017038 in ?? ()
#5 0x00007faa3a3bb410 in ?? ()
#6 0x0000000000000008 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (LWP 16642):
#0 0x00007faa46ddcfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000004 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007faa3abbc410 in ?? ()
#6 0x0000000000000008 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 16640):
#0 0x00007faa46de0d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20240506 16:19:17.714442 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 15195
I20240506 16:19:17.727197 15323 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:19:17.987459 17362 raft_consensus.cc:482] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 15538fc3b6084a96a18216a2ec74cc3a)
I20240506 16:19:17.987681 17362 raft_consensus.cc:504] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:19:17.994824 17362 leader_election.cc:288] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195), 5beda4231be140b8959d934a86e21bb1 (127.12.173.137:37831)
I20240506 16:19:18.010454 16728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" candidate_term: 5 candidate_status { last_received { term: 4 index: 15315 } } ignore_live_leader: false dest_uuid: "5beda4231be140b8959d934a86e21bb1" is_pre_election: true
I20240506 16:19:18.015102 17366 raft_consensus.cc:482] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 15538fc3b6084a96a18216a2ec74cc3a)
I20240506 16:19:18.015376 17366 raft_consensus.cc:504] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:19:18.031147 17366 leader_election.cc:288] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195), e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055)
I20240506 16:19:18.038391 15848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "5beda4231be140b8959d934a86e21bb1" candidate_term: 5 candidate_status { last_received { term: 4 index: 15315 } } ignore_live_leader: false dest_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" is_pre_election: true
I20240506 16:19:18.038662 15848 raft_consensus.cc:2454] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5beda4231be140b8959d934a86e21bb1 in term 4.
I20240506 16:19:18.039218 16661 leader_election.cc:302] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beda4231be140b8959d934a86e21bb1, e6d1cddba8c94532b40c0ae9c06740ef; no voters:
I20240506 16:19:18.039675 17366 raft_consensus.cc:2781] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240506 16:19:18.039845 17366 raft_consensus.cc:482] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 15538fc3b6084a96a18216a2ec74cc3a)
I20240506 16:19:18.039940 17366 raft_consensus.cc:3037] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 4 FOLLOWER]: Advancing to term 5
I20240506 16:19:18.045454 17366 raft_consensus.cc:504] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:19:18.046341 17366 leader_election.cc:288] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [CANDIDATE]: Term 5 election: Requested vote from peers 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195), e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055)
I20240506 16:19:18.046953 15848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea58089cc7849fbabaa4bde5c75d45f" candidate_uuid: "5beda4231be140b8959d934a86e21bb1" candidate_term: 5 candidate_status { last_received { term: 4 index: 15315 } } ignore_live_leader: false dest_uuid: "e6d1cddba8c94532b40c0ae9c06740ef"
I20240506 16:19:18.047180 15848 raft_consensus.cc:3037] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 4 FOLLOWER]: Advancing to term 5
I20240506 16:19:18.050386 15848 raft_consensus.cc:2454] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5beda4231be140b8959d934a86e21bb1 in term 5.
I20240506 16:19:18.050869 16661 leader_election.cc:302] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5beda4231be140b8959d934a86e21bb1, e6d1cddba8c94532b40c0ae9c06740ef; no voters:
I20240506 16:19:18.051160 17366 raft_consensus.cc:2781] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 5 FOLLOWER]: Leader election won for term 5
I20240506 16:19:18.051685 17366 raft_consensus.cc:686] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [term 5 LEADER]: Becoming Leader. State: Replica: 5beda4231be140b8959d934a86e21bb1, State: Running, Role: LEADER
I20240506 16:19:18.052301 17366 consensus_queue.cc:237] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15315, Committed index: 15315, Last appended: 4.15315, Last appended by leader: 15315, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } }
I20240506 16:19:18.056641 13021 catalog_manager.cc:5484] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 reported cstate change: term changed from 4 to 5, leader changed from 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129) to 5beda4231be140b8959d934a86e21bb1 (127.12.173.137). New cstate: current_term: 5 leader_uuid: "5beda4231be140b8959d934a86e21bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15538fc3b6084a96a18216a2ec74cc3a" member_type: VOTER last_known_addr { host: "127.12.173.129" port: 43195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5beda4231be140b8959d934a86e21bb1" member_type: VOTER last_known_addr { host: "127.12.173.137" port: 37831 } health_report { overall_health: HEALTHY } } }
I20240506 16:19:18.127604 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 15195
I20240506 16:19:18.138224 15848 raft_consensus.cc:1261] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 5 FOLLOWER]: Refusing update from remote peer 5beda4231be140b8959d934a86e21bb1: Log matching property violated. Preceding OpId in replica: term: 4 index: 15315. Preceding OpId from leader: term: 5 index: 15316. (index mismatch)
I20240506 16:19:18.139051 17366 consensus_queue.cc:1035] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6d1cddba8c94532b40c0ae9c06740ef" member_type: VOTER last_known_addr { host: "127.12.173.132" port: 39055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15316, Last known committed idx: 15315, Time since last communication: 0.000s
I20240506 16:19:18.216703 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 15054
W20240506 16:19:18.216871 17364 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 16:19:17.988934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.173.129:43195 (local address 127.12.173.132:58011)
0506 16:19:18.003116 (+ 14182us) negotiation.cc:107] Waiting for socket to connect
0506 16:19:18.003141 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0506 16:19:18.003330 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 16:19:18.215811 (+212481us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":14042,"thread_start_us":14490,"threads_started":1}
W20240506 16:19:18.216890 17367 negotiation.cc:337] Failed RPC negotiation. Trace:
0506 16:19:18.017797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.173.129:43195 (local address 127.12.173.137:51377)
0506 16:19:18.030531 (+ 12734us) negotiation.cc:107] Waiting for socket to connect
0506 16:19:18.030556 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0506 16:19:18.030748 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0506 16:19:18.215832 (+185084us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":12585,"thread_start_us":13041,"threads_started":1}
W20240506 16:19:18.217348 15782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 16:19:18.218261 16660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240506 16:19:18.230360 16660 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240506 16:19:18.231788 16660 leader_election.cc:334] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: connect: Connection refused (error 111)
W20240506 16:19:18.232285 16660 leader_election.cc:334] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: connect: Connection refused (error 111)
W20240506 16:19:18.233819 15782 leader_election.cc:334] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: connect: Connection refused (error 111)
I20240506 16:19:18.233958 15782 leader_election.cc:302] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6d1cddba8c94532b40c0ae9c06740ef; no voters: 15538fc3b6084a96a18216a2ec74cc3a, 5beda4231be140b8959d934a86e21bb1
I20240506 16:19:18.234230 17362 raft_consensus.cc:2726] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240506 16:19:18.238934 15178 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:19:18.374609 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 15054
W20240506 16:19:18.411550 13006 connection.cc:535] server connection from 127.12.173.130:60165 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 16:19:18.411857 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 15338
I20240506 16:19:18.422163 15463 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:19:18.594062 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 15338
I20240506 16:19:18.620103 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 15764
I20240506 16:19:18.627646 15891 generic_service.cc:196] Checking for leaks (request via RPC)
W20240506 16:19:18.646126 16660 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240506 16:19:19.104601 16660 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240506 16:19:19.142999 17388 raft_consensus.cc:482] T 5ea58089cc7849fbabaa4bde5c75d45f P e6d1cddba8c94532b40c0ae9c06740ef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5beda4231be140b8959d934a86e21bb1)
I20240506 16:19:19.143235 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 15764
I20240506 16:19:19.227298 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 14565
W20240506 16:19:19.228147 16661 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.132:39055: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240506 16:19:19.234671 14691 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:19:19.368659 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 14565
I20240506 16:19:19.396255 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 14425
I20240506 16:19:19.403184 14553 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:19:19.535053 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 14425
I20240506 16:19:19.562644 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 14704
I20240506 16:19:19.569686 14831 generic_service.cc:196] Checking for leaks (request via RPC)
W20240506 16:19:19.587720 16660 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer 15538fc3b6084a96a18216a2ec74cc3a (127.12.173.129:43195): Couldn't send request to peer 15538fc3b6084a96a18216a2ec74cc3a. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.129:43195: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240506 16:19:19.671017 16661 consensus_peers.cc:485] T 5ea58089cc7849fbabaa4bde5c75d45f P 5beda4231be140b8959d934a86e21bb1 -> Peer e6d1cddba8c94532b40c0ae9c06740ef (127.12.173.132:39055): Couldn't send request to peer e6d1cddba8c94532b40c0ae9c06740ef. Status: Network error: Client connection negotiation failed: client connection to 127.12.173.132:39055: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240506 16:19:19.705801 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 14704
I20240506 16:19:19.731731 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 16492
I20240506 16:19:19.738839 16622 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:19:19.843379 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 16492
I20240506 16:19:19.867089 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 16640
I20240506 16:19:19.873699 16771 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:19:20.290233 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 16640
W20240506 16:19:20.347851 13008 connection.cc:535] server connection from 127.12.173.137:52189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240506 16:19:20.348234 12982 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGIku5t/build/asan/bin/kudu pid 12990
I20240506 16:19:20.355432 13051 generic_service.cc:196] Checking for leaks (request via RPC)
I20240506 16:19:20.552155 12982 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGIku5t/build/asan/bin/kudu with pid 12990
2024-05-06T16:19:20Z chronyd exiting
I20240506 16:19:20.607257 12982 test_util.cc:175] -----------------------------------------------
I20240506 16:19:20.607348 12982 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGIku5t/test-tmp/maintenance_mode-itest.7.RollingRestartArgs_RollingRestartITest.TestWorkloads_3.1715012143134278-12982-0
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3, where GetParam() = 800-byte object <01-00 00-00 01-61 73-61 09-00 00-00 BE-BE BE-BE A0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-1F 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 03-00 00-00 03-00 00-00 00-BE BE-BE BE-BE BE-BE> (217401 ms)
[----------] 1 test from RollingRestartArgs/RollingRestartITest (217401 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (217401 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/3, where GetParam() = 800-byte object <01-00 00-00 01-61 73-61 09-00 00-00 BE-BE BE-BE A0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-1C 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-1F 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 03-00 00-00 03-00 00-00 00-BE BE-BE BE-BE BE-BE>
1 FAILED TEST