Note: This is test shard 5 of 8.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from RollingRestartArgs/RollingRestartITest
[ RUN ] RollingRestartArgs/RollingRestartITest.TestWorkloads/0
2025-01-13T08:54:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-13T08:54:30Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250113 08:54:30.695688 11762 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskqePfU0/build/asan/bin/kudu
/tmp/dist-test-taskqePfU0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.124.190:36259
--webserver_interface=127.11.124.190
--webserver_port=0
--builtin_ntp_servers=127.11.124.148:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.124.190:36259
--location_mapping_cmd=/tmp/dist-test-taskqePfU0/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/location-assignment.state --map /L0:4
--master_client_location_assignment_enabled=false with env {}
W20250113 08:54:31.030704 11770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:54:31.031035 11770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:54:31.031126 11770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:54:31.039032 11770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250113 08:54:31.039165 11770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:54:31.039211 11770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250113 08:54:31.039268 11770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250113 08:54:31.048904 11770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.124.148:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/wal
--location_mapping_cmd=/tmp/dist-test-taskqePfU0/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/location-assignment.state --map /L0:4
--ipki_ca_key_size=768
--master_addresses=127.11.124.190:36259
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.124.190:36259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.124.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:54:31.050536 11770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:54:31.052255 11770 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:54:31.063855 11775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:31.063872 11776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:31.064085 11778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:31.064755 11770 server_base.cc:1034] running on GCE node
I20250113 08:54:31.065559 11770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:54:31.067008 11770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:54:31.068230 11770 hybrid_clock.cc:648] HybridClock initialized: now 1736758471068170 us; error 48 us; skew 500 ppm
I20250113 08:54:31.068580 11770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:54:31.071172 11770 webserver.cc:458] Webserver started at http://127.11.124.190:44245/ using document root <none> and password file <none>
I20250113 08:54:31.072043 11770 fs_manager.cc:362] Metadata directory not provided
I20250113 08:54:31.072223 11770 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:54:31.072585 11770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:54:31.075356 11770 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/data/instance:
uuid: "a4ab4eabb3c943d9bcd33a4ccc2938a4"
format_stamp: "Formatted at 2025-01-13 08:54:31 on dist-test-slave-kk7k"
I20250113 08:54:31.076211 11770 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/wal/instance:
uuid: "a4ab4eabb3c943d9bcd33a4ccc2938a4"
format_stamp: "Formatted at 2025-01-13 08:54:31 on dist-test-slave-kk7k"
I20250113 08:54:31.081652 11770 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20250113 08:54:31.085255 11784 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:31.086678 11770 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250113 08:54:31.086879 11770 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/data,/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/wal
uuid: "a4ab4eabb3c943d9bcd33a4ccc2938a4"
format_stamp: "Formatted at 2025-01-13 08:54:31 on dist-test-slave-kk7k"
I20250113 08:54:31.087070 11770 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:54:31.102164 11770 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:54:31.103103 11770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:54:31.103418 11770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:54:31.122223 11770 rpc_server.cc:307] RPC server started. Bound to: 127.11.124.190:36259
I20250113 08:54:31.122246 11835 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.124.190:36259 every 8 connection(s)
I20250113 08:54:31.123685 11770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/data/info.pb
I20250113 08:54:31.124626 11762 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskqePfU0/build/asan/bin/kudu as pid 11770
I20250113 08:54:31.124962 11762 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/wal/instance
I20250113 08:54:31.127825 11836 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:31.139225 11836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4: Bootstrap starting.
I20250113 08:54:31.142783 11836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:31.143867 11836 log.cc:826] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4: Log is configured to *not* fsync() on all Append() calls
I20250113 08:54:31.147037 11836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4: No bootstrap required, opened a new log
I20250113 08:54:31.154429 11836 raft_consensus.cc:357] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ab4eabb3c943d9bcd33a4ccc2938a4" member_type: VOTER last_known_addr { host: "127.11.124.190" port: 36259 } }
I20250113 08:54:31.154956 11836 raft_consensus.cc:383] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:31.155061 11836 raft_consensus.cc:738] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ab4eabb3c943d9bcd33a4ccc2938a4, State: Initialized, Role: FOLLOWER
I20250113 08:54:31.155661 11836 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ab4eabb3c943d9bcd33a4ccc2938a4" member_type: VOTER last_known_addr { host: "127.11.124.190" port: 36259 } }
I20250113 08:54:31.155876 11836 raft_consensus.cc:397] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 08:54:31.156001 11836 raft_consensus.cc:491] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 08:54:31.156167 11836 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:31.158330 11836 raft_consensus.cc:513] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ab4eabb3c943d9bcd33a4ccc2938a4" member_type: VOTER last_known_addr { host: "127.11.124.190" port: 36259 } }
I20250113 08:54:31.158954 11836 leader_election.cc:304] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4ab4eabb3c943d9bcd33a4ccc2938a4; no voters:
I20250113 08:54:31.159662 11836 leader_election.cc:290] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 08:54:31.159813 11841 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:54:31.161278 11841 raft_consensus.cc:695] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [term 1 LEADER]: Becoming Leader. State: Replica: a4ab4eabb3c943d9bcd33a4ccc2938a4, State: Running, Role: LEADER
I20250113 08:54:31.161798 11841 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ab4eabb3c943d9bcd33a4ccc2938a4" member_type: VOTER last_known_addr { host: "127.11.124.190" port: 36259 } }
I20250113 08:54:31.161939 11836 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [sys.catalog]: configured and running, proceeding with master startup.
I20250113 08:54:31.165318 11843 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4ab4eabb3c943d9bcd33a4ccc2938a4. Latest consensus state: current_term: 1 leader_uuid: "a4ab4eabb3c943d9bcd33a4ccc2938a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ab4eabb3c943d9bcd33a4ccc2938a4" member_type: VOTER last_known_addr { host: "127.11.124.190" port: 36259 } } }
I20250113 08:54:31.165498 11842 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4ab4eabb3c943d9bcd33a4ccc2938a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ab4eabb3c943d9bcd33a4ccc2938a4" member_type: VOTER last_known_addr { host: "127.11.124.190" port: 36259 } } }
I20250113 08:54:31.165912 11842 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [sys.catalog]: This master's current role is: LEADER
I20250113 08:54:31.165774 11843 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4 [sys.catalog]: This master's current role is: LEADER
I20250113 08:54:31.168761 11848 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 08:54:31.175266 11848 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 08:54:31.185143 11848 catalog_manager.cc:1348] Generated new cluster ID: 0d65945293554386b3398c7c39414022
I20250113 08:54:31.185290 11848 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 08:54:31.206542 11848 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 08:54:31.207867 11848 catalog_manager.cc:1505] Loading token signing keys...
I20250113 08:54:31.226014 11848 catalog_manager.cc:5899] T 00000000000000000000000000000000 P a4ab4eabb3c943d9bcd33a4ccc2938a4: Generated new TSK 0
I20250113 08:54:31.226928 11848 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 08:54:31.235863 11762 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskqePfU0/build/asan/bin/kudu
/tmp/dist-test-taskqePfU0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.124.129:0
--local_ip_for_outbound_sockets=127.11.124.129
--webserver_interface=127.11.124.129
--webserver_port=0
--tserver_master_addrs=127.11.124.190:36259
--builtin_ntp_servers=127.11.124.148:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:54:31.619294 11860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:54:31.619705 11860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:54:31.619863 11860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:54:31.628188 11860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:54:31.628473 11860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.124.129
I20250113 08:54:31.638258 11860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.124.148:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.124.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.124.129
--webserver_port=0
--tserver_master_addrs=127.11.124.190:36259
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.124.129
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:54:31.639969 11860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:54:31.641779 11860 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:54:31.653712 11865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:31.653754 11866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:31.653888 11868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:31.655550 11860 server_base.cc:1034] running on GCE node
I20250113 08:54:31.656270 11860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:54:31.657267 11860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:54:31.658572 11860 hybrid_clock.cc:648] HybridClock initialized: now 1736758471658466 us; error 89 us; skew 500 ppm
I20250113 08:54:31.658903 11860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:54:31.665132 11860 webserver.cc:458] Webserver started at http://127.11.124.129:38221/ using document root <none> and password file <none>
I20250113 08:54:31.666018 11860 fs_manager.cc:362] Metadata directory not provided
I20250113 08:54:31.666162 11860 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:54:31.666477 11860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:54:31.669334 11860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data/instance:
uuid: "4a1af392ea8543f4827f17fdff2eaaac"
format_stamp: "Formatted at 2025-01-13 08:54:31 on dist-test-slave-kk7k"
I20250113 08:54:31.670121 11860 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/wal/instance:
uuid: "4a1af392ea8543f4827f17fdff2eaaac"
format_stamp: "Formatted at 2025-01-13 08:54:31 on dist-test-slave-kk7k"
I20250113 08:54:31.675623 11860 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250113 08:54:31.679194 11874 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:31.680788 11860 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250113 08:54:31.680959 11860 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/wal
uuid: "4a1af392ea8543f4827f17fdff2eaaac"
format_stamp: "Formatted at 2025-01-13 08:54:31 on dist-test-slave-kk7k"
I20250113 08:54:31.681164 11860 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:54:31.702456 11860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:54:31.703346 11860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:54:31.703655 11860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:54:31.704694 11860 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:54:31.706296 11860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:54:31.706380 11860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:31.706527 11860 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:54:31.706585 11860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:31.738708 11860 rpc_server.cc:307] RPC server started. Bound to: 127.11.124.129:40671
I20250113 08:54:31.738752 11986 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.124.129:40671 every 8 connection(s)
I20250113 08:54:31.740013 11860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data/info.pb
I20250113 08:54:31.749643 11762 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskqePfU0/build/asan/bin/kudu as pid 11860
I20250113 08:54:31.749897 11762 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/wal/instance
I20250113 08:54:31.754127 11762 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskqePfU0/build/asan/bin/kudu
/tmp/dist-test-taskqePfU0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.124.130:0
--local_ip_for_outbound_sockets=127.11.124.130
--webserver_interface=127.11.124.130
--webserver_port=0
--tserver_master_addrs=127.11.124.190:36259
--builtin_ntp_servers=127.11.124.148:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250113 08:54:31.754434 11987 heartbeater.cc:346] Connected to a master server at 127.11.124.190:36259
I20250113 08:54:31.754774 11987 heartbeater.cc:463] Registering TS with master...
I20250113 08:54:31.755597 11987 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:54:31.804301 11801 ts_manager.cc:194] Registered new tserver with Master: 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671)
I20250113 08:54:31.806876 11801 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.124.129:37763
W20250113 08:54:32.108341 11991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:54:32.108687 11991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:54:32.108790 11991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:54:32.116925 11991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:54:32.117173 11991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.124.130
I20250113 08:54:32.126973 11991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.124.148:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.124.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.124.130
--webserver_port=0
--tserver_master_addrs=127.11.124.190:36259
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.124.130
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:54:32.128719 11991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:54:32.130615 11991 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:54:32.143525 11998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:32.143599 11997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:32.144131 12000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:32.144879 11991 server_base.cc:1034] running on GCE node
I20250113 08:54:32.145534 11991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:54:32.146561 11991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:54:32.147809 11991 hybrid_clock.cc:648] HybridClock initialized: now 1736758472147741 us; error 38 us; skew 500 ppm
I20250113 08:54:32.148200 11991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:54:32.150641 11991 webserver.cc:458] Webserver started at http://127.11.124.130:44445/ using document root <none> and password file <none>
I20250113 08:54:32.151438 11991 fs_manager.cc:362] Metadata directory not provided
I20250113 08:54:32.151559 11991 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:54:32.151878 11991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:54:32.154749 11991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data/instance:
uuid: "cbb5dc448319472395e744e7bd16c57f"
format_stamp: "Formatted at 2025-01-13 08:54:32 on dist-test-slave-kk7k"
I20250113 08:54:32.155537 11991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/wal/instance:
uuid: "cbb5dc448319472395e744e7bd16c57f"
format_stamp: "Formatted at 2025-01-13 08:54:32 on dist-test-slave-kk7k"
I20250113 08:54:32.161048 11991 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20250113 08:54:32.164645 12006 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:32.166054 11991 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250113 08:54:32.166211 11991 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/wal
uuid: "cbb5dc448319472395e744e7bd16c57f"
format_stamp: "Formatted at 2025-01-13 08:54:32 on dist-test-slave-kk7k"
I20250113 08:54:32.166424 11991 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:54:32.184369 11991 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:54:32.185293 11991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:54:32.185599 11991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:54:32.186640 11991 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:54:32.188355 11991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:54:32.188433 11991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:32.188555 11991 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:54:32.188603 11991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:32.221521 11991 rpc_server.cc:307] RPC server started. Bound to: 127.11.124.130:46163
I20250113 08:54:32.221614 12118 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.124.130:46163 every 8 connection(s)
I20250113 08:54:32.222867 11991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data/info.pb
I20250113 08:54:32.224892 11762 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskqePfU0/build/asan/bin/kudu as pid 11991
I20250113 08:54:32.225044 11762 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/wal/instance
I20250113 08:54:32.229028 11762 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskqePfU0/build/asan/bin/kudu
/tmp/dist-test-taskqePfU0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.124.131:0
--local_ip_for_outbound_sockets=127.11.124.131
--webserver_interface=127.11.124.131
--webserver_port=0
--tserver_master_addrs=127.11.124.190:36259
--builtin_ntp_servers=127.11.124.148:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250113 08:54:32.237663 12119 heartbeater.cc:346] Connected to a master server at 127.11.124.190:36259
I20250113 08:54:32.237986 12119 heartbeater.cc:463] Registering TS with master...
I20250113 08:54:32.238832 12119 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:54:32.299036 11801 ts_manager.cc:194] Registered new tserver with Master: cbb5dc448319472395e744e7bd16c57f (127.11.124.130:46163)
I20250113 08:54:32.300086 11801 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.124.130:35101
W20250113 08:54:32.595912 12122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:54:32.596285 12122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:54:32.596418 12122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:54:32.604357 12122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:54:32.604638 12122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.124.131
I20250113 08:54:32.614238 12122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.124.148:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.124.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.124.131
--webserver_port=0
--tserver_master_addrs=127.11.124.190:36259
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.124.131
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:54:32.615964 12122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:54:32.617842 12122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:54:32.630028 12131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:32.630069 12128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:32.629989 12129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:32.630919 12122 server_base.cc:1034] running on GCE node
I20250113 08:54:32.631604 12122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:54:32.632655 12122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:54:32.633888 12122 hybrid_clock.cc:648] HybridClock initialized: now 1736758472633820 us; error 42 us; skew 500 ppm
I20250113 08:54:32.634210 12122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:54:32.636607 12122 webserver.cc:458] Webserver started at http://127.11.124.131:37733/ using document root <none> and password file <none>
I20250113 08:54:32.637374 12122 fs_manager.cc:362] Metadata directory not provided
I20250113 08:54:32.637506 12122 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:54:32.637817 12122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:54:32.640561 12122 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data/instance:
uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d"
format_stamp: "Formatted at 2025-01-13 08:54:32 on dist-test-slave-kk7k"
I20250113 08:54:32.641342 12122 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/wal/instance:
uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d"
format_stamp: "Formatted at 2025-01-13 08:54:32 on dist-test-slave-kk7k"
I20250113 08:54:32.646842 12122 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20250113 08:54:32.650738 12137 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:32.652397 12122 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250113 08:54:32.652572 12122 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/wal
uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d"
format_stamp: "Formatted at 2025-01-13 08:54:32 on dist-test-slave-kk7k"
I20250113 08:54:32.652781 12122 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:54:32.670543 12122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:54:32.671448 12122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:54:32.671737 12122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:54:32.672778 12122 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:54:32.674528 12122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:54:32.674618 12122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:32.674708 12122 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:54:32.674752 12122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:32.707700 12122 rpc_server.cc:307] RPC server started. Bound to: 127.11.124.131:39891
I20250113 08:54:32.707732 12249 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.124.131:39891 every 8 connection(s)
I20250113 08:54:32.709316 12122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data/info.pb
I20250113 08:54:32.719630 11762 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskqePfU0/build/asan/bin/kudu as pid 12122
I20250113 08:54:32.719852 11762 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/wal/instance
I20250113 08:54:32.723150 11762 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskqePfU0/build/asan/bin/kudu
/tmp/dist-test-taskqePfU0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.124.132:0
--local_ip_for_outbound_sockets=127.11.124.132
--webserver_interface=127.11.124.132
--webserver_port=0
--tserver_master_addrs=127.11.124.190:36259
--builtin_ntp_servers=127.11.124.148:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
I20250113 08:54:32.723361 12250 heartbeater.cc:346] Connected to a master server at 127.11.124.190:36259
I20250113 08:54:32.723713 12250 heartbeater.cc:463] Registering TS with master...
I20250113 08:54:32.724632 12250 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:54:32.771409 11801 ts_manager.cc:194] Registered new tserver with Master: aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:54:32.772485 11801 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.124.131:41597
I20250113 08:54:32.810920 11987 heartbeater.cc:502] Master 127.11.124.190:36259 was elected leader, sending a full tablet report...
W20250113 08:54:33.069650 12253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:54:33.069955 12253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:54:33.070101 12253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:54:33.078042 12253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:54:33.078307 12253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.124.132
I20250113 08:54:33.087967 12253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.124.148:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.124.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.11.124.132
--webserver_port=0
--tserver_master_addrs=127.11.124.190:36259
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.124.132
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:54:33.089715 12253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:54:33.091531 12253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:54:33.103812 12259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:33.103900 12260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:33.103847 12262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:33.104699 12253 server_base.cc:1034] running on GCE node
I20250113 08:54:33.105589 12253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:54:33.106701 12253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:54:33.107937 12253 hybrid_clock.cc:648] HybridClock initialized: now 1736758473107865 us; error 44 us; skew 500 ppm
I20250113 08:54:33.108275 12253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:54:33.110781 12253 webserver.cc:458] Webserver started at http://127.11.124.132:44139/ using document root <none> and password file <none>
I20250113 08:54:33.111609 12253 fs_manager.cc:362] Metadata directory not provided
I20250113 08:54:33.111739 12253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:54:33.112084 12253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 08:54:33.114795 12253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data/instance:
uuid: "b7d3e63990b74f76afc0e6cd5afbc13b"
format_stamp: "Formatted at 2025-01-13 08:54:33 on dist-test-slave-kk7k"
I20250113 08:54:33.115528 12253 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/wal/instance:
uuid: "b7d3e63990b74f76afc0e6cd5afbc13b"
format_stamp: "Formatted at 2025-01-13 08:54:33 on dist-test-slave-kk7k"
I20250113 08:54:33.120944 12253 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20250113 08:54:33.124573 12268 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:33.126055 12253 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250113 08:54:33.126214 12253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data,/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/wal
uuid: "b7d3e63990b74f76afc0e6cd5afbc13b"
format_stamp: "Formatted at 2025-01-13 08:54:33 on dist-test-slave-kk7k"
I20250113 08:54:33.126416 12253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:54:33.146126 12253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:54:33.147015 12253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:54:33.147307 12253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:54:33.148360 12253 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:54:33.149899 12253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:54:33.149973 12253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:33.150084 12253 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:54:33.150135 12253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:33.183214 12253 rpc_server.cc:307] RPC server started. Bound to: 127.11.124.132:36713
I20250113 08:54:33.183270 12380 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.124.132:36713 every 8 connection(s)
I20250113 08:54:33.185038 12253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data/info.pb
I20250113 08:54:33.193501 11762 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskqePfU0/build/asan/bin/kudu as pid 12253
I20250113 08:54:33.193686 11762 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/wal/instance
I20250113 08:54:33.199731 12381 heartbeater.cc:346] Connected to a master server at 127.11.124.190:36259
I20250113 08:54:33.200026 12381 heartbeater.cc:463] Registering TS with master...
I20250113 08:54:33.200848 12381 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:54:33.245220 11801 ts_manager.cc:194] Registered new tserver with Master: b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713)
I20250113 08:54:33.246227 11801 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.124.132:54303
I20250113 08:54:33.250281 11762 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250113 08:54:33.272123 11762 test_util.cc:274] Using random seed: -1747178731
I20250113 08:54:33.294780 11801 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59870:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250113 08:54:33.302798 12119 heartbeater.cc:502] Master 127.11.124.190:36259 was elected leader, sending a full tablet report...
I20250113 08:54:33.325009 12316 tablet_service.cc:1467] Processing CreateTablet for tablet 3034bef68d124e778f0b57a39a9b152c (DEFAULT_TABLE table=test-workload [id=f1414cc912c247a3ad439fd618ce9b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 08:54:33.326465 12316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3034bef68d124e778f0b57a39a9b152c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:33.326017 12185 tablet_service.cc:1467] Processing CreateTablet for tablet 3034bef68d124e778f0b57a39a9b152c (DEFAULT_TABLE table=test-workload [id=f1414cc912c247a3ad439fd618ce9b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 08:54:33.326035 11922 tablet_service.cc:1467] Processing CreateTablet for tablet 3034bef68d124e778f0b57a39a9b152c (DEFAULT_TABLE table=test-workload [id=f1414cc912c247a3ad439fd618ce9b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250113 08:54:33.327430 12185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3034bef68d124e778f0b57a39a9b152c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:33.327422 11922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3034bef68d124e778f0b57a39a9b152c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 08:54:33.338078 12405 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Bootstrap starting.
I20250113 08:54:33.341357 12406 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Bootstrap starting.
I20250113 08:54:33.342264 12407 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap starting.
I20250113 08:54:33.343602 12405 tablet_bootstrap.cc:654] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:33.345135 12405 log.cc:826] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Log is configured to *not* fsync() on all Append() calls
I20250113 08:54:33.346249 12407 tablet_bootstrap.cc:654] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:33.347172 12406 tablet_bootstrap.cc:654] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Neither blocks nor log segments found. Creating new log.
I20250113 08:54:33.347538 12407 log.cc:826] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Log is configured to *not* fsync() on all Append() calls
I20250113 08:54:33.348325 12406 log.cc:826] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Log is configured to *not* fsync() on all Append() calls
I20250113 08:54:33.348481 12405 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: No bootstrap required, opened a new log
I20250113 08:54:33.348773 12405 ts_tablet_manager.cc:1397] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20250113 08:54:33.350513 12407 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: No bootstrap required, opened a new log
I20250113 08:54:33.350700 12406 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: No bootstrap required, opened a new log
I20250113 08:54:33.350931 12407 ts_tablet_manager.cc:1397] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20250113 08:54:33.351034 12406 ts_tablet_manager.cc:1397] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20250113 08:54:33.357417 12406 raft_consensus.cc:357] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:54:33.357849 12406 raft_consensus.cc:383] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:33.357962 12406 raft_consensus.cc:738] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1af392ea8543f4827f17fdff2eaaac, State: Initialized, Role: FOLLOWER
I20250113 08:54:33.358664 12406 consensus_queue.cc:260] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:54:33.358654 12405 raft_consensus.cc:357] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:54:33.359097 12405 raft_consensus.cc:383] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:33.359206 12405 raft_consensus.cc:738] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d3e63990b74f76afc0e6cd5afbc13b, State: Initialized, Role: FOLLOWER
I20250113 08:54:33.359910 12405 consensus_queue.cc:260] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:54:33.360602 12407 raft_consensus.cc:357] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:54:33.361243 12381 heartbeater.cc:502] Master 127.11.124.190:36259 was elected leader, sending a full tablet report...
I20250113 08:54:33.361796 12405 ts_tablet_manager.cc:1428] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20250113 08:54:33.363691 12406 ts_tablet_manager.cc:1428] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20250113 08:54:33.364171 12407 raft_consensus.cc:383] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 08:54:33.364305 12407 raft_consensus.cc:738] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0b5c78b11d4d07bb963ae1d9a42b3d, State: Initialized, Role: FOLLOWER
I20250113 08:54:33.365049 12407 consensus_queue.cc:260] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:54:33.366253 12250 heartbeater.cc:502] Master 127.11.124.190:36259 was elected leader, sending a full tablet report...
I20250113 08:54:33.366819 12407 ts_tablet_manager.cc:1428] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Time spent starting tablet: real 0.016s user 0.011s sys 0.000s
I20250113 08:54:33.394161 12412 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:54:33.394440 12412 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:54:33.396710 12412 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:54:33.403632 11942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac" is_pre_election: true
I20250113 08:54:33.404012 12205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" is_pre_election: true
I20250113 08:54:33.404188 11942 raft_consensus.cc:2463] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7d3e63990b74f76afc0e6cd5afbc13b in term 0.
I20250113 08:54:33.404489 12205 raft_consensus.cc:2463] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7d3e63990b74f76afc0e6cd5afbc13b in term 0.
I20250113 08:54:33.404865 12270 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1af392ea8543f4827f17fdff2eaaac, b7d3e63990b74f76afc0e6cd5afbc13b; no voters:
I20250113 08:54:33.405247 12412 raft_consensus.cc:2798] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 08:54:33.405409 12412 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:54:33.405522 12412 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:33.407676 12412 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:54:33.408353 12412 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 1 election: Requested vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:54:33.408968 11942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac"
I20250113 08:54:33.409005 12205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d"
I20250113 08:54:33.409153 11942 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:33.409256 12205 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 0 FOLLOWER]: Advancing to term 1
I20250113 08:54:33.411314 11942 raft_consensus.cc:2463] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7d3e63990b74f76afc0e6cd5afbc13b in term 1.
I20250113 08:54:33.411762 12205 raft_consensus.cc:2463] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7d3e63990b74f76afc0e6cd5afbc13b in term 1.
I20250113 08:54:33.411821 12270 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1af392ea8543f4827f17fdff2eaaac, b7d3e63990b74f76afc0e6cd5afbc13b; no voters:
I20250113 08:54:33.412205 12412 raft_consensus.cc:2798] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 1 FOLLOWER]: Leader election won for term 1
I20250113 08:54:33.412804 12412 raft_consensus.cc:695] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 1 LEADER]: Becoming Leader. State: Replica: b7d3e63990b74f76afc0e6cd5afbc13b, State: Running, Role: LEADER
I20250113 08:54:33.413380 12412 consensus_queue.cc:237] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:54:33.418327 11799 catalog_manager.cc:5526] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b reported cstate change: term changed from 0 to 1, leader changed from <none> to b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132). New cstate: current_term: 1 leader_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } health_report { overall_health: UNKNOWN } } }
W20250113 08:54:33.437139 12382 tablet.cc:2367] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 08:54:33.441233 11762 maintenance_mode-itest.cc:745] Restarting batch of 1 tservers: cbb5dc448319472395e744e7bd16c57f
W20250113 08:54:33.461469 12251 tablet.cc:2367] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 08:54:33.487818 12205 raft_consensus.cc:1270] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 1 FOLLOWER]: Refusing update from remote peer b7d3e63990b74f76afc0e6cd5afbc13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:54:33.487882 11942 raft_consensus.cc:1270] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 1 FOLLOWER]: Refusing update from remote peer b7d3e63990b74f76afc0e6cd5afbc13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 08:54:33.488914 12415 consensus_queue.cc:1035] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 08:54:33.489207 12412 consensus_queue.cc:1035] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250113 08:54:33.492988 11988 tablet.cc:2367] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 08:54:33.522069 12427 mvcc.cc:204] Tried to move back new op lower bound from 7113762707394080768 to 7113762707105173504. Current Snapshot: MvccSnapshot[applied={T|T < 7113762707394080768}]
I20250113 08:54:34.599180 11795 ts_manager.cc:292] Set tserver state for cbb5dc448319472395e744e7bd16c57f to MAINTENANCE_MODE
I20250113 08:54:37.204016 12054 tablet_service.cc:1422] Tablet server cbb5dc448319472395e744e7bd16c57f set to quiescing
I20250113 08:54:37.204336 12054 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:54:38.914750 11762 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskqePfU0/build/asan/bin/kudu pid 11991
I20250113 08:54:38.955962 12114 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:54:39.205082 11762 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskqePfU0/build/asan/bin/kudu with pid 11991
I20250113 08:54:39.243392 11762 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskqePfU0/build/asan/bin/kudu
/tmp/dist-test-taskqePfU0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.124.130:46163
--local_ip_for_outbound_sockets=127.11.124.130
--tserver_master_addrs=127.11.124.190:36259
--webserver_port=44445
--webserver_interface=127.11.124.130
--builtin_ntp_servers=127.11.124.148:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:54:39.983575 12490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:54:39.984405 12490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:54:39.984675 12490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:54:39.998286 12490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:54:39.998646 12490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.124.130
I20250113 08:54:40.014879 12490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.124.148:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.124.130:46163
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.124.130
--webserver_port=44445
--tserver_master_addrs=127.11.124.190:36259
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.124.130
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:54:40.017520 12490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:54:40.020301 12490 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:54:40.052621 12498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:40.056622 12499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:40.077052 12502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:40.078138 12490 server_base.cc:1034] running on GCE node
I20250113 08:54:40.079084 12490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:54:40.081431 12490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:54:40.083164 12490 hybrid_clock.cc:648] HybridClock initialized: now 1736758480083027 us; error 112 us; skew 500 ppm
I20250113 08:54:40.083621 12490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:54:40.087711 12490 webserver.cc:458] Webserver started at http://127.11.124.130:44445/ using document root <none> and password file <none>
I20250113 08:54:40.089213 12490 fs_manager.cc:362] Metadata directory not provided
I20250113 08:54:40.089375 12490 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:54:40.117115 12490 fs_manager.cc:714] Time spent opening directory manager: real 0.023s user 0.008s sys 0.004s
I20250113 08:54:40.144039 12507 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:40.147001 12490 fs_manager.cc:730] Time spent opening block manager: real 0.028s user 0.013s sys 0.005s
I20250113 08:54:40.147353 12490 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/wal
uuid: "cbb5dc448319472395e744e7bd16c57f"
format_stamp: "Formatted at 2025-01-13 08:54:32 on dist-test-slave-kk7k"
I20250113 08:54:40.148792 12490 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:54:40.212424 12490 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:54:40.213733 12490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:54:40.214217 12490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:54:40.226495 12490 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:54:40.234573 12490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 08:54:40.234812 12490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:40.235020 12490 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 08:54:40.235152 12490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:41.146452 12490 rpc_server.cc:307] RPC server started. Bound to: 127.11.124.130:46163
I20250113 08:54:41.148861 12490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data/info.pb
I20250113 08:54:41.146512 12623 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.124.130:46163 every 8 connection(s)
I20250113 08:54:41.158838 11762 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskqePfU0/build/asan/bin/kudu as pid 12490
I20250113 08:54:41.230576 12624 heartbeater.cc:346] Connected to a master server at 127.11.124.190:36259
I20250113 08:54:41.230947 12624 heartbeater.cc:463] Registering TS with master...
I20250113 08:54:41.232384 12624 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:54:41.234942 11792 ts_manager.cc:194] Re-registered known tserver with Master: cbb5dc448319472395e744e7bd16c57f (127.11.124.130:46163)
I20250113 08:54:41.236136 11792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.124.130:58097
I20250113 08:54:42.240103 12624 heartbeater.cc:502] Master 127.11.124.190:36259 was elected leader, sending a full tablet report...
I20250113 08:54:42.264977 12185 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250113 08:54:42.279453 12316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:54:42.324119 11922 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:54:42.370319 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:54:43.736917 11794 ts_manager.cc:281] Unset tserver state for cbb5dc448319472395e744e7bd16c57f from MAINTENANCE_MODE
I20250113 08:54:43.757390 12250 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:54:44.253999 12624 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:54:44.634207 11987 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:54:44.638377 12381 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:54:45.250551 11762 maintenance_mode-itest.cc:745] Restarting batch of 1 tservers: 4a1af392ea8543f4827f17fdff2eaaac
I20250113 08:54:46.116441 11792 ts_manager.cc:292] Set tserver state for 4a1af392ea8543f4827f17fdff2eaaac to MAINTENANCE_MODE
I20250113 08:54:48.538676 11922 tablet_service.cc:1422] Tablet server 4a1af392ea8543f4827f17fdff2eaaac set to quiescing
I20250113 08:54:48.538991 11922 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:54:50.288208 11762 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskqePfU0/build/asan/bin/kudu pid 11860
I20250113 08:54:50.327448 11982 generic_service.cc:196] Checking for leaks (request via RPC)
W20250113 08:54:50.757498 12747 raft_consensus.cc:668] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: failed to trigger leader election: Illegal state: leader elections are disabled
I20250113 08:54:50.757834 11762 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskqePfU0/build/asan/bin/kudu with pid 11860
W20250113 08:54:50.903836 12270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.124.129:40671 (error 108)
I20250113 08:54:50.904742 11762 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskqePfU0/build/asan/bin/kudu
/tmp/dist-test-taskqePfU0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.124.129:40671
--local_ip_for_outbound_sockets=127.11.124.129
--tserver_master_addrs=127.11.124.190:36259
--webserver_port=38221
--webserver_interface=127.11.124.129
--builtin_ntp_servers=127.11.124.148:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:54:50.916288 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.129:40671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:54:51.332949 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.129:40671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:54:51.507754 12750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:54:51.508266 12750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:54:51.508459 12750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:54:51.519150 12750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:54:51.519546 12750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.124.129
I20250113 08:54:51.534512 12750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.124.148:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.124.129:40671
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.124.129
--webserver_port=38221
--tserver_master_addrs=127.11.124.190:36259
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.124.129
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:54:51.537009 12750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:54:51.539476 12750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:54:51.572885 12759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:51.601673 12758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:54:51.606622 12762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:54:51.607324 12750 server_base.cc:1034] running on GCE node
I20250113 08:54:51.608247 12750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:54:51.637492 12750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:54:51.643710 12750 hybrid_clock.cc:648] HybridClock initialized: now 1736758491643559 us; error 93 us; skew 500 ppm
I20250113 08:54:51.645817 12750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:54:51.688524 12750 webserver.cc:458] Webserver started at http://127.11.124.129:38221/ using document root <none> and password file <none>
I20250113 08:54:51.689729 12750 fs_manager.cc:362] Metadata directory not provided
I20250113 08:54:51.689975 12750 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:54:51.715795 12750 fs_manager.cc:714] Time spent opening directory manager: real 0.022s user 0.003s sys 0.015s
I20250113 08:54:51.756929 12769 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:54:51.779259 12750 fs_manager.cc:730] Time spent opening block manager: real 0.061s user 0.013s sys 0.019s
I20250113 08:54:51.781749 12750 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/wal
uuid: "4a1af392ea8543f4827f17fdff2eaaac"
format_stamp: "Formatted at 2025-01-13 08:54:31 on dist-test-slave-kk7k"
I20250113 08:54:51.784706 12750 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250113 08:54:51.834658 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.129:40671: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250113 08:54:51.859220 12750 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:54:51.860713 12750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:54:51.861100 12750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:54:51.862679 12750 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:54:51.878686 12776 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250113 08:54:51.887328 12750 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 08:54:51.887470 12750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.002s sys 0.005s
I20250113 08:54:51.887598 12750 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 08:54:51.892568 12750 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 08:54:51.892753 12750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.003s
I20250113 08:54:51.897013 12776 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Bootstrap starting.
W20250113 08:54:52.377480 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.129:40671: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:54:52.850919 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.129:40671: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250113 08:54:52.862521 12883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.124.129:40671 every 8 connection(s)
I20250113 08:54:52.862915 12750 rpc_server.cc:307] RPC server started. Bound to: 127.11.124.129:40671
I20250113 08:54:52.865002 12750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data/info.pb
I20250113 08:54:52.871327 11762 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskqePfU0/build/asan/bin/kudu as pid 12750
I20250113 08:54:53.048187 12884 heartbeater.cc:346] Connected to a master server at 127.11.124.190:36259
I20250113 08:54:53.048530 12884 heartbeater.cc:463] Registering TS with master...
I20250113 08:54:53.049693 12884 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:54:53.054756 11792 ts_manager.cc:194] Re-registered known tserver with Master: 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671)
I20250113 08:54:53.056404 11792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.124.129:33037
W20250113 08:54:53.428705 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
I20250113 08:54:53.602049 12776 log.cc:826] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Log is configured to *not* fsync() on all Append() calls
W20250113 08:54:53.997035 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250113 08:54:54.062322 12884 heartbeater.cc:502] Master 127.11.124.190:36259 was elected leader, sending a full tablet report...
W20250113 08:54:54.466149 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20250113 08:54:54.543875 12316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:54:54.554355 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:54:54.566955 12185 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250113 08:54:54.588855 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:54:55.055660 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
I20250113 08:54:55.110659 12753 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer 4a1af392ea8543f4827f17fdff2eaaac is lagging by at least 6 ops behind the committed index
W20250113 08:54:55.650310 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250113 08:54:56.145921 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
W20250113 08:54:56.646117 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
W20250113 08:54:57.153204 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
W20250113 08:54:57.588711 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
W20250113 08:54:58.121938 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
I20250113 08:54:58.184924 12753 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer 4a1af392ea8543f4827f17fdff2eaaac is lagging by at least 340 ops behind the committed index [suppressed 29 similar messages]
I20250113 08:54:58.624079 12776 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Bootstrap replaying log segment 1/1 (1.38M/2.44M this segment, stats: ops{read=829 overwritten=0 applied=829 ignored=0} inserts{seen=41400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:54:58.654973 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
W20250113 08:54:59.121745 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250113 08:54:59.600304 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
W20250113 08:55:00.123383 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
W20250113 08:55:00.584308 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
W20250113 08:55:01.096122 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
I20250113 08:55:01.240484 12937 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer 4a1af392ea8543f4827f17fdff2eaaac is lagging by at least 651 ops behind the committed index [suppressed 30 similar messages]
W20250113 08:55:01.577268 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
I20250113 08:55:02.021363 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:02.074837 12185 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
W20250113 08:55:02.102216 12270 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
I20250113 08:55:02.102681 12316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:55:02.191303 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:02.297801 12776 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Bootstrap replayed 1/1 log segments. Stats: ops{read=1460 overwritten=0 applied=1460 ignored=0} inserts{seen=72950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:55:02.298588 12776 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Bootstrap complete.
I20250113 08:55:02.300516 12776 ts_tablet_manager.cc:1397] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Time spent bootstrapping tablet: real 10.404s user 6.258s sys 0.520s
I20250113 08:55:02.306988 12776 raft_consensus.cc:357] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:02.308214 12776 raft_consensus.cc:738] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1af392ea8543f4827f17fdff2eaaac, State: Initialized, Role: FOLLOWER
I20250113 08:55:02.309077 12776 consensus_queue.cc:260] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1460, Last appended: 1.1460, Last appended by leader: 1460, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:02.326589 12776 ts_tablet_manager.cc:1428] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Time spent starting tablet: real 0.026s user 0.014s sys 0.000s
I20250113 08:55:02.819376 12975 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7d3e63990b74f76afc0e6cd5afbc13b)
I20250113 08:55:02.819656 12975 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:02.826261 12975 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:55:02.887435 12336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "4a1af392ea8543f4827f17fdff2eaaac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1885 } } ignore_live_leader: false dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" is_pre_election: true
I20250113 08:55:02.891407 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "4a1af392ea8543f4827f17fdff2eaaac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1885 } } ignore_live_leader: false dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" is_pre_election: true
I20250113 08:55:02.892357 12771 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1af392ea8543f4827f17fdff2eaaac; no voters: aa0b5c78b11d4d07bb963ae1d9a42b3d, b7d3e63990b74f76afc0e6cd5afbc13b
I20250113 08:55:03.205885 12975 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:55:03.561616 12975 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b7d3e63990b74f76afc0e6cd5afbc13b)
I20250113 08:55:03.561811 12975 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:03.562590 12975 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:55:03.564994 12205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "4a1af392ea8543f4827f17fdff2eaaac" candidate_term: 2 candidate_status { last_received { term: 1 index: 2698 } } ignore_live_leader: false dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" is_pre_election: true
I20250113 08:55:03.568243 12336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "4a1af392ea8543f4827f17fdff2eaaac" candidate_term: 2 candidate_status { last_received { term: 1 index: 2698 } } ignore_live_leader: false dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" is_pre_election: true
I20250113 08:55:03.568971 12772 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1af392ea8543f4827f17fdff2eaaac; no voters: aa0b5c78b11d4d07bb963ae1d9a42b3d, b7d3e63990b74f76afc0e6cd5afbc13b
I20250113 08:55:03.569483 12975 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250113 08:55:09.345572 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:09.404109 12185 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250113 08:55:09.412225 12316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:55:09.520345 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:10.830559 11794 ts_manager.cc:281] Unset tserver state for 4a1af392ea8543f4827f17fdff2eaaac from MAINTENANCE_MODE
I20250113 08:55:10.897972 12884 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:55:10.904027 12381 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:55:10.916639 12250 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:55:11.325917 12624 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:55:12.442188 11762 maintenance_mode-itest.cc:745] Restarting batch of 1 tservers: b7d3e63990b74f76afc0e6cd5afbc13b
I20250113 08:55:13.438720 11794 ts_manager.cc:292] Set tserver state for b7d3e63990b74f76afc0e6cd5afbc13b to MAINTENANCE_MODE
W20250113 08:55:13.771214 12392 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.11.124.132:36713, user_credentials={real_user=slave}} blocked reactor thread for 87874.3us
I20250113 08:55:16.284389 12316 tablet_service.cc:1422] Tablet server b7d3e63990b74f76afc0e6cd5afbc13b set to quiescing
I20250113 08:55:16.284637 12316 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:55:16.291129 13063 raft_consensus.cc:988] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: : Instructing follower aa0b5c78b11d4d07bb963ae1d9a42b3d to start an election
I20250113 08:55:16.291299 13063 raft_consensus.cc:1076] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 1 LEADER]: Signalling peer aa0b5c78b11d4d07bb963ae1d9a42b3d to start an election
I20250113 08:55:16.295065 13091 raft_consensus.cc:988] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: : Instructing follower aa0b5c78b11d4d07bb963ae1d9a42b3d to start an election
I20250113 08:55:16.295202 13091 raft_consensus.cc:1076] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 1 LEADER]: Signalling peer aa0b5c78b11d4d07bb963ae1d9a42b3d to start an election
I20250113 08:55:16.296059 12204 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c"
dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d"
from {username='slave'} at 127.11.124.132:54165
I20250113 08:55:16.296329 12204 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:55:16.296458 12204 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:55:16.299528 12204 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:16.300415 12205 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c"
dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d"
from {username='slave'} at 127.11.124.132:54165
I20250113 08:55:16.303236 13063 raft_consensus.cc:988] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: : Instructing follower 4a1af392ea8543f4827f17fdff2eaaac to start an election
I20250113 08:55:16.303356 13063 raft_consensus.cc:1076] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 1 LEADER]: Signalling peer 4a1af392ea8543f4827f17fdff2eaaac to start an election
I20250113 08:55:16.304366 12838 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c"
dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac"
from {username='slave'} at 127.11.124.132:48813
I20250113 08:55:16.304656 12838 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:55:16.304806 12838 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 1 FOLLOWER]: Advancing to term 2
I20250113 08:55:16.308310 12205 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:55:16.308454 12205 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:55:16.313117 12205 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:16.314200 12205 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 3 election: Requested vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713)
I20250113 08:55:16.316025 12204 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 2 election: Requested vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713)
I20250113 08:55:16.316243 12838 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:16.317951 12336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "4a1af392ea8543f4827f17fdff2eaaac" candidate_term: 2 candidate_status { last_received { term: 1 index: 3895 } } ignore_live_leader: true dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b"
I20250113 08:55:16.318120 12838 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [CANDIDATE]: Term 2 election: Requested vote from peers b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:55:16.319100 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "4a1af392ea8543f4827f17fdff2eaaac" candidate_term: 2 candidate_status { last_received { term: 1 index: 3895 } } ignore_live_leader: true dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d"
I20250113 08:55:16.319398 12204 raft_consensus.cc:2363] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1af392ea8543f4827f17fdff2eaaac for earlier term 2. Current term is 3.
I20250113 08:55:16.320492 12771 leader_election.cc:400] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [CANDIDATE]: Term 2 election: Vote denied by peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891) with higher term. Message: Invalid argument: T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1af392ea8543f4827f17fdff2eaaac for earlier term 2. Current term is 3.
I20250113 08:55:16.320605 12771 leader_election.cc:403] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250113 08:55:16.323866 13111 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 2 FOLLOWER]: Advancing to term 3
I20250113 08:55:16.327072 13111 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891) with higher term. Message: Invalid argument: T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1af392ea8543f4827f17fdff2eaaac for earlier term 2. Current term is 3.
I20250113 08:55:16.328991 12336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3895 } } ignore_live_leader: true dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b"
I20250113 08:55:16.329206 12335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3895 } } ignore_live_leader: true dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b"
I20250113 08:55:16.332492 12204 raft_consensus.cc:1235] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 FOLLOWER]: Rejecting Update request from peer b7d3e63990b74f76afc0e6cd5afbc13b for earlier term 1. Current term is 3. Ops: [1.3896-1.3896]
I20250113 08:55:16.333487 12839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3895 } } ignore_live_leader: true dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac"
I20250113 08:55:16.336825 12839 raft_consensus.cc:2463] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa0b5c78b11d4d07bb963ae1d9a42b3d in term 3.
I20250113 08:55:16.337527 12945 consensus_queue.cc:1046] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 }, Status: INVALID_TERM, Last received: 1.3895, Next index: 3896, Last known committed idx: 3895, Time since last communication: 0.000s
I20250113 08:55:16.339946 12838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3895 } } ignore_live_leader: true dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac"
I20250113 08:55:16.340348 12838 raft_consensus.cc:2363] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate aa0b5c78b11d4d07bb963ae1d9a42b3d for earlier term 2. Current term is 3.
I20250113 08:55:16.340747 12837 raft_consensus.cc:1235] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 3 FOLLOWER]: Rejecting Update request from peer b7d3e63990b74f76afc0e6cd5afbc13b for earlier term 1. Current term is 3. Ops: [1.3896-1.3897]
I20250113 08:55:16.341156 12139 leader_election.cc:400] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 2 election: Vote denied by peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671) with higher term. Message: Invalid argument: T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate aa0b5c78b11d4d07bb963ae1d9a42b3d for earlier term 2. Current term is 3.
I20250113 08:55:16.341238 12139 leader_election.cc:403] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250113 08:55:16.341946 13091 consensus_queue.cc:1046] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 }, Status: INVALID_TERM, Last received: 1.3895, Next index: 3896, Last known committed idx: 3895, Time since last communication: 0.000s
I20250113 08:55:16.347822 12945 raft_consensus.cc:3049] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 1 LEADER]: Stepping down as leader of term 1
I20250113 08:55:16.347966 12945 raft_consensus.cc:738] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b7d3e63990b74f76afc0e6cd5afbc13b, State: Running, Role: LEADER
I20250113 08:55:16.348404 12945 consensus_queue.cc:260] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3895, Committed index: 3895, Last appended: 1.3897, Last appended by leader: 3897, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:16.349332 12945 logging.cc:424] LogThrottler TrackedPeer Lag: suppressed but not reported on 11 messages since previous log ~15 seconds ago
I20250113 08:55:16.349727 12945 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 1 FOLLOWER]: Advancing to term 3
W20250113 08:55:16.354617 12426 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37570: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.018s ago.
I20250113 08:55:16.364496 13112 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671) with higher term. Message: Invalid argument: T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate aa0b5c78b11d4d07bb963ae1d9a42b3d for earlier term 2. Current term is 3.
I20250113 08:55:16.365481 12204 raft_consensus.cc:1235] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 FOLLOWER]: Rejecting Update request from peer b7d3e63990b74f76afc0e6cd5afbc13b for earlier term 1. Current term is 3. Ops: [1.3896-1.3897]
I20250113 08:55:16.365962 12139 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1af392ea8543f4827f17fdff2eaaac, aa0b5c78b11d4d07bb963ae1d9a42b3d; no voters: b7d3e63990b74f76afc0e6cd5afbc13b
I20250113 08:55:16.366792 13112 raft_consensus.cc:2798] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 FOLLOWER]: Leader election won for term 3
I20250113 08:55:16.367363 13112 raft_consensus.cc:695] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 LEADER]: Becoming Leader. State: Replica: aa0b5c78b11d4d07bb963ae1d9a42b3d, State: Running, Role: LEADER
I20250113 08:55:16.367908 13112 consensus_queue.cc:237] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3895, Committed index: 3895, Last appended: 1.3895, Last appended by leader: 3895, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:16.372484 11794 catalog_manager.cc:5526] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d reported cstate change: term changed from 1 to 3, leader changed from b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132) to aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131). New cstate: current_term: 3 leader_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } health_report { overall_health: HEALTHY } } }
W20250113 08:55:16.384630 12793 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48304: Illegal state: replica 4a1af392ea8543f4827f17fdff2eaaac is not leader of this config: current role FOLLOWER
I20250113 08:55:16.395123 12839 raft_consensus.cc:1270] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 3 FOLLOWER]: Refusing update from remote peer aa0b5c78b11d4d07bb963ae1d9a42b3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3895. Preceding OpId from leader: term: 3 index: 3897. (index mismatch)
I20250113 08:55:16.395738 12335 raft_consensus.cc:1270] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Refusing update from remote peer aa0b5c78b11d4d07bb963ae1d9a42b3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3897. Preceding OpId from leader: term: 3 index: 3897. (term mismatch)
I20250113 08:55:16.395887 12335 pending_rounds.cc:77] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Aborting all ops after (but not including) 3896
I20250113 08:55:16.396063 12335 pending_rounds.cc:99] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Aborting uncommitted WRITE_OP operation due to leader change: 1.3897
W20250113 08:55:16.396343 12335 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37570: Aborted: Op aborted by new leader
I20250113 08:55:16.397847 13112 consensus_queue.cc:1035] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3896, Last known committed idx: 3895, Time since last communication: 0.000s
I20250113 08:55:16.398118 13112 consensus_queue.cc:1225] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [LEADER]: Peer b7d3e63990b74f76afc0e6cd5afbc13b log is divergent from this leader: its last log entry 1.3896 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3895
I20250113 08:55:16.398248 13114 consensus_queue.cc:1035] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3896, Last known committed idx: 3895, Time since last communication: 0.000s
I20250113 08:55:16.399992 12335 pending_rounds.cc:77] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Aborting all ops after (but not including) 3895
I20250113 08:55:16.400113 12335 pending_rounds.cc:99] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Aborting uncommitted WRITE_OP operation due to leader change: 1.3896
W20250113 08:55:16.400899 12335 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37570: Aborted: Op aborted by new leader
I20250113 08:55:19.848171 12316 tablet_service.cc:1422] Tablet server b7d3e63990b74f76afc0e6cd5afbc13b set to quiescing
I20250113 08:55:19.848464 12316 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:21.413611 11762 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskqePfU0/build/asan/bin/kudu pid 12253
I20250113 08:55:21.462754 12375 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:55:21.860342 11762 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskqePfU0/build/asan/bin/kudu with pid 12253
W20250113 08:55:21.931799 12140 connection.cc:537] client connection to 127.11.124.132:36713 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 08:55:21.932138 12140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 08:55:21.932022 11762 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskqePfU0/build/asan/bin/kudu
/tmp/dist-test-taskqePfU0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.124.132:36713
--local_ip_for_outbound_sockets=127.11.124.132
--tserver_master_addrs=127.11.124.190:36259
--webserver_port=44139
--webserver_interface=127.11.124.132
--builtin_ntp_servers=127.11.124.148:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:55:21.945267 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.132:36713: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:55:22.484112 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.132:36713: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:55:22.535508 13157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:55:22.535933 13157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:55:22.536191 13157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:55:22.548810 13157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:55:22.549242 13157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.124.132
I20250113 08:55:22.564028 13157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.124.148:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.124.132:36713
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data/info.pb
--webserver_interface=127.11.124.132
--webserver_port=44139
--tserver_master_addrs=127.11.124.190:36259
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.124.132
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:55:22.566246 13157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:55:22.568356 13157 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:55:22.588440 13166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:22.601189 13165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:55:22.610931 13169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:55:22.617781 13157 server_base.cc:1034] running on GCE node
I20250113 08:55:22.623739 13157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:55:22.629927 13157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:55:22.640240 13157 hybrid_clock.cc:648] HybridClock initialized: now 1736758522639260 us; error 48 us; skew 500 ppm
I20250113 08:55:22.640740 13157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:55:22.659176 13157 webserver.cc:458] Webserver started at http://127.11.124.132:44139/ using document root <none> and password file <none>
I20250113 08:55:22.660235 13157 fs_manager.cc:362] Metadata directory not provided
I20250113 08:55:22.660399 13157 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:55:22.671455 13157 fs_manager.cc:714] Time spent opening directory manager: real 0.009s user 0.006s sys 0.000s
I20250113 08:55:22.702827 13174 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:55:22.719724 13157 fs_manager.cc:730] Time spent opening block manager: real 0.039s user 0.004s sys 0.012s
I20250113 08:55:22.720110 13157 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data,/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/wal
uuid: "b7d3e63990b74f76afc0e6cd5afbc13b"
format_stamp: "Formatted at 2025-01-13 08:54:33 on dist-test-slave-kk7k"
I20250113 08:55:22.721565 13157 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/wal
metadata directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/wal
1 data directories: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:55:22.796618 13157 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:55:22.797925 13157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:55:22.798395 13157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:55:22.799861 13157 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:55:22.809624 13181 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250113 08:55:22.836431 13157 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 08:55:22.836702 13157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.034s user 0.000s sys 0.007s
I20250113 08:55:22.836935 13157 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 08:55:22.842044 13181 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Bootstrap starting.
I20250113 08:55:22.856156 13157 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 08:55:22.856351 13157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.019s user 0.004s sys 0.000s
W20250113 08:55:22.953153 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.132:36713: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250113 08:55:23.211978 13157 rpc_server.cc:307] RPC server started. Bound to: 127.11.124.132:36713
I20250113 08:55:23.213074 13287 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.124.132:36713 every 8 connection(s)
I20250113 08:55:23.216428 11762 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskqePfU0/build/asan/bin/kudu as pid 13157
I20250113 08:55:23.216163 13157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data/info.pb
I20250113 08:55:23.326548 13289 heartbeater.cc:346] Connected to a master server at 127.11.124.190:36259
I20250113 08:55:23.326943 13289 heartbeater.cc:463] Registering TS with master...
I20250113 08:55:23.328186 13289 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:55:23.331274 11792 ts_manager.cc:194] Re-registered known tserver with Master: b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713)
I20250113 08:55:23.333041 11792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.124.132:37219
W20250113 08:55:23.460628 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
W20250113 08:55:23.957274 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
I20250113 08:55:24.343475 13289 heartbeater.cc:502] Master 127.11.124.190:36259 was elected leader, sending a full tablet report...
W20250113 08:55:24.434609 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
I20250113 08:55:24.511229 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:24.536584 13223 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:24.550156 12185 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250113 08:55:24.600646 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:55:24.947698 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
W20250113 08:55:25.472572 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20250113 08:55:25.545961 13181 log.cc:826] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Log is configured to *not* fsync() on all Append() calls
I20250113 08:55:25.859508 13124 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [LEADER]: Peer b7d3e63990b74f76afc0e6cd5afbc13b is lagging by at least 1 ops behind the committed index
W20250113 08:55:26.011468 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20250113 08:55:26.567265 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250113 08:55:27.105832 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
W20250113 08:55:27.646430 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
W20250113 08:55:28.151692 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
W20250113 08:55:28.663880 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
I20250113 08:55:28.951099 13341 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [LEADER]: Peer b7d3e63990b74f76afc0e6cd5afbc13b is lagging by at least 344 ops behind the committed index [suppressed 28 similar messages]
W20250113 08:55:29.213886 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
W20250113 08:55:29.699753 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
W20250113 08:55:30.310302 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250113 08:55:30.343359 12391 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.11.124.131:39891, user_credentials={real_user=slave}} blocked reactor thread for 51495.3us
I20250113 08:55:30.558502 13181 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Bootstrap replaying log segment 1/1 (1.85M/7.56M this segment, stats: ops{read=1069 overwritten=0 applied=1068 ignored=0} inserts{seen=53350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:55:30.833009 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
W20250113 08:55:31.402746 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
I20250113 08:55:31.610965 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:31.677116 12185 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250113 08:55:31.690071 13223 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:31.709444 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:55:31.909693 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
I20250113 08:55:32.039469 13159 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [LEADER]: Peer b7d3e63990b74f76afc0e6cd5afbc13b is lagging by at least 639 ops behind the committed index [suppressed 28 similar messages]
W20250113 08:55:32.374778 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
W20250113 08:55:32.883751 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20250113 08:55:33.319242 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
W20250113 08:55:33.849315 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
W20250113 08:55:34.339807 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
W20250113 08:55:34.897378 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
I20250113 08:55:35.106562 13340 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [LEADER]: Peer b7d3e63990b74f76afc0e6cd5afbc13b is lagging by at least 973 ops behind the committed index [suppressed 29 similar messages]
W20250113 08:55:35.464286 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
I20250113 08:55:35.561897 13181 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Bootstrap replaying log segment 1/1 (3.61M/7.56M this segment, stats: ops{read=2087 overwritten=0 applied=2086 ignored=0} inserts{seen=104250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:55:36.041395 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
W20250113 08:55:36.538729 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
W20250113 08:55:37.011909 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
W20250113 08:55:37.534200 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
W20250113 08:55:37.967644 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 156: this message will repeat every 5th retry.
I20250113 08:55:38.241761 13148 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [LEADER]: Peer b7d3e63990b74f76afc0e6cd5afbc13b is lagging by at least 1295 ops behind the committed index [suppressed 29 similar messages]
I20250113 08:55:38.495770 12185 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250113 08:55:38.533949 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:38.541775 13223 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:38.569710 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:55:38.683223 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 161: this message will repeat every 5th retry.
W20250113 08:55:39.200680 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 166: this message will repeat every 5th retry.
W20250113 08:55:39.712513 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 171: this message will repeat every 5th retry.
W20250113 08:55:40.258209 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 176: this message will repeat every 5th retry.
I20250113 08:55:40.565021 13181 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Bootstrap replaying log segment 1/1 (5.23M/7.56M this segment, stats: ops{read=3020 overwritten=0 applied=3019 ignored=0} inserts{seen=150900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:55:40.759222 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 181: this message will repeat every 5th retry.
I20250113 08:55:41.280337 13426 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [LEADER]: Peer b7d3e63990b74f76afc0e6cd5afbc13b is lagging by at least 1616 ops behind the committed index [suppressed 27 similar messages]
W20250113 08:55:41.308557 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 186: this message will repeat every 5th retry.
W20250113 08:55:41.840867 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 191: this message will repeat every 5th retry.
W20250113 08:55:42.334619 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 196: this message will repeat every 5th retry.
W20250113 08:55:42.829773 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 201: this message will repeat every 5th retry.
W20250113 08:55:43.315219 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 206: this message will repeat every 5th retry.
W20250113 08:55:43.868671 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 211: this message will repeat every 5th retry.
I20250113 08:55:44.335496 13431 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [LEADER]: Peer b7d3e63990b74f76afc0e6cd5afbc13b is lagging by at least 1956 ops behind the committed index [suppressed 29 similar messages]
W20250113 08:55:44.381803 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 216: this message will repeat every 5th retry.
W20250113 08:55:44.911386 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 221: this message will repeat every 5th retry.
I20250113 08:55:45.436204 13223 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:45.465456 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:55:45.466858 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 226: this message will repeat every 5th retry.
I20250113 08:55:45.472661 12185 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250113 08:55:45.491365 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:45.571249 13181 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Bootstrap replaying log segment 1/1 (6.70M/7.56M this segment, stats: ops{read=3871 overwritten=0 applied=3870 ignored=0} inserts{seen=193450 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:55:45.971774 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 231: this message will repeat every 5th retry.
W20250113 08:55:46.481340 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 236: this message will repeat every 5th retry.
W20250113 08:55:47.050356 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 241: this message will repeat every 5th retry.
I20250113 08:55:47.433212 13479 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [LEADER]: Peer b7d3e63990b74f76afc0e6cd5afbc13b is lagging by at least 2280 ops behind the committed index [suppressed 28 similar messages]
W20250113 08:55:47.547250 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 246: this message will repeat every 5th retry.
W20250113 08:55:48.154994 12140 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d -> Peer b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): Couldn't send request to peer b7d3e63990b74f76afc0e6cd5afbc13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 251: this message will repeat every 5th retry.
I20250113 08:55:48.248493 13181 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4391 overwritten=2 applied=4389 ignored=0} inserts{seen=219350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:55:48.253463 13181 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Bootstrap complete.
I20250113 08:55:48.256198 13181 ts_tablet_manager.cc:1397] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Time spent bootstrapping tablet: real 25.415s user 18.616s sys 0.869s
I20250113 08:55:48.265311 13181 raft_consensus.cc:357] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:48.266381 13181 raft_consensus.cc:738] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d3e63990b74f76afc0e6cd5afbc13b, State: Initialized, Role: FOLLOWER
I20250113 08:55:48.267210 13181 consensus_queue.cc:260] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4389, Last appended: 3.4389, Last appended by leader: 4389, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:48.275707 13181 ts_tablet_manager.cc:1428] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b: Time spent starting tablet: real 0.019s user 0.008s sys 0.006s
I20250113 08:55:48.695086 13487 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader aa0b5c78b11d4d07bb963ae1d9a42b3d)
I20250113 08:55:48.695381 13487 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:48.697762 13487 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:55:48.779704 12838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4814 } } ignore_live_leader: false dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac" is_pre_election: true
I20250113 08:55:48.839113 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4814 } } ignore_live_leader: false dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" is_pre_election: true
I20250113 08:55:48.839943 13176 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b7d3e63990b74f76afc0e6cd5afbc13b; no voters: 4a1af392ea8543f4827f17fdff2eaaac, aa0b5c78b11d4d07bb963ae1d9a42b3d
I20250113 08:55:49.409111 13487 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:55:49.819196 13487 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader aa0b5c78b11d4d07bb963ae1d9a42b3d)
I20250113 08:55:49.819396 13487 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:49.867368 13487 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:55:49.868368 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5664 } } ignore_live_leader: false dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" is_pre_election: true
I20250113 08:55:49.868981 12839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5664 } } ignore_live_leader: false dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac" is_pre_election: true
I20250113 08:55:49.869721 13176 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b7d3e63990b74f76afc0e6cd5afbc13b; no voters: 4a1af392ea8543f4827f17fdff2eaaac, aa0b5c78b11d4d07bb963ae1d9a42b3d
I20250113 08:55:49.870034 13487 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:55:50.219854 13487 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader aa0b5c78b11d4d07bb963ae1d9a42b3d)
I20250113 08:55:50.220093 13487 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:50.220834 13487 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:55:50.221333 12838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 4 candidate_status { last_received { term: 3 index: 6089 } } ignore_live_leader: false dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac" is_pre_election: true
I20250113 08:55:50.224581 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 4 candidate_status { last_received { term: 3 index: 6089 } } ignore_live_leader: false dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" is_pre_election: true
I20250113 08:55:50.225239 13176 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b7d3e63990b74f76afc0e6cd5afbc13b; no voters: 4a1af392ea8543f4827f17fdff2eaaac, aa0b5c78b11d4d07bb963ae1d9a42b3d
I20250113 08:55:50.225510 13487 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:55:50.528746 13480 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [LEADER]: Peer b7d3e63990b74f76afc0e6cd5afbc13b is lagging by at least 458 ops behind the committed index [suppressed 12 similar messages]
I20250113 08:55:51.246330 13506 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader aa0b5c78b11d4d07bb963ae1d9a42b3d)
I20250113 08:55:51.246585 13506 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:51.247658 13506 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:55:51.253757 12839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 4 candidate_status { last_received { term: 3 index: 7364 } } ignore_live_leader: false dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac" is_pre_election: true
I20250113 08:55:51.253717 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 4 candidate_status { last_received { term: 3 index: 7364 } } ignore_live_leader: false dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" is_pre_election: true
I20250113 08:55:51.256352 13176 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b7d3e63990b74f76afc0e6cd5afbc13b; no voters: 4a1af392ea8543f4827f17fdff2eaaac, aa0b5c78b11d4d07bb963ae1d9a42b3d
I20250113 08:55:51.257025 13506 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250113 08:55:52.734480 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:52.787225 12185 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250113 08:55:52.836520 13223 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:55:52.852804 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:55:52.989593 12391 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.11.124.131:39891, user_credentials={real_user=slave}} blocked reactor thread for 76908.1us
I20250113 08:55:54.270503 11794 ts_manager.cc:281] Unset tserver state for b7d3e63990b74f76afc0e6cd5afbc13b from MAINTENANCE_MODE
I20250113 08:55:54.435743 12624 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:55:54.508428 12884 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:55:54.817509 12250 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:55:54.829521 13289 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:55:56.320294 11762 maintenance_mode-itest.cc:745] Restarting batch of 1 tservers: aa0b5c78b11d4d07bb963ae1d9a42b3d
I20250113 08:55:57.084252 11794 ts_manager.cc:292] Set tserver state for aa0b5c78b11d4d07bb963ae1d9a42b3d to MAINTENANCE_MODE
W20250113 08:55:57.124161 12391 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.11.124.131:39891, user_credentials={real_user=slave}} blocked reactor thread for 68252.9us
I20250113 08:55:59.483318 12185 tablet_service.cc:1422] Tablet server aa0b5c78b11d4d07bb963ae1d9a42b3d set to quiescing
I20250113 08:55:59.483574 12185 tablet_service.cc:1429] Tablet server has 1 leaders and 3 scanners
I20250113 08:55:59.490758 13504 raft_consensus.cc:988] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: : Instructing follower 4a1af392ea8543f4827f17fdff2eaaac to start an election
I20250113 08:55:59.490943 13504 raft_consensus.cc:1076] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 LEADER]: Signalling peer 4a1af392ea8543f4827f17fdff2eaaac to start an election
I20250113 08:55:59.492895 13480 raft_consensus.cc:988] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: : Instructing follower b7d3e63990b74f76afc0e6cd5afbc13b to start an election
I20250113 08:55:59.493039 13480 raft_consensus.cc:1076] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 LEADER]: Signalling peer b7d3e63990b74f76afc0e6cd5afbc13b to start an election
I20250113 08:55:59.493266 13480 raft_consensus.cc:988] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: : Instructing follower 4a1af392ea8543f4827f17fdff2eaaac to start an election
I20250113 08:55:59.493395 13480 raft_consensus.cc:1076] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 LEADER]: Signalling peer 4a1af392ea8543f4827f17fdff2eaaac to start an election
I20250113 08:55:59.494566 13504 raft_consensus.cc:988] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: : Instructing follower b7d3e63990b74f76afc0e6cd5afbc13b to start an election
I20250113 08:55:59.494410 13243 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c"
dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b"
from {username='slave'} at 127.11.124.131:45087
I20250113 08:55:59.494688 13504 raft_consensus.cc:1076] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 LEADER]: Signalling peer b7d3e63990b74f76afc0e6cd5afbc13b to start an election
I20250113 08:55:59.494724 13243 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:55:59.494890 13243 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:55:59.495234 12839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c"
dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac"
from {username='slave'} at 127.11.124.131:53855
I20250113 08:55:59.495430 12839 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:55:59.495541 12839 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 3 FOLLOWER]: Advancing to term 4
I20250113 08:55:59.497453 12838 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c"
dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac"
from {username='slave'} at 127.11.124.131:53855
I20250113 08:55:59.498310 12839 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:59.498643 13242 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c"
dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b"
from {username='slave'} at 127.11.124.131:45087
I20250113 08:55:59.504024 13243 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:59.504683 13242 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:55:59.504843 13242 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:55:59.507440 13242 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:59.507949 13241 raft_consensus.cc:1235] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 5 FOLLOWER]: Rejecting Update request from peer aa0b5c78b11d4d07bb963ae1d9a42b3d for earlier term 3. Current term is 5. Ops: [3.8125-3.8125]
I20250113 08:55:59.508205 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 4 candidate_status { last_received { term: 3 index: 8124 } } ignore_live_leader: true dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d"
I20250113 08:55:59.508417 12204 raft_consensus.cc:3049] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 LEADER]: Stepping down as leader of term 3
I20250113 08:55:59.508523 12204 raft_consensus.cc:738] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: aa0b5c78b11d4d07bb963ae1d9a42b3d, State: Running, Role: LEADER
I20250113 08:55:59.508862 12204 consensus_queue.cc:260] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8124, Committed index: 8124, Last appended: 3.8125, Last appended by leader: 8125, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:59.509115 12204 logging.cc:424] LogThrottler TrackedPeer Lag: suppressed but not reported on 1 messages since previous log ~8 seconds ago
I20250113 08:55:59.509232 12204 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 3 FOLLOWER]: Advancing to term 4
W20250113 08:55:59.509466 12428 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57968: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.014s ago.
W20250113 08:55:59.510852 12428 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57968: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.016s ago.
I20250113 08:55:59.512009 12837 raft_consensus.cc:1235] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 4 FOLLOWER]: Rejecting Update request from peer aa0b5c78b11d4d07bb963ae1d9a42b3d for earlier term 3. Current term is 4. Ops: [3.8125-3.8125]
I20250113 08:55:59.513872 12204 raft_consensus.cc:2405] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b7d3e63990b74f76afc0e6cd5afbc13b for term 4 because replica has last-logged OpId of term: 3 index: 8125, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8124.
I20250113 08:55:59.514153 12837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 4 candidate_status { last_received { term: 3 index: 8124 } } ignore_live_leader: true dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac"
I20250113 08:55:59.514402 12837 raft_consensus.cc:2388] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b7d3e63990b74f76afc0e6cd5afbc13b in current term 4: Already voted for candidate 4a1af392ea8543f4827f17fdff2eaaac in this term.
I20250113 08:55:59.515473 13176 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b7d3e63990b74f76afc0e6cd5afbc13b; no voters: 4a1af392ea8543f4827f17fdff2eaaac, aa0b5c78b11d4d07bb963ae1d9a42b3d
W20250113 08:55:59.518188 12793 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48304: Illegal state: replica 4a1af392ea8543f4827f17fdff2eaaac is not leader of this config: current role FOLLOWER
W20250113 08:55:59.518325 12798 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48304: Illegal state: replica 4a1af392ea8543f4827f17fdff2eaaac is not leader of this config: current role FOLLOWER
I20250113 08:55:59.522099 13243 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 4 election: Requested vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:55:59.524678 13242 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 5 election: Requested vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:55:59.528956 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "4a1af392ea8543f4827f17fdff2eaaac" candidate_term: 4 candidate_status { last_received { term: 3 index: 8124 } } ignore_live_leader: true dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d"
I20250113 08:55:59.529229 12839 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [CANDIDATE]: Term 4 election: Requested vote from peers b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:55:59.529759 12204 raft_consensus.cc:2405] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1af392ea8543f4827f17fdff2eaaac for term 4 because replica has last-logged OpId of term: 3 index: 8125, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8124.
I20250113 08:55:59.530104 12838 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250113 08:55:59.530436 12838 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:55:59.533308 12838 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:59.541311 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "4a1af392ea8543f4827f17fdff2eaaac" candidate_term: 5 candidate_status { last_received { term: 3 index: 8124 } } ignore_live_leader: true dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d"
I20250113 08:55:59.541672 12204 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 4 FOLLOWER]: Advancing to term 5
I20250113 08:55:59.544857 12204 raft_consensus.cc:2405] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1af392ea8543f4827f17fdff2eaaac for term 5 because replica has last-logged OpId of term: 3 index: 8125, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8124.
I20250113 08:55:59.541103 12838 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [CANDIDATE]: Term 5 election: Requested vote from peers b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:55:59.549860 13242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "4a1af392ea8543f4827f17fdff2eaaac" candidate_term: 4 candidate_status { last_received { term: 3 index: 8124 } } ignore_live_leader: true dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b"
I20250113 08:55:59.549865 13243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "4a1af392ea8543f4827f17fdff2eaaac" candidate_term: 5 candidate_status { last_received { term: 3 index: 8124 } } ignore_live_leader: true dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b"
I20250113 08:55:59.550961 13243 raft_consensus.cc:2388] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1af392ea8543f4827f17fdff2eaaac in current term 5: Already voted for candidate b7d3e63990b74f76afc0e6cd5afbc13b in this term.
I20250113 08:55:59.551386 12772 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1af392ea8543f4827f17fdff2eaaac; no voters: aa0b5c78b11d4d07bb963ae1d9a42b3d, b7d3e63990b74f76afc0e6cd5afbc13b
I20250113 08:55:59.553262 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 5 candidate_status { last_received { term: 3 index: 8124 } } ignore_live_leader: true dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d"
I20250113 08:55:59.553649 12204 raft_consensus.cc:2405] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b7d3e63990b74f76afc0e6cd5afbc13b for term 5 because replica has last-logged OpId of term: 3 index: 8125, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8124.
I20250113 08:55:59.554317 12838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 5 candidate_status { last_received { term: 3 index: 8124 } } ignore_live_leader: true dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac"
I20250113 08:55:59.554646 12838 raft_consensus.cc:2388] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b7d3e63990b74f76afc0e6cd5afbc13b in current term 5: Already voted for candidate 4a1af392ea8543f4827f17fdff2eaaac in this term.
I20250113 08:55:59.555130 13176 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b7d3e63990b74f76afc0e6cd5afbc13b; no voters: 4a1af392ea8543f4827f17fdff2eaaac, aa0b5c78b11d4d07bb963ae1d9a42b3d
I20250113 08:55:59.555792 13613 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:55:59.555986 13613 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250113 08:55:59.565761 13617 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250113 08:55:59.567036 12772 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1af392ea8543f4827f17fdff2eaaac; no voters: aa0b5c78b11d4d07bb963ae1d9a42b3d, b7d3e63990b74f76afc0e6cd5afbc13b
I20250113 08:55:59.568253 13617 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250113 08:55:59.576941 13200 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33078: Illegal state: replica b7d3e63990b74f76afc0e6cd5afbc13b is not leader of this config: current role FOLLOWER
W20250113 08:55:59.578189 13198 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33078: Illegal state: replica b7d3e63990b74f76afc0e6cd5afbc13b is not leader of this config: current role FOLLOWER
W20250113 08:55:59.616736 12165 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57968: Illegal state: replica aa0b5c78b11d4d07bb963ae1d9a42b3d is not leader of this config: current role FOLLOWER
W20250113 08:55:59.616945 12162 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57968: Illegal state: replica aa0b5c78b11d4d07bb963ae1d9a42b3d is not leader of this config: current role FOLLOWER
W20250113 08:55:59.624259 12798 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48304: Illegal state: replica 4a1af392ea8543f4827f17fdff2eaaac is not leader of this config: current role FOLLOWER
W20250113 08:55:59.624485 12793 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48304: Illegal state: replica 4a1af392ea8543f4827f17fdff2eaaac is not leader of this config: current role FOLLOWER
W20250113 08:55:59.642693 13198 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33078: Illegal state: replica b7d3e63990b74f76afc0e6cd5afbc13b is not leader of this config: current role FOLLOWER
W20250113 08:55:59.643473 13200 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33078: Illegal state: replica b7d3e63990b74f76afc0e6cd5afbc13b is not leader of this config: current role FOLLOWER
W20250113 08:55:59.658504 12162 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57968: Illegal state: replica aa0b5c78b11d4d07bb963ae1d9a42b3d is not leader of this config: current role FOLLOWER
W20250113 08:55:59.658475 12165 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57968: Illegal state: replica aa0b5c78b11d4d07bb963ae1d9a42b3d is not leader of this config: current role FOLLOWER
W20250113 08:55:59.683127 12793 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48304: Illegal state: replica 4a1af392ea8543f4827f17fdff2eaaac is not leader of this config: current role FOLLOWER
W20250113 08:55:59.684579 12793 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48304: Illegal state: replica 4a1af392ea8543f4827f17fdff2eaaac is not leader of this config: current role FOLLOWER
W20250113 08:55:59.707600 13198 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33078: Illegal state: replica b7d3e63990b74f76afc0e6cd5afbc13b is not leader of this config: current role FOLLOWER
W20250113 08:55:59.707707 13200 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33078: Illegal state: replica b7d3e63990b74f76afc0e6cd5afbc13b is not leader of this config: current role FOLLOWER
W20250113 08:55:59.731431 12160 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57968: Illegal state: replica aa0b5c78b11d4d07bb963ae1d9a42b3d is not leader of this config: current role FOLLOWER
W20250113 08:55:59.733350 12160 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57968: Illegal state: replica aa0b5c78b11d4d07bb963ae1d9a42b3d is not leader of this config: current role FOLLOWER
W20250113 08:55:59.748980 12793 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48304: Illegal state: replica 4a1af392ea8543f4827f17fdff2eaaac is not leader of this config: current role FOLLOWER
W20250113 08:55:59.752562 12793 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48304: Illegal state: replica 4a1af392ea8543f4827f17fdff2eaaac is not leader of this config: current role FOLLOWER
W20250113 08:55:59.767978 13200 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33078: Illegal state: replica b7d3e63990b74f76afc0e6cd5afbc13b is not leader of this config: current role FOLLOWER
W20250113 08:55:59.771137 13200 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33078: Illegal state: replica b7d3e63990b74f76afc0e6cd5afbc13b is not leader of this config: current role FOLLOWER
W20250113 08:55:59.789428 12164 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57968: Illegal state: replica aa0b5c78b11d4d07bb963ae1d9a42b3d is not leader of this config: current role FOLLOWER
W20250113 08:55:59.789428 12160 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57968: Illegal state: replica aa0b5c78b11d4d07bb963ae1d9a42b3d is not leader of this config: current role FOLLOWER
W20250113 08:55:59.809176 12793 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48304: Illegal state: replica 4a1af392ea8543f4827f17fdff2eaaac is not leader of this config: current role FOLLOWER
W20250113 08:55:59.818642 13572 raft_consensus.cc:668] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: failed to trigger leader election: Illegal state: leader elections are disabled
W20250113 08:55:59.830340 13200 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33078: Illegal state: replica b7d3e63990b74f76afc0e6cd5afbc13b is not leader of this config: current role FOLLOWER
W20250113 08:55:59.847636 13200 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33078: Illegal state: replica b7d3e63990b74f76afc0e6cd5afbc13b is not leader of this config: current role FOLLOWER
W20250113 08:55:59.853142 12161 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57968: Illegal state: replica aa0b5c78b11d4d07bb963ae1d9a42b3d is not leader of this config: current role FOLLOWER
W20250113 08:55:59.866322 12793 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48304: Illegal state: replica 4a1af392ea8543f4827f17fdff2eaaac is not leader of this config: current role FOLLOWER
I20250113 08:55:59.866770 13618 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 08:55:59.866954 13618 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:59.867889 13618 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:55:59.868769 12838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 6 candidate_status { last_received { term: 3 index: 8124 } } ignore_live_leader: false dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac" is_pre_election: true
I20250113 08:55:59.869021 12838 raft_consensus.cc:2463] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7d3e63990b74f76afc0e6cd5afbc13b in term 5.
I20250113 08:55:59.869354 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 6 candidate_status { last_received { term: 3 index: 8124 } } ignore_live_leader: false dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" is_pre_election: true
I20250113 08:55:59.869640 12204 raft_consensus.cc:2405] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b7d3e63990b74f76afc0e6cd5afbc13b for term 6 because replica has last-logged OpId of term: 3 index: 8125, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8124.
I20250113 08:55:59.870221 13176 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1af392ea8543f4827f17fdff2eaaac, b7d3e63990b74f76afc0e6cd5afbc13b; no voters:
I20250113 08:55:59.870779 13618 raft_consensus.cc:2798] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250113 08:55:59.870914 13618 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 08:55:59.871017 13618 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 5 FOLLOWER]: Advancing to term 6
W20250113 08:55:59.873255 12793 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48304: Illegal state: replica 4a1af392ea8543f4827f17fdff2eaaac is not leader of this config: current role FOLLOWER
I20250113 08:55:59.874617 13618 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:59.875689 12838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 6 candidate_status { last_received { term: 3 index: 8124 } } ignore_live_leader: false dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac"
I20250113 08:55:59.875903 12838 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:55:59.878738 12838 raft_consensus.cc:2463] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7d3e63990b74f76afc0e6cd5afbc13b in term 6.
I20250113 08:55:59.879786 13176 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1af392ea8543f4827f17fdff2eaaac, b7d3e63990b74f76afc0e6cd5afbc13b; no voters:
I20250113 08:55:59.880177 13613 raft_consensus.cc:2798] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 6 FOLLOWER]: Leader election won for term 6
I20250113 08:55:59.884568 12204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" candidate_term: 6 candidate_status { last_received { term: 3 index: 8124 } } ignore_live_leader: false dest_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d"
I20250113 08:55:59.884805 12204 raft_consensus.cc:3054] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 5 FOLLOWER]: Advancing to term 6
I20250113 08:55:59.889025 12204 raft_consensus.cc:2405] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b7d3e63990b74f76afc0e6cd5afbc13b for term 6 because replica has last-logged OpId of term: 3 index: 8125, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 8124.
W20250113 08:55:59.890084 12164 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57968: Illegal state: replica aa0b5c78b11d4d07bb963ae1d9a42b3d is not leader of this config: current role FOLLOWER
I20250113 08:55:59.890551 13613 raft_consensus.cc:695] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [term 6 LEADER]: Becoming Leader. State: Replica: b7d3e63990b74f76afc0e6cd5afbc13b, State: Running, Role: LEADER
I20250113 08:55:59.891155 13613 consensus_queue.cc:237] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8124, Committed index: 8124, Last appended: 3.8124, Last appended by leader: 8124, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:55:59.895089 11792 catalog_manager.cc:5526] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b reported cstate change: term changed from 3 to 6, leader changed from aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131) to b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132). New cstate: current_term: 6 leader_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } health_report { overall_health: UNKNOWN } } }
I20250113 08:55:59.897660 13618 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [CANDIDATE]: Term 6 election: Requested vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:55:59.988464 12838 raft_consensus.cc:1270] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 6 FOLLOWER]: Refusing update from remote peer b7d3e63990b74f76afc0e6cd5afbc13b: Log matching property violated. Preceding OpId in replica: term: 3 index: 8124. Preceding OpId from leader: term: 6 index: 8125. (index mismatch)
I20250113 08:55:59.989303 13618 consensus_queue.cc:1035] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8125, Last known committed idx: 8124, Time since last communication: 0.000s
I20250113 08:55:59.996163 12204 raft_consensus.cc:1270] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Refusing update from remote peer b7d3e63990b74f76afc0e6cd5afbc13b: Log matching property violated. Preceding OpId in replica: term: 3 index: 8125. Preceding OpId from leader: term: 6 index: 8125. (term mismatch)
I20250113 08:55:59.996306 12204 pending_rounds.cc:77] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Aborting all ops after (but not including) 8124
I20250113 08:55:59.996377 12204 pending_rounds.cc:99] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Aborting uncommitted WRITE_OP operation due to leader change: 3.8125
W20250113 08:55:59.996734 12204 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57968: Aborted: Op aborted by new leader
I20250113 08:55:59.997745 13618 consensus_queue.cc:1035] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8125, Last known committed idx: 8124, Time since last communication: 0.000s
I20250113 08:56:01.692521 13590 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33078 (ReqId={client: 84100a491bf94f0f929066c17eae5bc2, seq_no=8123, attempt_no=17}) took 1797 ms. Trace:
I20250113 08:56:01.692694 13590 rpcz_store.cc:276] 0113 08:55:59.894855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:55:59.894931 (+ 76us) service_pool.cc:225] Handling call
0113 08:56:01.692455 (+1797524us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 08:56:01.666412 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3034bef68d124e778f0b57a39a9b152c
0113 08:56:01.666488 (+ 76us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:56:01.666493 (+ 5us) write_op.cc:435] Acquired schema lock
0113 08:56:01.666495 (+ 2us) tablet.cc:659] Decoding operations
0113 08:56:01.666846 (+ 351us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:56:01.666854 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0113 08:56:01.666858 (+ 4us) tablet.cc:682] Acquiring locks for 50 operations
0113 08:56:01.667189 (+ 331us) tablet.cc:698] Row locks acquired
0113 08:56:01.667192 (+ 3us) write_op.cc:260] PREPARE: finished
0113 08:56:01.667275 (+ 83us) write_op.cc:270] Start()
0113 08:56:01.667313 (+ 38us) write_op.cc:276] Timestamp: P: 1736758561667275 usec, L: 0
0113 08:56:01.667321 (+ 8us) op_driver.cc:348] REPLICATION: starting
0113 08:56:01.667548 (+ 227us) log.cc:844] Serialized 2467 byte log entry
0113 08:56:01.672670 (+ 5122us) spinlock_profiling.cc:243] Waited 4.4 ms on lock 0x6160004741b8. stack: 00007f9f2b58afd6 00007f9f2b58ad90 00007f9f2a56a48d 000000000096d544 000000000096d50d 000000000096ceda 00007f9f36d64f23 00007f9f36d58ff3 00007f9f36d568d7 00007f9f36d2e8f7 00007f9f36d36c4f 00007f9f36d3693d 0000000000a71ecc 00007f9f2b5d1b64 00007f9f2b5d518b 00007f9f2b5d5008
0113 08:56:01.688998 (+ 16328us) op_driver.cc:464] REPLICATION: finished
0113 08:56:01.689794 (+ 796us) write_op.cc:301] APPLY: starting
0113 08:56:01.689820 (+ 26us) tablet.cc:1364] starting BulkCheckPresence
0113 08:56:01.689990 (+ 170us) tablet.cc:1367] finished BulkCheckPresence
0113 08:56:01.689995 (+ 5us) tablet.cc:1369] starting ApplyRowOperation cycle
0113 08:56:01.691719 (+ 1724us) tablet.cc:1380] finished ApplyRowOperation cycle
0113 08:56:01.691811 (+ 92us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0113 08:56:01.691824 (+ 13us) write_op.cc:312] APPLY: finished
0113 08:56:01.692097 (+ 273us) log.cc:844] Serialized 320 byte log entry
0113 08:56:01.692185 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:56:01.692246 (+ 61us) write_op.cc:454] Released schema lock
0113 08:56:01.692260 (+ 14us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":712,"mutex_wait_us":29,"num_ops":50,"prepare.queue_time_us":1770985,"prepare.run_cpu_time_us":7425,"prepare.run_wall_time_us":8379,"raft.queue_time_us":5390,"raft.run_cpu_time_us":1301,"raft.run_wall_time_us":5605,"replication_time_us":21642,"spinlock_wait_cycles":10129280,"thread_start_us":7050,"threads_started":1}]]}
I20250113 08:56:01.709414 13590 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33078 (ReqId={client: 84100a491bf94f0f929066c17eae5bc2, seq_no=8121, attempt_no=1}) took 1701 ms. Trace:
I20250113 08:56:01.709583 13590 rpcz_store.cc:276] 0113 08:56:00.008413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:56:00.008553 (+ 140us) service_pool.cc:225] Handling call
0113 08:56:01.709389 (+1700836us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 08:56:01.676022 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3034bef68d124e778f0b57a39a9b152c
0113 08:56:01.676151 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:56:01.676156 (+ 5us) write_op.cc:435] Acquired schema lock
0113 08:56:01.676158 (+ 2us) tablet.cc:659] Decoding operations
0113 08:56:01.676528 (+ 370us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:56:01.676537 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0113 08:56:01.676541 (+ 4us) tablet.cc:682] Acquiring locks for 50 operations
0113 08:56:01.676857 (+ 316us) tablet.cc:698] Row locks acquired
0113 08:56:01.676860 (+ 3us) write_op.cc:260] PREPARE: finished
0113 08:56:01.676903 (+ 43us) write_op.cc:270] Start()
0113 08:56:01.676930 (+ 27us) write_op.cc:276] Timestamp: P: 1736758561676907 usec, L: 0
0113 08:56:01.676933 (+ 3us) op_driver.cc:348] REPLICATION: starting
0113 08:56:01.677138 (+ 205us) log.cc:844] Serialized 2467 byte log entry
0113 08:56:01.707696 (+ 30558us) op_driver.cc:464] REPLICATION: finished
0113 08:56:01.707921 (+ 225us) write_op.cc:301] APPLY: starting
0113 08:56:01.707933 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0113 08:56:01.708015 (+ 82us) tablet.cc:1367] finished BulkCheckPresence
0113 08:56:01.708041 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0113 08:56:01.708910 (+ 869us) tablet.cc:1380] finished ApplyRowOperation cycle
0113 08:56:01.708979 (+ 69us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0113 08:56:01.708987 (+ 8us) write_op.cc:312] APPLY: finished
0113 08:56:01.709144 (+ 157us) log.cc:844] Serialized 320 byte log entry
0113 08:56:01.709240 (+ 96us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:56:01.709287 (+ 47us) write_op.cc:454] Released schema lock
0113 08:56:01.709296 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":172,"num_ops":50,"prepare.queue_time_us":1666784,"prepare.run_cpu_time_us":1175,"prepare.run_wall_time_us":1173,"replication_time_us":30736,"spinlock_wait_cycles":1152}]]}
I20250113 08:56:01.710450 13492 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33078 (ReqId={client: 84100a491bf94f0f929066c17eae5bc2, seq_no=8124, attempt_no=16}) took 1802 ms. Trace:
I20250113 08:56:01.710561 13492 rpcz_store.cc:276] 0113 08:55:59.907851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0113 08:55:59.907930 (+ 79us) service_pool.cc:225] Handling call
0113 08:56:01.710386 (+1802456us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0113 08:56:01.674829 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3034bef68d124e778f0b57a39a9b152c
0113 08:56:01.674944 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0113 08:56:01.674950 (+ 6us) write_op.cc:435] Acquired schema lock
0113 08:56:01.674954 (+ 4us) tablet.cc:659] Decoding operations
0113 08:56:01.675337 (+ 383us) write_op.cc:620] Acquiring the partition lock for write op
0113 08:56:01.675347 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0113 08:56:01.675351 (+ 4us) tablet.cc:682] Acquiring locks for 50 operations
0113 08:56:01.675659 (+ 308us) tablet.cc:698] Row locks acquired
0113 08:56:01.675663 (+ 4us) write_op.cc:260] PREPARE: finished
0113 08:56:01.675712 (+ 49us) write_op.cc:270] Start()
0113 08:56:01.675738 (+ 26us) write_op.cc:276] Timestamp: P: 1736758561675716 usec, L: 0
0113 08:56:01.675742 (+ 4us) op_driver.cc:348] REPLICATION: starting
0113 08:56:01.675904 (+ 162us) log.cc:844] Serialized 2467 byte log entry
0113 08:56:01.707204 (+ 31300us) op_driver.cc:464] REPLICATION: finished
0113 08:56:01.707816 (+ 612us) write_op.cc:301] APPLY: starting
0113 08:56:01.707841 (+ 25us) tablet.cc:1364] starting BulkCheckPresence
0113 08:56:01.707979 (+ 138us) tablet.cc:1367] finished BulkCheckPresence
0113 08:56:01.708014 (+ 35us) tablet.cc:1369] starting ApplyRowOperation cycle
0113 08:56:01.709947 (+ 1933us) tablet.cc:1380] finished ApplyRowOperation cycle
0113 08:56:01.710002 (+ 55us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0113 08:56:01.710014 (+ 12us) write_op.cc:312] APPLY: finished
0113 08:56:01.710167 (+ 153us) log.cc:844] Serialized 320 byte log entry
0113 08:56:01.710247 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0113 08:56:01.710295 (+ 48us) write_op.cc:454] Released schema lock
0113 08:56:01.710309 (+ 14us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":530,"num_ops":50,"prepare.queue_time_us":1766096,"prepare.run_cpu_time_us":1127,"prepare.run_wall_time_us":1125,"replication_time_us":31433}]]}
W20250113 08:56:01.882550 12392 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.11.124.132:36713, user_credentials={real_user=slave}} blocked reactor thread for 72597.3us
I20250113 08:56:02.999280 12185 tablet_service.cc:1422] Tablet server aa0b5c78b11d4d07bb963ae1d9a42b3d set to quiescing
I20250113 08:56:02.999507 12185 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250113 08:56:07.437474 12185 tablet_service.cc:1422] Tablet server aa0b5c78b11d4d07bb963ae1d9a42b3d set to quiescing
I20250113 08:56:07.437718 12185 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250113 08:56:11.798439 12185 tablet_service.cc:1422] Tablet server aa0b5c78b11d4d07bb963ae1d9a42b3d set to quiescing
I20250113 08:56:11.798720 12185 tablet_service.cc:1429] Tablet server has 0 leaders and 2 scanners
I20250113 08:56:16.245920 12185 tablet_service.cc:1422] Tablet server aa0b5c78b11d4d07bb963ae1d9a42b3d set to quiescing
I20250113 08:56:16.246172 12185 tablet_service.cc:1429] Tablet server has 0 leaders and 1 scanners
I20250113 08:56:20.011250 12185 tablet_service.cc:1422] Tablet server aa0b5c78b11d4d07bb963ae1d9a42b3d set to quiescing
I20250113 08:56:20.011518 12185 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:21.904906 11762 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskqePfU0/build/asan/bin/kudu pid 12122
I20250113 08:56:21.942376 12244 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:56:22.470402 11762 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskqePfU0/build/asan/bin/kudu with pid 12122
W20250113 08:56:22.563089 13176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.124.131:39891 (error 108)
I20250113 08:56:22.564859 11762 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskqePfU0/build/asan/bin/kudu
/tmp/dist-test-taskqePfU0/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.124.131:39891
--local_ip_for_outbound_sockets=127.11.124.131
--tserver_master_addrs=127.11.124.190:36259
--webserver_port=37733
--webserver_interface=127.11.124.131
--builtin_ntp_servers=127.11.124.148:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--scanner_default_batch_size_bytes=100 with env {}
W20250113 08:56:22.568698 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.131:39891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 08:56:23.037285 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.131:39891: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250113 08:56:23.144317 13768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:56:23.144857 13768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 08:56:23.145047 13768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:56:23.160027 13768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 08:56:23.160465 13768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.124.131
I20250113 08:56:23.175740 13768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.124.148:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.124.131:39891
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.124.131
--webserver_port=37733
--tserver_master_addrs=127.11.124.190:36259
--scanner_default_batch_size_bytes=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.124.131
--log_dir=/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 08:43:25 UTC on 24a791456cd2
build id 4155
ASAN enabled
I20250113 08:56:23.178190 13768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 08:56:23.180616 13768 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 08:56:23.234689 13774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 08:56:23.319774 13775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:23.336104 13768 server_base.cc:1034] running on GCE node
W20250113 08:56:23.337818 13778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 08:56:23.340202 13768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 08:56:23.346781 13768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 08:56:23.348951 13768 hybrid_clock.cc:648] HybridClock initialized: now 1736758583348528 us; error 39 us; skew 500 ppm
I20250113 08:56:23.349591 13768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 08:56:23.368280 13768 webserver.cc:458] Webserver started at http://127.11.124.131:37733/ using document root <none> and password file <none>
I20250113 08:56:23.369596 13768 fs_manager.cc:362] Metadata directory not provided
I20250113 08:56:23.369832 13768 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 08:56:23.403929 13768 fs_manager.cc:714] Time spent opening directory manager: real 0.030s user 0.012s sys 0.017s
I20250113 08:56:23.425556 13784 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 08:56:23.430452 13768 fs_manager.cc:730] Time spent opening block manager: real 0.024s user 0.011s sys 0.007s
I20250113 08:56:23.430810 13768 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/wal
uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d"
format_stamp: "Formatted at 2025-01-13 08:54:32 on dist-test-slave-kk7k"
I20250113 08:56:23.432574 13768 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 08:56:23.489584 13768 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 08:56:23.491582 13768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 08:56:23.492121 13768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 08:56:23.495103 13768 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 08:56:23.505822 13791 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20250113 08:56:23.524233 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.131:39891: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250113 08:56:23.525718 13768 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250113 08:56:23.526418 13768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.022s user 0.000s sys 0.000s
I20250113 08:56:23.526648 13768 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250113 08:56:23.531013 13768 ts_tablet_manager.cc:610] Registered 1 tablets
I20250113 08:56:23.531184 13768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250113 08:56:23.539479 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap starting.
I20250113 08:56:24.022576 13791 log.cc:826] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Log is configured to *not* fsync() on all Append() calls
W20250113 08:56:24.061522 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.131:39891: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250113 08:56:24.646700 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.131:39891: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250113 08:56:24.957850 13899 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.124.131:39891 every 8 connection(s)
I20250113 08:56:24.958258 13768 rpc_server.cc:307] RPC server started. Bound to: 127.11.124.131:39891
I20250113 08:56:24.960693 13768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data/info.pb
I20250113 08:56:24.964213 11762 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskqePfU0/build/asan/bin/kudu as pid 13768
I20250113 08:56:25.032124 13900 heartbeater.cc:346] Connected to a master server at 127.11.124.190:36259
I20250113 08:56:25.032558 13900 heartbeater.cc:463] Registering TS with master...
I20250113 08:56:25.033865 13900 heartbeater.cc:510] Master 127.11.124.190:36259 requested a full tablet report, sending...
I20250113 08:56:25.037240 11792 ts_manager.cc:194] Re-registered known tserver with Master: aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891)
I20250113 08:56:25.038784 11792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.124.131:36989
W20250113 08:56:25.202279 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 26: this message will repeat every 5th retry.
W20250113 08:56:25.733183 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 31: this message will repeat every 5th retry.
I20250113 08:56:26.044340 13900 heartbeater.cc:502] Master 127.11.124.190:36259 was elected leader, sending a full tablet report...
W20250113 08:56:26.595398 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 36: this message will repeat every 5th retry.
I20250113 08:56:26.624774 13880 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 6 ops behind the committed index
I20250113 08:56:27.081635 13834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:27.115873 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250113 08:56:27.125674 13223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:56:27.186242 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:56:27.385727 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 41: this message will repeat every 5th retry.
W20250113 08:56:27.942133 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 46: this message will repeat every 5th retry.
W20250113 08:56:28.460480 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 51: this message will repeat every 5th retry.
W20250113 08:56:29.000118 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 56: this message will repeat every 5th retry.
I20250113 08:56:29.032829 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap replaying log segment 1/3 (1.19M/8.00M this segment, stats: ops{read=715 overwritten=0 applied=713 ignored=0} inserts{seen=35600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:56:29.623356 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 61: this message will repeat every 5th retry.
I20250113 08:56:29.659593 13947 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 304 ops behind the committed index [suppressed 24 similar messages]
W20250113 08:56:30.223045 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 66: this message will repeat every 5th retry.
W20250113 08:56:30.738855 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 71: this message will repeat every 5th retry.
W20250113 08:56:31.334940 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 76: this message will repeat every 5th retry.
I20250113 08:56:31.363157 11790 maintenance_manager.cc:418] P a4ab4eabb3c943d9bcd33a4ccc2938a4: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033390
I20250113 08:56:31.458878 11789 maintenance_manager.cc:642] P a4ab4eabb3c943d9bcd33a4ccc2938a4: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.095s user 0.018s sys 0.019s Metrics: {"bytes_written":6976,"cfile_init":1,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1109,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":9,"thread_start_us":6356,"threads_started":1,"wal-append.queue_time_us":6470}
I20250113 08:56:31.460633 11790 maintenance_manager.cc:418] P a4ab4eabb3c943d9bcd33a4ccc2938a4: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 631 bytes on disk
I20250113 08:56:31.461694 11789 maintenance_manager.cc:642] P a4ab4eabb3c943d9bcd33a4ccc2938a4: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20250113 08:56:31.948238 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 81: this message will repeat every 5th retry.
W20250113 08:56:32.355893 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 86: this message will repeat every 5th retry.
I20250113 08:56:32.660218 13962 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 629 ops behind the committed index [suppressed 26 similar messages]
W20250113 08:56:32.982451 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 91: this message will repeat every 5th retry.
W20250113 08:56:33.645943 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 96: this message will repeat every 5th retry.
I20250113 08:56:34.034201 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap replaying log segment 1/3 (2.89M/8.00M this segment, stats: ops{read=1728 overwritten=0 applied=1728 ignored=0} inserts{seen=86350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
I20250113 08:56:34.088454 13834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:34.116114 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250113 08:56:34.136771 13223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:56:34.239931 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:56:34.256989 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 101: this message will repeat every 5th retry.
W20250113 08:56:34.764959 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 106: this message will repeat every 5th retry.
W20250113 08:56:34.921231 12392 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.11.124.132:36713, user_credentials={real_user=slave}} blocked reactor thread for 68944.9us
W20250113 08:56:35.294559 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 111: this message will repeat every 5th retry.
I20250113 08:56:35.722924 13962 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 879 ops behind the committed index [suppressed 24 similar messages]
W20250113 08:56:35.933969 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 116: this message will repeat every 5th retry.
W20250113 08:56:36.469718 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 121: this message will repeat every 5th retry.
W20250113 08:56:36.994680 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 126: this message will repeat every 5th retry.
W20250113 08:56:37.540560 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 131: this message will repeat every 5th retry.
W20250113 08:56:38.018604 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 136: this message will repeat every 5th retry.
W20250113 08:56:38.578951 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 141: this message will repeat every 5th retry.
I20250113 08:56:38.775336 14014 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 1204 ops behind the committed index [suppressed 28 similar messages]
I20250113 08:56:39.043232 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap replaying log segment 1/3 (4.49M/8.00M this segment, stats: ops{read=2670 overwritten=0 applied=2668 ignored=0} inserts{seen=133350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:56:39.138337 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 146: this message will repeat every 5th retry.
W20250113 08:56:39.586730 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 151: this message will repeat every 5th retry.
W20250113 08:56:40.063899 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 156: this message will repeat every 5th retry.
W20250113 08:56:40.688827 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 161: this message will repeat every 5th retry.
W20250113 08:56:41.247649 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 166: this message will repeat every 5th retry.
I20250113 08:56:41.294981 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250113 08:56:41.331818 13834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:41.391821 13223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:56:41.421113 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:41.863338 14014 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 1491 ops behind the committed index [suppressed 27 similar messages]
W20250113 08:56:41.910254 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 171: this message will repeat every 5th retry.
W20250113 08:56:42.400537 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 176: this message will repeat every 5th retry.
W20250113 08:56:42.914906 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 181: this message will repeat every 5th retry.
W20250113 08:56:43.265421 13176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.11.124.129:40671, user_credentials={real_user=slave}} blocked reactor thread for 45552.5us
W20250113 08:56:43.508256 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 186: this message will repeat every 5th retry.
W20250113 08:56:43.948573 12392 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.11.124.132:36713, user_credentials={real_user=slave}} blocked reactor thread for 65005.8us
W20250113 08:56:44.018972 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 191: this message will repeat every 5th retry.
I20250113 08:56:44.045825 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap replaying log segment 1/3 (6.00M/8.00M this segment, stats: ops{read=3561 overwritten=0 applied=3559 ignored=0} inserts{seen=177900 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:56:44.593097 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 196: this message will repeat every 5th retry.
I20250113 08:56:44.872882 14066 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 1801 ops behind the committed index [suppressed 27 similar messages]
W20250113 08:56:45.138638 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 201: this message will repeat every 5th retry.
W20250113 08:56:45.731178 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 206: this message will repeat every 5th retry.
W20250113 08:56:46.277002 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 211: this message will repeat every 5th retry.
W20250113 08:56:46.760994 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 216: this message will repeat every 5th retry.
W20250113 08:56:46.804697 12392 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.11.124.132:36713, user_credentials={real_user=slave}} blocked reactor thread for 65472.8us
W20250113 08:56:47.331797 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 221: this message will repeat every 5th retry.
W20250113 08:56:47.854969 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 226: this message will repeat every 5th retry.
I20250113 08:56:47.873900 14065 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 2084 ops behind the committed index [suppressed 27 similar messages]
W20250113 08:56:48.391184 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 231: this message will repeat every 5th retry.
I20250113 08:56:48.606663 13834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:48.632450 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250113 08:56:48.658982 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:48.685931 13223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20250113 08:56:48.931505 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 236: this message will repeat every 5th retry.
I20250113 08:56:49.045948 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap replaying log segment 1/3 (7.56M/8.00M this segment, stats: ops{read=4474 overwritten=0 applied=4474 ignored=0} inserts{seen=223600 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:56:49.423950 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 241: this message will repeat every 5th retry.
W20250113 08:56:49.973481 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 246: this message will repeat every 5th retry.
I20250113 08:56:50.421561 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap replayed 1/3 log segments. Stats: ops{read=4729 overwritten=0 applied=4728 ignored=0} inserts{seen=236300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
W20250113 08:56:50.572588 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 251: this message will repeat every 5th retry.
I20250113 08:56:50.948258 14110 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 2393 ops behind the committed index [suppressed 27 similar messages]
W20250113 08:56:51.077633 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 256: this message will repeat every 5th retry.
W20250113 08:56:51.749550 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 261: this message will repeat every 5th retry.
W20250113 08:56:52.263043 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 266: this message will repeat every 5th retry.
W20250113 08:56:52.821091 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 271: this message will repeat every 5th retry.
W20250113 08:56:53.387596 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 276: this message will repeat every 5th retry.
W20250113 08:56:53.932533 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 281: this message will repeat every 5th retry.
I20250113 08:56:54.011464 14110 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 2729 ops behind the committed index [suppressed 26 similar messages]
I20250113 08:56:54.048369 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap replaying log segment 2/3 (1.34M/8.00M this segment, stats: ops{read=5501 overwritten=0 applied=5501 ignored=0} inserts{seen=274950 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:56:54.512533 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 286: this message will repeat every 5th retry.
W20250113 08:56:55.069295 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 291: this message will repeat every 5th retry.
W20250113 08:56:55.641423 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 296: this message will repeat every 5th retry.
W20250113 08:56:56.157716 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 301: this message will repeat every 5th retry.
I20250113 08:56:56.671813 13834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:56.678706 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250113 08:56:56.858397 13223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20250113 08:56:56.889662 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 306: this message will repeat every 5th retry.
I20250113 08:56:56.891407 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:56:57.030829 14110 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 3014 ops behind the committed index [suppressed 25 similar messages]
W20250113 08:56:57.473266 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 311: this message will repeat every 5th retry.
W20250113 08:56:57.903712 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 316: this message will repeat every 5th retry.
W20250113 08:56:58.469444 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 321: this message will repeat every 5th retry.
W20250113 08:56:58.954166 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 326: this message will repeat every 5th retry.
I20250113 08:56:59.052237 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap replaying log segment 2/3 (2.68M/8.00M this segment, stats: ops{read=6276 overwritten=0 applied=6276 ignored=0} inserts{seen=313700 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:56:59.113224 12392 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.11.124.132:36713, user_credentials={real_user=slave}} blocked reactor thread for 81083.7us
W20250113 08:56:59.419296 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 331: this message will repeat every 5th retry.
W20250113 08:56:59.916213 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 336: this message will repeat every 5th retry.
I20250113 08:57:00.126703 14122 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 3321 ops behind the committed index [suppressed 29 similar messages]
W20250113 08:57:00.485868 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 341: this message will repeat every 5th retry.
W20250113 08:57:00.958112 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 346: this message will repeat every 5th retry.
W20250113 08:57:01.491487 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 351: this message will repeat every 5th retry.
W20250113 08:57:02.021878 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 356: this message will repeat every 5th retry.
I20250113 08:57:02.497032 12885 maintenance_manager.cc:418] P 4a1af392ea8543f4827f17fdff2eaaac: Scheduling FlushMRSOp(3034bef68d124e778f0b57a39a9b152c): perf score=0.071663
W20250113 08:57:02.529008 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 361: this message will repeat every 5th retry.
W20250113 08:57:03.050997 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 366: this message will repeat every 5th retry.
I20250113 08:57:03.128158 14162 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 3631 ops behind the committed index [suppressed 28 similar messages]
W20250113 08:57:03.596050 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 371: this message will repeat every 5th retry.
I20250113 08:57:04.053275 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap replaying log segment 2/3 (4.25M/8.00M this segment, stats: ops{read=7183 overwritten=0 applied=7183 ignored=0} inserts{seen=359050 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:57:04.227416 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 376: this message will repeat every 5th retry.
I20250113 08:57:04.577107 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250113 08:57:04.633890 13834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:57:04.635629 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:57:04.660780 13223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20250113 08:57:04.769640 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 381: this message will repeat every 5th retry.
W20250113 08:57:05.356016 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 386: this message will repeat every 5th retry.
W20250113 08:57:05.887806 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 391: this message will repeat every 5th retry.
W20250113 08:57:05.947387 12392 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.11.124.132:36713, user_credentials={real_user=slave}} blocked reactor thread for 119687us
I20250113 08:57:06.326750 14164 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 3843 ops behind the committed index [suppressed 26 similar messages]
W20250113 08:57:06.605635 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 396: this message will repeat every 5th retry.
W20250113 08:57:07.114828 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 401: this message will repeat every 5th retry.
W20250113 08:57:07.588332 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 406: this message will repeat every 5th retry.
W20250113 08:57:08.212533 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 411: this message will repeat every 5th retry.
W20250113 08:57:08.708787 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 416: this message will repeat every 5th retry.
I20250113 08:57:09.062000 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap replaying log segment 2/3 (5.43M/8.00M this segment, stats: ops{read=7864 overwritten=0 applied=7863 ignored=0} inserts{seen=393050 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:57:09.298458 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 421: this message will repeat every 5th retry.
I20250113 08:57:09.372144 14215 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 4079 ops behind the committed index [suppressed 27 similar messages]
W20250113 08:57:09.838758 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 426: this message will repeat every 5th retry.
W20250113 08:57:10.358768 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 431: this message will repeat every 5th retry.
W20250113 08:57:10.904598 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 436: this message will repeat every 5th retry.
W20250113 08:57:11.465519 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 441: this message will repeat every 5th retry.
I20250113 08:57:11.935822 13834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:57:11.953199 13223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:57:11.961341 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250113 08:57:11.989549 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:57:12.080577 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 446: this message will repeat every 5th retry.
I20250113 08:57:12.459177 14221 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 4318 ops behind the committed index [suppressed 27 similar messages]
W20250113 08:57:12.587285 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 451: this message will repeat every 5th retry.
W20250113 08:57:12.620941 12392 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.11.124.132:36713, user_credentials={real_user=slave}} blocked reactor thread for 195791us
W20250113 08:57:13.213721 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 456: this message will repeat every 5th retry.
W20250113 08:57:13.713510 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 461: this message will repeat every 5th retry.
I20250113 08:57:14.061591 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap replaying log segment 2/3 (6.48M/8.00M this segment, stats: ops{read=8484 overwritten=1 applied=8483 ignored=0} inserts{seen=424000 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:57:14.362967 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 466: this message will repeat every 5th retry.
W20250113 08:57:15.082515 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 471: this message will repeat every 5th retry.
I20250113 08:57:15.514547 14268 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 4591 ops behind the committed index [suppressed 25 similar messages]
W20250113 08:57:15.605355 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 476: this message will repeat every 5th retry.
W20250113 08:57:16.041765 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 481: this message will repeat every 5th retry.
W20250113 08:57:16.556151 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 486: this message will repeat every 5th retry.
W20250113 08:57:17.152336 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 491: this message will repeat every 5th retry.
W20250113 08:57:17.634048 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 496: this message will repeat every 5th retry.
W20250113 08:57:18.235016 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 501: this message will repeat every 5th retry.
I20250113 08:57:18.591109 14275 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 4822 ops behind the committed index [suppressed 28 similar messages]
W20250113 08:57:18.773082 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 506: this message will repeat every 5th retry.
I20250113 08:57:19.072404 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap replaying log segment 2/3 (7.61M/8.00M this segment, stats: ops{read=9153 overwritten=1 applied=9151 ignored=0} inserts{seen=457400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
W20250113 08:57:19.205163 12392 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.11.124.132:36713, user_credentials={real_user=slave}} blocked reactor thread for 121268us
W20250113 08:57:19.336167 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 511: this message will repeat every 5th retry.
I20250113 08:57:19.777431 13223 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250113 08:57:19.811543 12819 tablet_service.cc:1429] Tablet server has 0 leaders and 3 scanners
I20250113 08:57:19.830456 13834 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250113 08:57:19.848392 12557 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250113 08:57:19.908721 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 516: this message will repeat every 5th retry.
W20250113 08:57:20.448223 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 521: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/maintenance_mode-itest.cc:753: Failure
Failed
Bad status: Timed out: cluster didn't become healthy: Running 'cluster ksck 127.11.124.190:36259' did not succeed in 60 seconds: Runtime error: error running 'cluster ksck 127.11.124.190:36259': stdout: Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
a4ab4eabb3c943d9bcd33a4ccc2938a4 | 127.11.124.190:36259 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.11.124.148:37215 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server States
Server | State
----------------------------------+------------------
aa0b5c78b11d4d07bb963ae1d9a42b3d | MAINTENANCE_MODE
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
4a1af392ea8543f4827f17fdff2eaaac | 127.11.124.129:40671 | HEALTHY | /L0 | 0 | 3
aa0b5c78b11d4d07bb963ae1d9a42b3d | 127.11.124.131:39891 | HEALTHY | /L0 | 0 | 0
b7d3e63990b74f76afc0e6cd5afbc13b | 127.11.124.132:36713 | HEALTHY | /L0 | 1 | 0
cbb5dc448319472395e744e7bd16c57f | 127.11.124.130:46163 | HEALTHY | /L0 | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/L0 | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.11.124.129 | experimental | 127.11.124.129:40671
local_ip_for_outbound_sockets | 127.11.124.130 | experimental | 127.11.124.130:46163
local_ip_for_outbound_sockets | 127.11.124.131 | experimental | 127.11.124.131:39891
local_ip_for_outbound_sockets | 127.11.124.132 | experimental | 127.11.124.132:36713
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/data/info.pb | hidden | 127.11.124.129:40671
server_dump_info_path | /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-1/data/info.pb | hidden | 127.11.124.130:46163
server_dump_info_path | /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-2/data/info.pb | hidden | 127.11.124.131:39891
server_dump_info_path | /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-3/data/info.pb | hidden | 127.11.124.132:36713
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.11.124.148:37215 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
Tablet 3034bef68d124e778f0b57a39a9b152c of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING
4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): RUNNING
b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713): RUNNING [LEADER]
aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): not running
State: BOOTSTRAPPING
Data state: TABLET_DATA_READY
Last status: Bootstrap replaying log segment 2/3 (7.61M/8.00M this segment, stats: ops{read=9153 overwritten=1 applied=9151 ignored=0} inserts{seen=457400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0)
All reported replicas are:
A = 4a1af392ea8543f4827f17fdff2eaaac
B = b7d3e63990b74f76afc0e6cd5afbc13b
C = aa0b5c78b11d4d07bb963ae1d9a42b3d
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B* C | 6 | -1 | Yes
B | A B* C | 6 | -1 | Yes
C | A B C | 6 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+------------------+---------------+---------+------------+------------------+-------------
test-workload | 3 | UNDER_REPLICATED | 1 | 0 | 0 | 1 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Tablet Replica Count Outliers
Type | UUID | Host | Replica Count
-------+----------------------------------+----------------------+---------------
Small | cbb5dc448319472395e744e7bd16c57f | 127.11.124.130:46163 | 0
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
, stderr: W20250113 08:57:18.738175 14274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 08:57:18.742667 14274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 08:57:18.795408 14274 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Runtime error: ksck discovered errors
: /tmp/dist-test-taskqePfU0/build/asan/bin/kudu: process exited with non-zero status 1
W20250113 08:57:20.928920 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 526: this message will repeat every 5th retry.
I20250113 08:57:21.146418 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap replayed 2/3 log segments. Stats: ops{read=9385 overwritten=1 applied=9380 ignored=0} inserts{seen=468850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates
W20250113 08:57:21.444128 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 531: this message will repeat every 5th retry.
I20250113 08:57:21.634302 14266 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 4995 ops behind the committed index [suppressed 27 similar messages]
W20250113 08:57:21.947566 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 536: this message will repeat every 5th retry.
W20250113 08:57:22.424266 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 541: this message will repeat every 5th retry.
W20250113 08:57:22.983187 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 546: this message will repeat every 5th retry.
W20250113 08:57:23.506830 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 551: this message will repeat every 5th retry.
I20250113 08:57:23.830782 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap replayed 3/3 log segments. Stats: ops{read=10032 overwritten=1 applied=10031 ignored=0} inserts{seen=501400 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250113 08:57:23.832726 13791 tablet_bootstrap.cc:492] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Bootstrap complete.
I20250113 08:57:23.840207 13791 ts_tablet_manager.cc:1397] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Time spent bootstrapping tablet: real 60.302s user 39.494s sys 1.325s
I20250113 08:57:23.846062 13791 raft_consensus.cc:357] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:57:23.846946 13791 raft_consensus.cc:738] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0b5c78b11d4d07bb963ae1d9a42b3d, State: Initialized, Role: FOLLOWER
I20250113 08:57:23.847781 13791 consensus_queue.cc:260] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10031, Last appended: 6.10031, Last appended by leader: 10031, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:57:23.866566 13791 ts_tablet_manager.cc:1428] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d: Time spent starting tablet: real 0.026s user 0.017s sys 0.008s
I20250113 08:57:24.725203 14318 consensus_queue.cc:786] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b [LEADER]: Peer aa0b5c78b11d4d07bb963ae1d9a42b3d is lagging by at least 3720 ops behind the committed index [suppressed 24 similar messages]
I20250113 08:57:25.349385 14328 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b7d3e63990b74f76afc0e6cd5afbc13b)
I20250113 08:57:25.349772 14328 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:57:25.352643 14328 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713)
I20250113 08:57:25.412256 13242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 7 candidate_status { last_received { term: 6 index: 12156 } } ignore_live_leader: false dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" is_pre_election: true
I20250113 08:57:25.416424 12838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 7 candidate_status { last_received { term: 6 index: 12156 } } ignore_live_leader: false dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac" is_pre_election: true
I20250113 08:57:25.417326 13786 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa0b5c78b11d4d07bb963ae1d9a42b3d; no voters: 4a1af392ea8543f4827f17fdff2eaaac, b7d3e63990b74f76afc0e6cd5afbc13b
I20250113 08:57:25.417841 14328 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:57:25.749321 14328 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b7d3e63990b74f76afc0e6cd5afbc13b)
I20250113 08:57:25.749528 14328 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:57:25.750176 14328 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713)
I20250113 08:57:25.750941 13242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 7 candidate_status { last_received { term: 6 index: 12581 } } ignore_live_leader: false dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" is_pre_election: true
I20250113 08:57:25.751665 12838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 7 candidate_status { last_received { term: 6 index: 12581 } } ignore_live_leader: false dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac" is_pre_election: true
I20250113 08:57:25.752233 13786 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa0b5c78b11d4d07bb963ae1d9a42b3d; no voters: 4a1af392ea8543f4827f17fdff2eaaac, b7d3e63990b74f76afc0e6cd5afbc13b
I20250113 08:57:25.752784 14328 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:57:26.579702 14334 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b7d3e63990b74f76afc0e6cd5afbc13b)
I20250113 08:57:26.579939 14334 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:57:26.580960 14334 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713)
I20250113 08:57:26.581418 12838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 7 candidate_status { last_received { term: 6 index: 13856 } } ignore_live_leader: false dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac" is_pre_election: true
I20250113 08:57:26.588670 13242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 7 candidate_status { last_received { term: 6 index: 13856 } } ignore_live_leader: false dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" is_pre_election: true
I20250113 08:57:26.592308 13787 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa0b5c78b11d4d07bb963ae1d9a42b3d; no voters: 4a1af392ea8543f4827f17fdff2eaaac, b7d3e63990b74f76afc0e6cd5afbc13b
I20250113 08:57:26.592908 14334 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:57:27.102681 14334 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b7d3e63990b74f76afc0e6cd5afbc13b)
I20250113 08:57:27.102914 14334 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:57:27.103813 14334 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713)
I20250113 08:57:27.104439 12838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 7 candidate_status { last_received { term: 6 index: 14281 } } ignore_live_leader: false dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac" is_pre_election: true
I20250113 08:57:27.105392 13242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 7 candidate_status { last_received { term: 6 index: 14281 } } ignore_live_leader: false dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" is_pre_election: true
I20250113 08:57:27.105904 13787 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa0b5c78b11d4d07bb963ae1d9a42b3d; no voters: 4a1af392ea8543f4827f17fdff2eaaac, b7d3e63990b74f76afc0e6cd5afbc13b
I20250113 08:57:27.114118 14334 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:57:27.462693 14334 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b7d3e63990b74f76afc0e6cd5afbc13b)
I20250113 08:57:27.462901 14334 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:57:27.463721 14334 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713)
I20250113 08:57:27.464520 12838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 7 candidate_status { last_received { term: 6 index: 14706 } } ignore_live_leader: false dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac" is_pre_election: true
I20250113 08:57:27.465512 13242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 7 candidate_status { last_received { term: 6 index: 14706 } } ignore_live_leader: false dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" is_pre_election: true
I20250113 08:57:27.466094 13787 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa0b5c78b11d4d07bb963ae1d9a42b3d; no voters: 4a1af392ea8543f4827f17fdff2eaaac, b7d3e63990b74f76afc0e6cd5afbc13b
I20250113 08:57:27.466555 14334 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:57:27.807430 14334 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b7d3e63990b74f76afc0e6cd5afbc13b)
I20250113 08:57:27.807642 14334 raft_consensus.cc:513] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a1af392ea8543f4827f17fdff2eaaac" member_type: VOTER last_known_addr { host: "127.11.124.129" port: 40671 } } peers { permanent_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" member_type: VOTER last_known_addr { host: "127.11.124.132" port: 36713 } } peers { permanent_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" member_type: VOTER last_known_addr { host: "127.11.124.131" port: 39891 } }
I20250113 08:57:27.808378 14334 leader_election.cc:290] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671), b7d3e63990b74f76afc0e6cd5afbc13b (127.11.124.132:36713)
I20250113 08:57:27.809219 12838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 7 candidate_status { last_received { term: 6 index: 15131 } } ignore_live_leader: false dest_uuid: "4a1af392ea8543f4827f17fdff2eaaac" is_pre_election: true
I20250113 08:57:27.810384 13242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3034bef68d124e778f0b57a39a9b152c" candidate_uuid: "aa0b5c78b11d4d07bb963ae1d9a42b3d" candidate_term: 7 candidate_status { last_received { term: 6 index: 15131 } } ignore_live_leader: false dest_uuid: "b7d3e63990b74f76afc0e6cd5afbc13b" is_pre_election: true
I20250113 08:57:27.811236 13787 leader_election.cc:304] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa0b5c78b11d4d07bb963ae1d9a42b3d; no voters: 4a1af392ea8543f4827f17fdff2eaaac, b7d3e63990b74f76afc0e6cd5afbc13b
I20250113 08:57:27.812359 14334 raft_consensus.cc:2743] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250113 08:57:31.297883 11762 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20250113 08:57:31.298027 11762 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 4a1af392ea8543f4827f17fdff2eaaac and pid 12750
************************ BEGIN STACKS **************************
[New LWP 12754]
[New LWP 12755]
[New LWP 12756]
[New LWP 12757]
[New LWP 12763]
[New LWP 12765]
[New LWP 12767]
[New LWP 12770]
[New LWP 12771]
[New LWP 12772]
[New LWP 12773]
[New LWP 12774]
[New LWP 12775]
[New LWP 12777]
[New LWP 12778]
[New LWP 12779]
[New LWP 12780]
[New LWP 12781]
[New LWP 12782]
[New LWP 12783]
[New LWP 12784]
[New LWP 12785]
[New LWP 12786]
[New LWP 12787]
[New LWP 12788]
[New LWP 12789]
[New LWP 12790]
[New LWP 12791]
[New LWP 12792]
[New LWP 12793]
[New LWP 12794]
[New LWP 12795]
[New LWP 12796]
[New LWP 12797]
[New LWP 12798]
[New LWP 12799]
[New LWP 12800]
[New LWP 12801]
[New LWP 12802]
[New LWP 12803]
[New LWP 12804]
[New LWP 12805]
[New LWP 12806]
[New LWP 12807]
[New LWP 12808]
[New LWP 12809]
[New LWP 12810]
[New LWP 12811]
[New LWP 12813]
[New LWP 12814]
[New LWP 12815]
[New LWP 12816]
[New LWP 12817]
[New LWP 12818]
[New LWP 12819]
[New LWP 12820]
[New LWP 12821]
[New LWP 12822]
[New LWP 12823]
[New LWP 12824]
[New LWP 12825]
[New LWP 12826]
[New LWP 12827]
[New LWP 12828]
[New LWP 12829]
[New LWP 12830]
[New LWP 12831]
[New LWP 12832]
[New LWP 12833]
[New LWP 12834]
[New LWP 12835]
[New LWP 12836]
[New LWP 12837]
[New LWP 12838]
[New LWP 12839]
[New LWP 12840]
[New LWP 12841]
[New LWP 12842]
[New LWP 12843]
[New LWP 12844]
[New LWP 12845]
[New LWP 12846]
[New LWP 12847]
[New LWP 12848]
[New LWP 12849]
[New LWP 12850]
[New LWP 12851]
[New LWP 12852]
[New LWP 12853]
[New LWP 12854]
[New LWP 12855]
[New LWP 12856]
[New LWP 12857]
[New LWP 12858]
[New LWP 12859]
[New LWP 12860]
[New LWP 12861]
[New LWP 12862]
[New LWP 12863]
[New LWP 12864]
[New LWP 12865]
[New LWP 12866]
[New LWP 12867]
[New LWP 12868]
[New LWP 12869]
[New LWP 12870]
[New LWP 12871]
[New LWP 12872]
[New LWP 12873]
[New LWP 12874]
[New LWP 12875]
[New LWP 12876]
[New LWP 12877]
[New LWP 12878]
[New LWP 12879]
[New LWP 12880]
[New LWP 12881]
[New LWP 12882]
[New LWP 12883]
[New LWP 12884]
[New LWP 12885]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007fdb19b06d50 in ?? ()
Id Target Id Frame
* 1 LWP 12750 "kudu" 0x00007fdb19b06d50 in ?? ()
2 LWP 12754 "kudu" 0x00007fdb19b02fb9 in ?? ()
3 LWP 12755 "kudu" 0x00007fdb19b02fb9 in ?? ()
4 LWP 12756 "kudu" 0x00007fdb19b02fb9 in ?? ()
5 LWP 12757 "kernel-watcher-" 0x00007fdb19b02fb9 in ?? ()
6 LWP 12763 "ntp client-1276" 0x00007fdb19b069e2 in ?? ()
7 LWP 12765 "file cache-evic" 0x00007fdb19b02fb9 in ?? ()
8 LWP 12767 "sq_acceptor" 0x00007fdb1285fcb9 in ?? ()
9 LWP 12770 "rpc reactor-127" 0x00007fdb1286ca47 in ?? ()
10 LWP 12771 "rpc reactor-127" 0x00007fdb1286ca47 in ?? ()
11 LWP 12772 "rpc reactor-127" 0x00007fdb1286ca47 in ?? ()
12 LWP 12773 "rpc reactor-127" 0x00007fdb1286ca47 in ?? ()
13 LWP 12774 "MaintenanceMgr " 0x0000000000925f4e in __sanitizer::theDepot ()
14 LWP 12775 "txn-status-mana" 0x00007fdb19b02fb9 in ?? ()
15 LWP 12777 "collect_and_rem" 0x00007fdb19b02fb9 in ?? ()
16 LWP 12778 "tc-session-exp-" 0x00007fdb19b02fb9 in ?? ()
17 LWP 12779 "rpc worker-1277" 0x00007fdb19b02ad3 in ?? ()
18 LWP 12780 "rpc worker-1278" 0x00007fdb19b02ad3 in ?? ()
19 LWP 12781 "rpc worker-1278" 0x00007fdb19b02ad3 in ?? ()
20 LWP 12782 "rpc worker-1278" 0x00007fdb19b02ad3 in ?? ()
21 LWP 12783 "rpc worker-1278" 0x00007fdb19b02ad3 in ?? ()
22 LWP 12784 "rpc worker-1278" 0x00007fdb19b02ad3 in ?? ()
23 LWP 12785 "rpc worker-1278" 0x00007fdb19b02ad3 in ?? ()
24 LWP 12786 "rpc worker-1278" 0x00007fdb19b02ad3 in ?? ()
25 LWP 12787 "rpc worker-1278" 0x00007fdb19b02ad3 in ?? ()
26 LWP 12788 "rpc worker-1278" 0x00007fdb19b02ad3 in ?? ()
27 LWP 12789 "rpc worker-1278" 0x00007fdb19b02ad3 in ?? ()
28 LWP 12790 "rpc worker-1279" 0x00007fdb19b02ad3 in ?? ()
29 LWP 12791 "rpc worker-1279" 0x00007fdb19b02ad3 in ?? ()
30 LWP 12792 "rpc worker-1279" 0x00007fdb19b02ad3 in ?? ()
31 LWP 12793 "rpc worker-1279" 0x00007fdb19b02ad3 in ?? ()
32 LWP 12794 "rpc worker-1279" 0x00007fdb19b02ad3 in ?? ()
33 LWP 12795 "rpc worker-1279" 0x00007fdb19b02ad3 in ?? ()
34 LWP 12796 "rpc worker-1279" 0x00007fdb19b02ad3 in ?? ()
35 LWP 12797 "rpc worker-1279" 0x00007fdb19b02ad3 in ?? ()
36 LWP 12798 "rpc worker-1279" 0x00007fdb19b02ad3 in ?? ()
37 LWP 12799 "rpc worker-1279" 0x00007fdb19b02ad3 in ?? ()
38 LWP 12800 "rpc worker-1280" 0x00007fdb19b02ad3 in ?? ()
39 LWP 12801 "rpc worker-1280" 0x00007fdb19b02ad3 in ?? ()
40 LWP 12802 "rpc worker-1280" 0x00007fdb19b02ad3 in ?? ()
41 LWP 12803 "rpc worker-1280" 0x00007fdb19b02ad3 in ?? ()
42 LWP 12804 "rpc worker-1280" 0x00007fdb19b02ad3 in ?? ()
43 LWP 12805 "rpc worker-1280" 0x00007fdb19b02ad3 in ?? ()
44 LWP 12806 "rpc worker-1280" 0x00007fdb19b02ad3 in ?? ()
45 LWP 12807 "rpc worker-1280" 0x00007fdb19b02ad3 in ?? ()
46 LWP 12808 "rpc worker-1280" 0x00007fdb19b02ad3 in ?? ()
47 LWP 12809 "rpc worker-1280" 0x00007fdb19b02ad3 in ?? ()
48 LWP 12810 "rpc worker-1281" 0x00007fdb19b02ad3 in ?? ()
49 LWP 12811 "rpc worker-1281" 0x00007fdb19b02ad3 in ?? ()
50 LWP 12813 "rpc worker-1281" 0x00007fdb19b02ad3 in ?? ()
51 LWP 12814 "rpc worker-1281" 0x00007fdb19b02ad3 in ?? ()
52 LWP 12815 "rpc worker-1281" 0x00007fdb19b02ad3 in ?? ()
53 LWP 12816 "rpc worker-1281" 0x00007fdb19b02ad3 in ?? ()
54 LWP 12817 "rpc worker-1281" 0x00007fdb19b02ad3 in ?? ()
55 LWP 12818 "rpc worker-1281" 0x00007fdb19b02ad3 in ?? ()
56 LWP 12819 "rpc worker-1281" 0x00007fdb19b02ad3 in ?? ()
57 LWP 12820 "rpc worker-1282" 0x00007fdb19b02ad3 in ?? ()
58 LWP 12821 "rpc worker-1282" 0x00007fdb19b02ad3 in ?? ()
59 LWP 12822 "rpc worker-1282" 0x00007fdb19b02ad3 in ?? ()
60 LWP 12823 "rpc worker-1282" 0x00007fdb19b02ad3 in ?? ()
61 LWP 12824 "rpc worker-1282" 0x00007fdb19b02ad3 in ?? ()
62 LWP 12825 "rpc worker-1282" 0x00007fdb19b02ad3 in ?? ()
63 LWP 12826 "rpc worker-1282" 0x00007fdb19b02ad3 in ?? ()
64 LWP 12827 "rpc worker-1282" 0x00007fdb19b02ad3 in ?? ()
65 LWP 12828 "rpc worker-1282" 0x00007fdb19b02ad3 in ?? ()
66 LWP 12829 "rpc worker-1282" 0x00007fdb19b02ad3 in ?? ()
67 LWP 12830 "rpc worker-1283" 0x00007fdb19b02ad3 in ?? ()
68 LWP 12831 "rpc worker-1283" 0x00007fdb19b02ad3 in ?? ()
69 LWP 12832 "rpc worker-1283" 0x00007fdb19b02ad3 in ?? ()
70 LWP 12833 "rpc worker-1283" 0x00007fdb19b02ad3 in ?? ()
71 LWP 12834 "rpc worker-1283" 0x00007fdb19b02ad3 in ?? ()
72 LWP 12835 "rpc worker-1283" 0x00007fdb19b02ad3 in ?? ()
73 LWP 12836 "rpc worker-1283" 0x00007fdb19b02ad3 in ?? ()
74 LWP 12837 "rpc worker-1283" 0x00007fdb19b02ad3 in ?? ()
75 LWP 12838 "rpc worker-1283" 0x00007fdb19b02ad3 in ?? ()
76 LWP 12839 "rpc worker-1283" 0x00007fdb19b02ad3 in ?? ()
77 LWP 12840 "rpc worker-1284" 0x00007fdb19b02ad3 in ?? ()
78 LWP 12841 "rpc worker-1284" 0x00007fdb19b02ad3 in ?? ()
79 LWP 12842 "rpc worker-1284" 0x00007fdb19b02ad3 in ?? ()
80 LWP 12843 "rpc worker-1284" 0x00007fdb19b02ad3 in ?? ()
81 LWP 12844 "rpc worker-1284" 0x00007fdb19b02ad3 in ?? ()
82 LWP 12845 "rpc worker-1284" 0x00007fdb19b02ad3 in ?? ()
83 LWP 12846 "rpc worker-1284" 0x00007fdb19b02ad3 in ?? ()
84 LWP 12847 "rpc worker-1284" 0x00007fdb19b02ad3 in ?? ()
85 LWP 12848 "rpc worker-1284" 0x00007fdb19b02ad3 in ?? ()
86 LWP 12849 "rpc worker-1284" 0x00007fdb19b02ad3 in ?? ()
87 LWP 12850 "rpc worker-1285" 0x00007fdb19b02ad3 in ?? ()
88 LWP 12851 "rpc worker-1285" 0x00007fdb19b02ad3 in ?? ()
89 LWP 12852 "rpc worker-1285" 0x00007fdb19b02ad3 in ?? ()
90 LWP 12853 "rpc worker-1285" 0x00007fdb19b02ad3 in ?? ()
91 LWP 12854 "rpc worker-1285" 0x00007fdb19b02ad3 in ?? ()
92 LWP 12855 "rpc worker-1285" 0x00007fdb19b02ad3 in ?? ()
93 LWP 12856 "rpc worker-1285" 0x00007fdb19b02ad3 in ?? ()
94 LWP 12857 "rpc worker-1285" 0x00007fdb19b02ad3 in ?? ()
95 LWP 12858 "rpc worker-1285" 0x00007fdb19b02ad3 in ?? ()
96 LWP 12859 "rpc worker-1285" 0x00007fdb19b02ad3 in ?? ()
97 LWP 12860 "rpc worker-1286" 0x00007fdb19b02ad3 in ?? ()
98 LWP 12861 "rpc worker-1286" 0x00007fdb19b02ad3 in ?? ()
99 LWP 12862 "rpc worker-1286" 0x00007fdb19b02ad3 in ?? ()
100 LWP 12863 "rpc worker-1286" 0x00007fdb19b02ad3 in ?? ()
101 LWP 12864 "rpc worker-1286" 0x00007fdb19b02ad3 in ?? ()
102 LWP 12865 "rpc worker-1286" 0x00007fdb19b02ad3 in ?? ()
103 LWP 12866 "rpc worker-1286" 0x00007fdb19b02ad3 in ?? ()
104 LWP 12867 "rpc worker-1286" 0x00007fdb19b02ad3 in ?? ()
105 LWP 12868 "rpc worker-1286" 0x00007fdb19b02ad3 in ?? ()
106 LWP 12869 "rpc worker-1286" 0x00007fdb19b02ad3 in ?? ()
107 LWP 12870 "rpc worker-1287" 0x00007fdb19b02ad3 in ?? ()
108 LWP 12871 "rpc worker-1287" 0x00007fdb19b02ad3 in ?? ()
109 LWP 12872 "rpc worker-1287" 0x00007fdb19b02ad3 in ?? ()
110 LWP 12873 "rpc worker-1287" 0x00007fdb19b02ad3 in ?? ()
111 LWP 12874 "rpc worker-1287" 0x00007fdb19b02ad3 in ?? ()
112 LWP 12875 "rpc worker-1287" 0x00007fdb19b02ad3 in ?? ()
113 LWP 12876 "rpc worker-1287" 0x00007fdb19b02ad3 in ?? ()
114 LWP 12877 "rpc worker-1287" 0x00007fdb19b02ad3 in ?? ()
115 LWP 12878 "rpc worker-1287" 0x00007fdb19b02ad3 in ?? ()
116 LWP 12879 "rpc worker-1287" 0x00007fdb19b02ad3 in ?? ()
117 LWP 12880 "diag-logger-128" 0x00007fdb19b02fb9 in ?? ()
118 LWP 12881 "result-tracker-" 0x00007fdb19b02fb9 in ?? ()
119 LWP 12882 "excess-log-dele" 0x00007fdb19b02fb9 in ?? ()
120 LWP 12883 "acceptor-12883" 0x00007fdb1286e0c7 in ?? ()
121 LWP 12884 "heartbeat-12884" 0x00007fdb19b02fb9 in ?? ()
122 LWP 12885 "maintenance_sch" 0x00007fdb19b02fb9 in ?? ()
Thread 122 (LWP 12885):
#0 0x00007fdb19b02fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000027d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007fdaceb8e550 in ?? ()
#6 0x00000000000004fa in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 12884):
#0 0x00007fdb19b02fb9 in ?? ()
#1 0x99d1b6c4365c6052 in ?? ()
#2 0x00000000000000a3 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007fdacf3a65b0 in ?? ()
#6 0x0000000000000146 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 12883):
#0 0x00007fdb1286e0c7 in ?? ()
#1 0x00006140000218a8 in ?? ()
#2 0x00007fdacfbcdd50 in ?? ()
#3 0x00007fdacfbcdd80 in ?? ()
#4 0x00007fdacfbcde80 in ?? ()
#5 0x00007fdacfbcdd70 in ?? ()
#6 0x00007fdacfbcdde0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007fdb166f8620 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 12882):
#0 0x00007fdb19b02fb9 in ?? ()
#1 0x00007fdad03e6f00 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffcda2e1250 in ?? ()
#5 0x00007fdad03e6ef0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 12881):
#0 0x00007fdb19b02fb9 in ?? ()
#1 0x00007fdad0c000c0 in ?? ()
#2 0x000000000000009f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007fdad0c000b0 in ?? ()
#6 0x000000000000013e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 12880):
#0 0x00007fdb19b02fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 12879):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000028 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002d6828 in ?? ()
#4 0x00007fdad1c2fe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fdad1c2fe70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 12878):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000018 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002d0028 in ?? ()
#4 0x00007fdad2447e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fdad2447e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 12877):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 12876):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 12875):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 12874):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 12873):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 12872):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 12871):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 12870):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 12869):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 12868):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 12867):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 12866):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 12865):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 12864):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 12863):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 12862):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 12861):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 12860):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 12859):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 12858):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 12857):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 12856):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 12855):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 12854):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 12853):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 12852):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 12851):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 12850):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 12849):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 12848):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 12847):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 12846):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 12845):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 12844):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 12843):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 12842):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 12841):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 12840):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 12839):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x000000000000204c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001c6868 in ?? ()
#4 0x00007fdae6000e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fdae6000e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 12838):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000001957 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001c006c in ?? ()
#4 0x00007fdae6818e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fdae6818e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001c0020 in ?? ()
#9 0x00007fdb19b02770 in ?? ()
#10 0x00007fdae6818e70 in ?? ()
#11 0x00000ffb5cd031d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 74 (LWP 12837):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000f07 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001b687c in ?? ()
#4 0x00007fdae7030e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fdae7030e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001b6830 in ?? ()
#9 0x00007fdb19b02770 in ?? ()
#10 0x00007fdae7030e70 in ?? ()
#11 0x00000ffb5ce061d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 73 (LWP 12836):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 12835):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 12834):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 12833):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 12832):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 12831):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 12830):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 12829):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 12828):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 12827):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 12826):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 12825):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 12824):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 12823):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 12822):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 12821):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 12820):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 12819):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000140018 in ?? ()
#4 0x00007fdaf01e1e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fdaf01e1e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 12818):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 12817):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 12816):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 12815):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 12814):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 12813):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 12811):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 12810):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 12809):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 12808):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 12807):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 12806):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 12805):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 12804):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 12803):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 12802):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 12801):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 12800):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 12799):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 12798):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000002242 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000acfe8 in ?? ()
#4 0x00007fdaf9baae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fdaf9baae70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 12797):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000001f5f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007fdafa3c2e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fdafa3c2e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007fdb19b02770 in ?? ()
#10 0x00007fdafa3c2e70 in ?? ()
#11 0x00000ffb5f4785d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 34 (LWP 12796):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000001c37 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007fdafabdae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fdafabdae70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007fdb19b02770 in ?? ()
#10 0x00007fdafabdae70 in ?? ()
#11 0x00000ffb5f57b5d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 33 (LWP 12795):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x00000000000021f8 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000967f8 in ?? ()
#4 0x00007fdafb3f2e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fdafb3f2e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 32 (LWP 12794):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x000000000000149b in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00008fffc in ?? ()
#4 0x00007fdafbc0ae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fdafbc0ae70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00008ffb0 in ?? ()
#9 0x00007fdb19b02770 in ?? ()
#10 0x00007fdafbc0ae70 in ?? ()
#11 0x00000ffb5f7815d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 31 (LWP 12793):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000001868 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000b67d8 in ?? ()
#4 0x00007fdafc422e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fdafc422e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 30 (LWP 12792):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000003 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00008d00c in ?? ()
#4 0x00007fdafcc3ae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fdafcc3ae70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00008cfc0 in ?? ()
#9 0x00007fdb19b02770 in ?? ()
#10 0x00007fdafcc3ae70 in ?? ()
#11 0x00000ffb5f9875d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 29 (LWP 12791):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 12790):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 12789):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 12788):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 12787):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 12786):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 12785):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 12784):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 12783):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 12782):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 12781):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 12780):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 12779):
#0 0x00007fdb19b02ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 12778):
#0 0x00007fdb19b02fb9 in ?? ()
#1 0x00007fdb03cc2c80 in ?? ()
#2 0x0000000000000010 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ba60 in ?? ()
#5 0x00007fdb03cc2c70 in ?? ()
#6 0x0000000000000020 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 12777):
#0 0x00007fdb19b02fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 12775):
#0 0x00007fdb19b02fb9 in ?? ()
#1 0x00007fdb054c51c0 in ?? ()
#2 0x0000000000000010 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000f9a8 in ?? ()
#5 0x00007fdb054c51b0 in ?? ()
#6 0x0000000000000020 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 12774):
#0 0x0000000000925f4e in __sanitizer::theDepot ()
#1 0x00007fdb2340bb2f in ?? ()
#2 0x000062d00f4491b0 in ?? ()
#3 0x000062d00f44921c in ?? ()
#4 0x0000000000905018 in __sanitizer::theDepot ()
#5 0x000000000000006c in ?? ()
#6 0x00007fdb05cc1e50 in ?? ()
#7 0x000062d00f4491b0 in ?? ()
#8 0x0000000000900f43 in __sanitizer::theDepot ()
#9 0x00007fdb233eddce in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 12 (LWP 12773):
#0 0x00007fdb1286ca47 in ?? ()
#1 0x00007fdb064c82e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007fdb064c82d0 in ?? ()
#4 0x00007fdb064c84e0 in ?? ()
#5 0x00007fdb064c8320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007fdb064c83a0 in ?? ()
#8 0x00007fdb14ecc25d in ?? ()
#9 0x3fb98bdeecc36000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x00000000d8000000 in ?? ()
#14 0x00007fdb00000000 in ?? ()
#15 0x41d9e134cd666726 in ?? ()
#16 0x00000ffbe0c91074 in ?? ()
#17 0x00007fdb064c8380 in ?? ()
#18 0x00007fdb14ed0ba3 in ?? ()
#19 0x00007fdb064c8350 in ?? ()
#20 0x3fb98bdeecc36000 in ?? ()
#21 0x00000000064c83a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb98bdeecc36000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 12772):
#0 0x00007fdb1286ca47 in ?? ()
#1 0x00007fdb06cc92e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007fdb06cc92d0 in ?? ()
#4 0x00007fdb06cc94e0 in ?? ()
#5 0x00007fdb06cc9320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007fdb06cc93a0 in ?? ()
#8 0x00007fdb14ecc25d in ?? ()
#9 0x3fb98b8428fd6000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x000000000e000000 in ?? ()
#14 0x00007fdb00000000 in ?? ()
#15 0x41d9e134cd666726 in ?? ()
#16 0x00000ffbe0d91274 in ?? ()
#17 0x00007fdb06cc9380 in ?? ()
#18 0x00007fdb14ed0ba3 in ?? ()
#19 0x00007fdb06cc9350 in ?? ()
#20 0x3fb98b8428fd6000 in ?? ()
#21 0x0000000006cc93a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb98b8428fd6000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 12771):
#0 0x00007fdb1286ca47 in ?? ()
#1 0x00007fdb074ca2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007fdb074ca2d0 in ?? ()
#4 0x00007fdb074ca4e0 in ?? ()
#5 0x00007fdb074ca320 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007fdb074ca3a0 in ?? ()
#8 0x00007fdb14ecc25d in ?? ()
#9 0x3fb1d55c798ee000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x00000000f1c00000 in ?? ()
#14 0x00007fdb00000000 in ?? ()
#15 0x41d9e134cd666726 in ?? ()
#16 0x00000ffbe0e91474 in ?? ()
#17 0x00007fdb074ca380 in ?? ()
#18 0x00007fdb14ed0ba3 in ?? ()
#19 0x00007fdb074ca350 in ?? ()
#20 0x3fb1d55c798ee000 in ?? ()
#21 0x00000000074ca3a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb1d55c798ee000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 12770):
#0 0x00007fdb1286ca47 in ?? ()
#1 0x00007fdb090b32e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007fdb090b32d0 in ?? ()
#4 0x00007fdb090b34e0 in ?? ()
#5 0x00007fdb090b3320 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007fdb090b33a0 in ?? ()
#8 0x00007fdb14ecc25d in ?? ()
#9 0x3fb9888b58d02000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x0000000081000000 in ?? ()
#14 0x00007fdb00000000 in ?? ()
#15 0x41d9e134cd666725 in ?? ()
#16 0x00000ffbe120e674 in ?? ()
#17 0x00007fdb090b3380 in ?? ()
#18 0x00007fdb14ed0ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 12767):
#0 0x00007fdb1285fcb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fdb0b0b7788 in ?? ()
#3 0x0000602000017cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 12765):
#0 0x00007fdb19b02fb9 in ?? ()
#1 0x00007fdb0a8b6080 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007fdb0a8b6070 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 12763):
#0 0x00007fdb19b069e2 in ?? ()
#1 0x00007fdb098b2b60 in ?? ()
#2 0x00007fdb098b2c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fdb098b2bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007fdb098b2c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007fdb098b2c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007fdb1f42af0c in ?? ()
#12 0x00007fdb1f41a7b9 in ?? ()
#13 0x00007fdb1f41eda6 in ?? ()
#14 0x00007fdb1f4237e0 in ?? ()
#15 0x00007fdb1f423609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007fdb167190be in ?? ()
#18 0x00007fdb19afc6db in ?? ()
#19 0x00000ffbe130e47c in ?? ()
#20 0x00007fdb098b2400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 12757):
#0 0x00007fdb19b02fb9 in ?? ()
#1 0x00007fdb0c0b8c40 in ?? ()
#2 0x0000000000000322 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007fdb0c0b8c30 in ?? ()
#6 0x0000000000000644 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 12756):
#0 0x00007fdb19b02fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 12755):
#0 0x00007fdb19b02fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 12754):
#0 0x00007fdb19b02fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fdb0d8bc410 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 12750):
#0 0x00007fdb19b06d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250113 08:57:32.652408 11762 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID cbb5dc448319472395e744e7bd16c57f and pid 12490
************************ BEGIN STACKS **************************
[New LWP 12494]
[New LWP 12495]
[New LWP 12496]
[New LWP 12497]
[New LWP 12503]
[New LWP 12504]
[New LWP 12505]
[New LWP 12508]
[New LWP 12509]
[New LWP 12510]
[New LWP 12511]
[New LWP 12512]
[New LWP 12513]
[New LWP 12514]
[New LWP 12515]
[New LWP 12516]
[New LWP 12517]
[New LWP 12518]
[New LWP 12519]
[New LWP 12520]
[New LWP 12521]
[New LWP 12522]
[New LWP 12523]
[New LWP 12524]
[New LWP 12525]
[New LWP 12526]
[New LWP 12527]
[New LWP 12528]
[New LWP 12529]
[New LWP 12530]
[New LWP 12531]
[New LWP 12532]
[New LWP 12533]
[New LWP 12534]
[New LWP 12535]
[New LWP 12537]
[New LWP 12538]
[New LWP 12539]
[New LWP 12540]
[New LWP 12542]
[New LWP 12543]
[New LWP 12544]
[New LWP 12545]
[New LWP 12546]
[New LWP 12547]
[New LWP 12548]
[New LWP 12549]
[New LWP 12550]
[New LWP 12551]
[New LWP 12552]
[New LWP 12553]
[New LWP 12554]
[New LWP 12555]
[New LWP 12556]
[New LWP 12557]
[New LWP 12558]
[New LWP 12559]
[New LWP 12560]
[New LWP 12561]
[New LWP 12562]
[New LWP 12563]
[New LWP 12564]
[New LWP 12565]
[New LWP 12566]
[New LWP 12567]
[New LWP 12568]
[New LWP 12569]
[New LWP 12570]
[New LWP 12571]
[New LWP 12572]
[New LWP 12573]
[New LWP 12574]
[New LWP 12575]
[New LWP 12576]
[New LWP 12577]
[New LWP 12578]
[New LWP 12579]
[New LWP 12580]
[New LWP 12581]
[New LWP 12582]
[New LWP 12583]
[New LWP 12584]
[New LWP 12585]
[New LWP 12586]
[New LWP 12587]
[New LWP 12589]
[New LWP 12591]
[New LWP 12592]
[New LWP 12593]
[New LWP 12594]
[New LWP 12595]
[New LWP 12596]
[New LWP 12597]
[New LWP 12598]
[New LWP 12599]
[New LWP 12600]
[New LWP 12601]
[New LWP 12602]
[New LWP 12603]
[New LWP 12604]
[New LWP 12605]
[New LWP 12606]
[New LWP 12607]
[New LWP 12608]
[New LWP 12609]
[New LWP 12610]
[New LWP 12611]
[New LWP 12612]
[New LWP 12613]
[New LWP 12614]
[New LWP 12615]
[New LWP 12616]
[New LWP 12617]
[New LWP 12618]
[New LWP 12619]
[New LWP 12620]
[New LWP 12621]
[New LWP 12622]
[New LWP 12623]
[New LWP 12624]
[New LWP 12625]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007f624e0e4d50 in ?? ()
Id Target Id Frame
* 1 LWP 12490 "kudu" 0x00007f624e0e4d50 in ?? ()
2 LWP 12494 "kudu" 0x00007f624e0e0fb9 in ?? ()
3 LWP 12495 "kudu" 0x00007f624e0e0fb9 in ?? ()
4 LWP 12496 "kudu" 0x00007f624e0e0fb9 in ?? ()
5 LWP 12497 "kernel-watcher-" 0x00007f624e0e0fb9 in ?? ()
6 LWP 12503 "ntp client-1250" 0x00007f624e0e49e2 in ?? ()
7 LWP 12504 "file cache-evic" 0x00007f624e0e0fb9 in ?? ()
8 LWP 12505 "sq_acceptor" 0x00007f6246e3dcb9 in ?? ()
9 LWP 12508 "rpc reactor-125" 0x00007f6246e4aa47 in ?? ()
10 LWP 12509 "rpc reactor-125" 0x00007f6246e4aa47 in ?? ()
11 LWP 12510 "rpc reactor-125" 0x00007f6246e4aa47 in ?? ()
12 LWP 12511 "rpc reactor-125" 0x00007f6246e4aa47 in ?? ()
13 LWP 12512 "MaintenanceMgr " 0x00007f624e0e0ad3 in ?? ()
14 LWP 12513 "txn-status-mana" 0x00007f624e0e0fb9 in ?? ()
15 LWP 12514 "collect_and_rem" 0x00007f624e0e0fb9 in ?? ()
16 LWP 12515 "tc-session-exp-" 0x00007f624e0e0fb9 in ?? ()
17 LWP 12516 "rpc worker-1251" 0x00007f624e0e0ad3 in ?? ()
18 LWP 12517 "rpc worker-1251" 0x00007f624e0e0ad3 in ?? ()
19 LWP 12518 "rpc worker-1251" 0x00007f624e0e0ad3 in ?? ()
20 LWP 12519 "rpc worker-1251" 0x00007f624e0e0ad3 in ?? ()
21 LWP 12520 "rpc worker-1252" 0x00007f624e0e0ad3 in ?? ()
22 LWP 12521 "rpc worker-1252" 0x00007f624e0e0ad3 in ?? ()
23 LWP 12522 "rpc worker-1252" 0x00007f624e0e0ad3 in ?? ()
24 LWP 12523 "rpc worker-1252" 0x00007f624e0e0ad3 in ?? ()
25 LWP 12524 "rpc worker-1252" 0x00007f624e0e0ad3 in ?? ()
26 LWP 12525 "rpc worker-1252" 0x00007f624e0e0ad3 in ?? ()
27 LWP 12526 "rpc worker-1252" 0x00007f624e0e0ad3 in ?? ()
28 LWP 12527 "rpc worker-1252" 0x00007f624e0e0ad3 in ?? ()
29 LWP 12528 "rpc worker-1252" 0x00007f624e0e0ad3 in ?? ()
30 LWP 12529 "rpc worker-1252" 0x00007f624e0e0ad3 in ?? ()
31 LWP 12530 "rpc worker-1253" 0x00007f624e0e0ad3 in ?? ()
32 LWP 12531 "rpc worker-1253" 0x00007f624e0e0ad3 in ?? ()
33 LWP 12532 "rpc worker-1253" 0x00007f624e0e0ad3 in ?? ()
34 LWP 12533 "rpc worker-1253" 0x00007f624e0e0ad3 in ?? ()
35 LWP 12534 "rpc worker-1253" 0x00007f624e0e0ad3 in ?? ()
36 LWP 12535 "rpc worker-1253" 0x00007f624e0e0ad3 in ?? ()
37 LWP 12537 "rpc worker-1253" 0x00007f624e0e0ad3 in ?? ()
38 LWP 12538 "rpc worker-1253" 0x00007f624e0e0ad3 in ?? ()
39 LWP 12539 "rpc worker-1253" 0x00007f624e0e0ad3 in ?? ()
40 LWP 12540 "rpc worker-1254" 0x00007f624e0e0ad3 in ?? ()
41 LWP 12542 "rpc worker-1254" 0x00007f624e0e0ad3 in ?? ()
42 LWP 12543 "rpc worker-1254" 0x00007f624e0e0ad3 in ?? ()
43 LWP 12544 "rpc worker-1254" 0x00007f624e0e0ad3 in ?? ()
44 LWP 12545 "rpc worker-1254" 0x00007f624e0e0ad3 in ?? ()
45 LWP 12546 "rpc worker-1254" 0x00007f624e0e0ad3 in ?? ()
46 LWP 12547 "rpc worker-1254" 0x00007f624e0e0ad3 in ?? ()
47 LWP 12548 "rpc worker-1254" 0x00007f624e0e0ad3 in ?? ()
48 LWP 12549 "rpc worker-1254" 0x00007f624e0e0ad3 in ?? ()
49 LWP 12550 "rpc worker-1255" 0x00007f624e0e0ad3 in ?? ()
50 LWP 12551 "rpc worker-1255" 0x00007f624e0e0ad3 in ?? ()
51 LWP 12552 "rpc worker-1255" 0x00007f624e0e0ad3 in ?? ()
52 LWP 12553 "rpc worker-1255" 0x00007f624e0e0ad3 in ?? ()
53 LWP 12554 "rpc worker-1255" 0x00007f624e0e0ad3 in ?? ()
54 LWP 12555 "rpc worker-1255" 0x00007f624e0e0ad3 in ?? ()
55 LWP 12556 "rpc worker-1255" 0x00007f624e0e0ad3 in ?? ()
56 LWP 12557 "rpc worker-1255" 0x00007f624e0e0ad3 in ?? ()
57 LWP 12558 "rpc worker-1255" 0x00007f624e0e0ad3 in ?? ()
58 LWP 12559 "rpc worker-1255" 0x00007f624e0e0ad3 in ?? ()
59 LWP 12560 "rpc worker-1256" 0x00007f624e0e0ad3 in ?? ()
60 LWP 12561 "rpc worker-1256" 0x00007f624e0e0ad3 in ?? ()
61 LWP 12562 "rpc worker-1256" 0x00007f624e0e0ad3 in ?? ()
62 LWP 12563 "rpc worker-1256" 0x00007f624e0e0ad3 in ?? ()
63 LWP 12564 "rpc worker-1256" 0x00007f624e0e0ad3 in ?? ()
64 LWP 12565 "rpc worker-1256" 0x00007f624e0e0ad3 in ?? ()
65 LWP 12566 "rpc worker-1256" 0x00007f624e0e0ad3 in ?? ()
66 LWP 12567 "rpc worker-1256" 0x00007f624e0e0ad3 in ?? ()
67 LWP 12568 "rpc worker-1256" 0x00007f624e0e0ad3 in ?? ()
68 LWP 12569 "rpc worker-1256" 0x00007f624e0e0ad3 in ?? ()
69 LWP 12570 "rpc worker-1257" 0x00007f624e0e0ad3 in ?? ()
70 LWP 12571 "rpc worker-1257" 0x00007f624e0e0ad3 in ?? ()
71 LWP 12572 "rpc worker-1257" 0x00007f624e0e0ad3 in ?? ()
72 LWP 12573 "rpc worker-1257" 0x00007f624e0e0ad3 in ?? ()
73 LWP 12574 "rpc worker-1257" 0x00007f624e0e0ad3 in ?? ()
74 LWP 12575 "rpc worker-1257" 0x00007f624e0e0ad3 in ?? ()
75 LWP 12576 "rpc worker-1257" 0x00007f624e0e0ad3 in ?? ()
76 LWP 12577 "rpc worker-1257" 0x00007f624e0e0ad3 in ?? ()
77 LWP 12578 "rpc worker-1257" 0x00007f624e0e0ad3 in ?? ()
78 LWP 12579 "rpc worker-1257" 0x00007f624e0e0ad3 in ?? ()
79 LWP 12580 "rpc worker-1258" 0x00007f624e0e0ad3 in ?? ()
80 LWP 12581 "rpc worker-1258" 0x00007f624e0e0ad3 in ?? ()
81 LWP 12582 "rpc worker-1258" 0x00007f624e0e0ad3 in ?? ()
82 LWP 12583 "rpc worker-1258" 0x00007f624e0e0ad3 in ?? ()
83 LWP 12584 "rpc worker-1258" 0x00007f624e0e0ad3 in ?? ()
84 LWP 12585 "rpc worker-1258" 0x00007f624e0e0ad3 in ?? ()
85 LWP 12586 "rpc worker-1258" 0x00007f624e0e0ad3 in ?? ()
86 LWP 12587 "rpc worker-1258" 0x00007f624e0e0ad3 in ?? ()
87 LWP 12589 "rpc worker-1258" 0x00007f624e0e0ad3 in ?? ()
88 LWP 12591 "rpc worker-1259" 0x00007f624e0e0ad3 in ?? ()
89 LWP 12592 "rpc worker-1259" 0x00007f624e0e0ad3 in ?? ()
90 LWP 12593 "rpc worker-1259" 0x00007f624e0e0ad3 in ?? ()
91 LWP 12594 "rpc worker-1259" 0x00007f624e0e0ad3 in ?? ()
92 LWP 12595 "rpc worker-1259" 0x00007f624e0e0ad3 in ?? ()
93 LWP 12596 "rpc worker-1259" 0x00007f624e0e0ad3 in ?? ()
94 LWP 12597 "rpc worker-1259" 0x00007f624e0e0ad3 in ?? ()
95 LWP 12598 "rpc worker-1259" 0x00007f624e0e0ad3 in ?? ()
96 LWP 12599 "rpc worker-1259" 0x00007f624e0e0ad3 in ?? ()
97 LWP 12600 "rpc worker-1260" 0x00007f624e0e0ad3 in ?? ()
98 LWP 12601 "rpc worker-1260" 0x00007f624e0e0ad3 in ?? ()
99 LWP 12602 "rpc worker-1260" 0x00007f624e0e0ad3 in ?? ()
100 LWP 12603 "rpc worker-1260" 0x00007f624e0e0ad3 in ?? ()
101 LWP 12604 "rpc worker-1260" 0x00007f624e0e0ad3 in ?? ()
102 LWP 12605 "rpc worker-1260" 0x00007f624e0e0ad3 in ?? ()
103 LWP 12606 "rpc worker-1260" 0x00007f624e0e0ad3 in ?? ()
104 LWP 12607 "rpc worker-1260" 0x00007f624e0e0ad3 in ?? ()
105 LWP 12608 "rpc worker-1260" 0x00007f624e0e0ad3 in ?? ()
106 LWP 12609 "rpc worker-1260" 0x00007f624e0e0ad3 in ?? ()
107 LWP 12610 "rpc worker-1261" 0x00007f624e0e0ad3 in ?? ()
108 LWP 12611 "rpc worker-1261" 0x00007f624e0e0ad3 in ?? ()
109 LWP 12612 "rpc worker-1261" 0x00007f624e0e0ad3 in ?? ()
110 LWP 12613 "rpc worker-1261" 0x00007f624e0e0ad3 in ?? ()
111 LWP 12614 "rpc worker-1261" 0x00007f624e0e0ad3 in ?? ()
112 LWP 12615 "rpc worker-1261" 0x00007f624e0e0ad3 in ?? ()
113 LWP 12616 "rpc worker-1261" 0x00007f624e0e0ad3 in ?? ()
114 LWP 12617 "rpc worker-1261" 0x00007f624e0e0ad3 in ?? ()
115 LWP 12618 "rpc worker-1261" 0x00007f624e0e0ad3 in ?? ()
116 LWP 12619 "rpc worker-1261" 0x00007f624e0e0ad3 in ?? ()
117 LWP 12620 "diag-logger-126" 0x00007f624e0e0fb9 in ?? ()
118 LWP 12621 "result-tracker-" 0x00007f624e0e0fb9 in ?? ()
119 LWP 12622 "excess-log-dele" 0x00007f624e0e0fb9 in ?? ()
120 LWP 12623 "acceptor-12623" 0x00007f6246e4c0c7 in ?? ()
121 LWP 12624 "heartbeat-12624" 0x00007f624e0e0fb9 in ?? ()
122 LWP 12625 "maintenance_sch" 0x00007f624e0e0fb9 in ?? ()
Thread 122 (LWP 12625):
#0 0x00007f624e0e0fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000002b1 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007f62039c4550 in ?? ()
#6 0x0000000000000562 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 12624):
#0 0x00007f624e0e0fb9 in ?? ()
#1 0x99d1b6c4365c6052 in ?? ()
#2 0x00000000000000ac in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007f62041dc5b0 in ?? ()
#6 0x0000000000000158 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 12623):
#0 0x00007f6246e4c0c7 in ?? ()
#1 0x00006140000216a8 in ?? ()
#2 0x00007f6204a03d50 in ?? ()
#3 0x00007f6204a03d80 in ?? ()
#4 0x00007f6204a03e80 in ?? ()
#5 0x00007f6204a03d70 in ?? ()
#6 0x00007f6204a03de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007f624acd6620 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 12622):
#0 0x00007f624e0e0fb9 in ?? ()
#1 0x00007f620521df00 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffebaf82ff0 in ?? ()
#5 0x00007f620521def0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 12621):
#0 0x00007f624e0e0fb9 in ?? ()
#1 0x00007f6205a360c0 in ?? ()
#2 0x00000000000000ac in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007f6205a360b0 in ?? ()
#6 0x0000000000000158 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 12620):
#0 0x00007f624e0e0fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 12619):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000024 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002b0048 in ?? ()
#4 0x00007f6206a65e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f6206a65e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 12618):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000020 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002a6858 in ?? ()
#4 0x00007f620727de50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f620727de70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 12617):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 12616):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 12615):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 12614):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 12613):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 12612):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 12611):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 12610):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 12609):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 12608):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 12607):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 12606):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 12605):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 12604):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 12603):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 12602):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 12601):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 12600):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 12599):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 12598):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 12597):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 12596):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 12595):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 12594):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 12593):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 12592):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 12591):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 12589):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 12587):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 12586):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 12585):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 12584):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 12583):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 12582):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 12581):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 12580):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 12579):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 12578):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 12577):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007f621ae36e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f621ae36e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007f624e0e0770 in ?? ()
#10 0x00007f621ae36e70 in ?? ()
#11 0x00000fec435c6dd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 12576):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 12575):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 12574):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 12573):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 12572):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 12571):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 12570):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 12569):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 12568):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 12567):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 12566):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 12565):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 12564):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 12563):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 12562):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 12561):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 12560):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 12559):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 12558):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 12557):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007f6225017e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f6225017e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007f624e0e0770 in ?? ()
#10 0x00007f6225017e70 in ?? ()
#11 0x00000fec44a02fd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 12556):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 12555):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 12554):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 12553):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 12552):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 12551):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 12550):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 12549):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 12548):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 12547):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 12546):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 12545):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 12544):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 12543):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 12542):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 12540):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 12539):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 12538):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 12537):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 12535):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000011 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007f622e9e0e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f622e9e0e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007f624e0e0770 in ?? ()
#10 0x00007f622e9e0e70 in ?? ()
#11 0x00000fec45d3c1d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 12534):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 12533):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 12532):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 12531):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 12530):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 12529):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 12528):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 12527):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 12526):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 12525):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 12524):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 12523):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 12522):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 12521):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 12520):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 12519):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 12518):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 12517):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 12516):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 12515):
#0 0x00007f624e0e0fb9 in ?? ()
#1 0x00007f6238ac3c80 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001b360 in ?? ()
#5 0x00007f6238ac3c70 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 12514):
#0 0x00007f624e0e0fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 12513):
#0 0x00007f624e0e0fb9 in ?? ()
#1 0x00007f6239ac51c0 in ?? ()
#2 0x0000000000000011 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000f9a8 in ?? ()
#5 0x00007f6239ac51b0 in ?? ()
#6 0x0000000000000022 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 12512):
#0 0x00007f624e0e0ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 12511):
#0 0x00007f6246e4aa47 in ?? ()
#1 0x00007f623aac82e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007f623aac82d0 in ?? ()
#4 0x00007f623aac84e0 in ?? ()
#5 0x00007f623aac8320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007f623aac83a0 in ?? ()
#8 0x00007f62494aa25d in ?? ()
#9 0x3fb963d9e2680000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000020000000 in ?? ()
#14 0x00007f6200000000 in ?? ()
#15 0x41d9e134cd666726 in ?? ()
#16 0x00000fecc7551074 in ?? ()
#17 0x00007f623aac8380 in ?? ()
#18 0x00007f62494aeba3 in ?? ()
#19 0x00007f623aac8350 in ?? ()
#20 0x3fb963d9e2680000 in ?? ()
#21 0x000000003aac83a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb963d9e2680000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 12510):
#0 0x00007f6246e4aa47 in ?? ()
#1 0x00007f623b2c92e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007f623b2c92d0 in ?? ()
#4 0x00007f623b2c94e0 in ?? ()
#5 0x00007f623b2c9320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f623b2c93a0 in ?? ()
#8 0x00007f62494aa25d in ?? ()
#9 0x3fb986fa78e08000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x0000000028000000 in ?? ()
#14 0x00007f6200000000 in ?? ()
#15 0x41d9e134cd666727 in ?? ()
#16 0x00000fecc7651274 in ?? ()
#17 0x00007f623b2c9380 in ?? ()
#18 0x00007f62494aeba3 in ?? ()
#19 0x00007f623b2c9350 in ?? ()
#20 0x3fb986fa78e08000 in ?? ()
#21 0x000000003b2c93a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb986fa78e08000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 12509):
#0 0x00007f6246e4aa47 in ?? ()
#1 0x00007f623baca2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007f623baca2d0 in ?? ()
#4 0x00007f623baca4e0 in ?? ()
#5 0x00007f623baca320 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f623baca3a0 in ?? ()
#8 0x00007f62494aa25d in ?? ()
#9 0x3fb961e338e42000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x00000000fb000000 in ?? ()
#14 0x00007f6200000000 in ?? ()
#15 0x41d9e134cd666725 in ?? ()
#16 0x00000fecc7751474 in ?? ()
#17 0x00007f623baca380 in ?? ()
#18 0x00007f62494aeba3 in ?? ()
#19 0x00007f623baca350 in ?? ()
#20 0x3fb961e338e42000 in ?? ()
#21 0x000000003baca3a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb961e338e42000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 12508):
#0 0x00007f6246e4aa47 in ?? ()
#1 0x00007f623d6b32e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007f623d6b32d0 in ?? ()
#4 0x00007f623d6b34e0 in ?? ()
#5 0x00007f623d6b3320 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f623d6b33a0 in ?? ()
#8 0x00007f62494aa25d in ?? ()
#9 0x3fb98d04d4034000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x00000000f0800000 in ?? ()
#14 0x00007f6200000000 in ?? ()
#15 0x41d9e134cd666728 in ?? ()
#16 0x00000fecc7ace674 in ?? ()
#17 0x00007f623d6b3380 in ?? ()
#18 0x00007f62494aeba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 12505):
#0 0x00007f6246e3dcb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f623f6b7788 in ?? ()
#3 0x0000602000017cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 12504):
#0 0x00007f624e0e0fb9 in ?? ()
#1 0x00007f623eeb6080 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007f623eeb6070 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 12503):
#0 0x00007f624e0e49e2 in ?? ()
#1 0x00007f623deb2b60 in ?? ()
#2 0x00007f623deb2c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f623deb2bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007f623deb2c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007f623deb2c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007f6253a08f0c in ?? ()
#12 0x00007f62539f87b9 in ?? ()
#13 0x00007f62539fcda6 in ?? ()
#14 0x00007f6253a017e0 in ?? ()
#15 0x00007f6253a01609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007f624acf70be in ?? ()
#18 0x00007f624e0da6db in ?? ()
#19 0x00000fecc7bce47c in ?? ()
#20 0x00007f623deb2400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 12497):
#0 0x00007f624e0e0fb9 in ?? ()
#1 0x00007f62406b8c40 in ?? ()
#2 0x0000000000000361 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007f62406b8c30 in ?? ()
#6 0x00000000000006c2 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 12496):
#0 0x00007f624e0e0fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 12495):
#0 0x00007f624e0e0fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 12494):
#0 0x00007f624e0e0fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f6241ebc410 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 12490):
#0 0x00007f624e0e4d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250113 08:57:33.859504 11762 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID aa0b5c78b11d4d07bb963ae1d9a42b3d and pid 13768
************************ BEGIN STACKS **************************
I20250113 08:57:34.790851 12774 maintenance_manager.cc:642] P 4a1af392ea8543f4827f17fdff2eaaac: FlushMRSOp(3034bef68d124e778f0b57a39a9b152c) complete. Timing: real 32.293s user 21.622s sys 0.321s Metrics: {"bytes_written":19462425,"cfile_init":1,"dirs.queue_time_us":6364,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":6150,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":108601,"lbm_writes_1-10_ms":10,"lbm_writes_lt_1ms":838,"peak_mem_usage":0,"rows_written":701550,"thread_start_us":13034,"threads_started":2,"wal-append.queue_time_us":6348}
I20250113 08:57:34.792732 12885 maintenance_manager.cc:418] P 4a1af392ea8543f4827f17fdff2eaaac: Scheduling LogGCOp(3034bef68d124e778f0b57a39a9b152c): free 16790082 bytes of WAL
I20250113 08:57:34.793056 12774 log_reader.cc:385] T 3034bef68d124e778f0b57a39a9b152c: removed 2 log segments from log reader
I20250113 08:57:34.793154 12774 log.cc:1078] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Deleting log segment in path: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/wal/wals/3034bef68d124e778f0b57a39a9b152c/wal-000000001 (ops 1-4892)
I20250113 08:57:34.793231 12774 log.cc:1078] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac: Deleting log segment in path: /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0/minicluster-data/ts-0/wal/wals/3034bef68d124e778f0b57a39a9b152c/wal-000000002 (ops 4893-9635)
I20250113 08:57:34.798506 12774 maintenance_manager.cc:642] P 4a1af392ea8543f4827f17fdff2eaaac: LogGCOp(3034bef68d124e778f0b57a39a9b152c) complete. Timing: real 0.006s user 0.000s sys 0.004s Metrics: {}
I20250113 08:57:34.799077 12885 maintenance_manager.cc:418] P 4a1af392ea8543f4827f17fdff2eaaac: Scheduling UndoDeltaBlockGCOp(3034bef68d124e778f0b57a39a9b152c): 8104844 bytes on disk
I20250113 08:57:34.800981 12774 maintenance_manager.cc:642] P 4a1af392ea8543f4827f17fdff2eaaac: UndoDeltaBlockGCOp(3034bef68d124e778f0b57a39a9b152c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
[New LWP 13770]
[New LWP 13771]
[New LWP 13772]
[New LWP 13773]
[New LWP 13780]
[New LWP 13781]
[New LWP 13782]
[New LWP 13785]
[New LWP 13786]
[New LWP 13787]
[New LWP 13788]
[New LWP 13789]
[New LWP 13790]
[New LWP 13792]
[New LWP 13793]
[New LWP 13794]
[New LWP 13795]
[New LWP 13796]
[New LWP 13797]
[New LWP 13798]
[New LWP 13799]
[New LWP 13800]
[New LWP 13801]
[New LWP 13802]
[New LWP 13803]
[New LWP 13804]
[New LWP 13805]
[New LWP 13806]
[New LWP 13807]
[New LWP 13808]
[New LWP 13809]
[New LWP 13810]
[New LWP 13811]
[New LWP 13812]
[New LWP 13813]
[New LWP 13814]
[New LWP 13815]
[New LWP 13816]
[New LWP 13817]
[New LWP 13818]
[New LWP 13819]
[New LWP 13820]
[New LWP 13822]
[New LWP 13823]
[New LWP 13824]
[New LWP 13825]
[New LWP 13826]
[New LWP 13827]
[New LWP 13828]
[New LWP 13829]
[New LWP 13830]
[New LWP 13831]
[New LWP 13832]
[New LWP 13833]
[New LWP 13834]
[New LWP 13835]
[New LWP 13836]
[New LWP 13837]
[New LWP 13838]
[New LWP 13839]
[New LWP 13840]
[New LWP 13841]
[New LWP 13842]
[New LWP 13843]
[New LWP 13844]
[New LWP 13845]
[New LWP 13846]
[New LWP 13847]
[New LWP 13848]
[New LWP 13849]
[New LWP 13850]
[New LWP 13851]
[New LWP 13852]
[New LWP 13853]
[New LWP 13854]
[New LWP 13855]
[New LWP 13856]
[New LWP 13857]
[New LWP 13858]
[New LWP 13859]
[New LWP 13860]
[New LWP 13861]
[New LWP 13862]
[New LWP 13863]
[New LWP 13864]
[New LWP 13865]
[New LWP 13866]
[New LWP 13867]
[New LWP 13868]
[New LWP 13869]
[New LWP 13870]
[New LWP 13871]
[New LWP 13872]
[New LWP 13873]
[New LWP 13874]
[New LWP 13875]
[New LWP 13876]
[New LWP 13877]
[New LWP 13878]
[New LWP 13879]
[New LWP 13881]
[New LWP 13882]
[New LWP 13883]
[New LWP 13884]
[New LWP 13885]
[New LWP 13886]
[New LWP 13887]
[New LWP 13888]
[New LWP 13889]
[New LWP 13890]
[New LWP 13891]
[New LWP 13892]
[New LWP 13893]
[New LWP 13894]
[New LWP 13895]
[New LWP 13896]
[New LWP 13897]
[New LWP 13898]
[New LWP 13899]
[New LWP 13900]
[New LWP 13901]
[New LWP 14320]
[New LWP 14321]
[New LWP 14322]
[New LWP 14323]
[New LWP 14324]
[New LWP 14325]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007f6b77804d50 in ?? ()
Id Target Id Frame
* 1 LWP 13768 "kudu" 0x00007f6b77804d50 in ?? ()
2 LWP 13770 "kudu" 0x00007f6b77800fb9 in ?? ()
3 LWP 13771 "kudu" 0x00007f6b77800fb9 in ?? ()
4 LWP 13772 "kudu" 0x00007f6b77800fb9 in ?? ()
5 LWP 13773 "kernel-watcher-" 0x00007f6b77800fb9 in ?? ()
6 LWP 13780 "ntp client-1378" 0x00007f6b778049e2 in ?? ()
7 LWP 13781 "file cache-evic" 0x00007f6b77800fb9 in ?? ()
8 LWP 13782 "sq_acceptor" 0x00007f6b7055dcb9 in ?? ()
9 LWP 13785 "rpc reactor-137" 0x00007f6b7056aa47 in ?? ()
10 LWP 13786 "rpc reactor-137" 0x00007f6b7056aa47 in ?? ()
11 LWP 13787 "rpc reactor-137" 0x00007f6b7056aa47 in ?? ()
12 LWP 13788 "rpc reactor-137" 0x00007f6b7056aa47 in ?? ()
13 LWP 13789 "MaintenanceMgr " 0x00007f6b77800ad3 in ?? ()
14 LWP 13790 "txn-status-mana" 0x00007f6b77800fb9 in ?? ()
15 LWP 13792 "collect_and_rem" 0x00007f6b77800fb9 in ?? ()
16 LWP 13793 "tc-session-exp-" 0x00007f6b77800fb9 in ?? ()
17 LWP 13794 "rpc worker-1379" 0x00007f6b77800ad3 in ?? ()
18 LWP 13795 "rpc worker-1379" 0x00007f6b77800ad3 in ?? ()
19 LWP 13796 "rpc worker-1379" 0x00007f6b77800ad3 in ?? ()
20 LWP 13797 "rpc worker-1379" 0x00007f6b77800ad3 in ?? ()
21 LWP 13798 "rpc worker-1379" 0x00007f6b77800ad3 in ?? ()
22 LWP 13799 "rpc worker-1379" 0x00007f6b77800ad3 in ?? ()
23 LWP 13800 "rpc worker-1380" 0x00007f6b77800ad3 in ?? ()
24 LWP 13801 "rpc worker-1380" 0x00007f6b77800ad3 in ?? ()
25 LWP 13802 "rpc worker-1380" 0x00007f6b77800ad3 in ?? ()
26 LWP 13803 "rpc worker-1380" 0x00007f6b77800ad3 in ?? ()
27 LWP 13804 "rpc worker-1380" 0x00007f6b77800ad3 in ?? ()
28 LWP 13805 "rpc worker-1380" 0x00007f6b77800ad3 in ?? ()
29 LWP 13806 "rpc worker-1380" 0x00007f6b77800ad3 in ?? ()
30 LWP 13807 "rpc worker-1380" 0x00007f6b77800ad3 in ?? ()
31 LWP 13808 "rpc worker-1380" 0x00007f6b77800ad3 in ?? ()
32 LWP 13809 "rpc worker-1380" 0x00007f6b77800ad3 in ?? ()
33 LWP 13810 "rpc worker-1381" 0x00007f6b77800ad3 in ?? ()
34 LWP 13811 "rpc worker-1381" 0x00007f6b77800ad3 in ?? ()
35 LWP 13812 "rpc worker-1381" 0x00007f6b77800ad3 in ?? ()
36 LWP 13813 "rpc worker-1381" 0x00007f6b77800ad3 in ?? ()
37 LWP 13814 "rpc worker-1381" 0x00007f6b77800ad3 in ?? ()
38 LWP 13815 "rpc worker-1381" 0x00007f6b77800ad3 in ?? ()
39 LWP 13816 "rpc worker-1381" 0x00007f6b77800ad3 in ?? ()
40 LWP 13817 "rpc worker-1381" 0x00007f6b77800ad3 in ?? ()
41 LWP 13818 "rpc worker-1381" 0x00007f6b77800ad3 in ?? ()
42 LWP 13819 "rpc worker-1381" 0x00007f6b77800ad3 in ?? ()
43 LWP 13820 "rpc worker-1382" 0x00007f6b77800ad3 in ?? ()
44 LWP 13822 "rpc worker-1382" 0x00007f6b77800ad3 in ?? ()
45 LWP 13823 "rpc worker-1382" 0x00007f6b77800ad3 in ?? ()
46 LWP 13824 "rpc worker-1382" 0x00007f6b77800ad3 in ?? ()
47 LWP 13825 "rpc worker-1382" 0x00007f6b77800ad3 in ?? ()
48 LWP 13826 "rpc worker-1382" 0x00007f6b77800ad3 in ?? ()
49 LWP 13827 "rpc worker-1382" 0x00007f6b77800ad3 in ?? ()
50 LWP 13828 "rpc worker-1382" 0x00007f6b77800ad3 in ?? ()
51 LWP 13829 "rpc worker-1382" 0x00007f6b77800ad3 in ?? ()
52 LWP 13830 "rpc worker-1383" 0x00007f6b77800ad3 in ?? ()
53 LWP 13831 "rpc worker-1383" 0x00007f6b77800ad3 in ?? ()
54 LWP 13832 "rpc worker-1383" 0x00007f6b77800ad3 in ?? ()
55 LWP 13833 "rpc worker-1383" 0x00007f6b77800ad3 in ?? ()
56 LWP 13834 "rpc worker-1383" 0x00007f6b77800ad3 in ?? ()
57 LWP 13835 "rpc worker-1383" 0x00007f6b77800ad3 in ?? ()
58 LWP 13836 "rpc worker-1383" 0x00007f6b77800ad3 in ?? ()
59 LWP 13837 "rpc worker-1383" 0x00007f6b77800ad3 in ?? ()
60 LWP 13838 "rpc worker-1383" 0x00007f6b77800ad3 in ?? ()
61 LWP 13839 "rpc worker-1383" 0x00007f6b77800ad3 in ?? ()
62 LWP 13840 "rpc worker-1384" 0x00007f6b77800ad3 in ?? ()
63 LWP 13841 "rpc worker-1384" 0x00007f6b77800ad3 in ?? ()
64 LWP 13842 "rpc worker-1384" 0x00007f6b77800ad3 in ?? ()
65 LWP 13843 "rpc worker-1384" 0x00007f6b77800ad3 in ?? ()
66 LWP 13844 "rpc worker-1384" 0x00007f6b77800ad3 in ?? ()
67 LWP 13845 "rpc worker-1384" 0x00007f6b77800ad3 in ?? ()
68 LWP 13846 "rpc worker-1384" 0x00007f6b77800ad3 in ?? ()
69 LWP 13847 "rpc worker-1384" 0x00007f6b77800ad3 in ?? ()
70 LWP 13848 "rpc worker-1384" 0x00007f6b77800ad3 in ?? ()
71 LWP 13849 "rpc worker-1384" 0x00007f6b77800ad3 in ?? ()
72 LWP 13850 "rpc worker-1385" 0x00007f6b77800ad3 in ?? ()
73 LWP 13851 "rpc worker-1385" 0x00007f6b77800ad3 in ?? ()
74 LWP 13852 "rpc worker-1385" 0x00007f6b77800ad3 in ?? ()
75 LWP 13853 "rpc worker-1385" 0x00007f6b77800ad3 in ?? ()
76 LWP 13854 "rpc worker-1385" 0x00007f6b77800ad3 in ?? ()
77 LWP 13855 "rpc worker-1385" 0x00007f6b77800ad3 in ?? ()
78 LWP 13856 "rpc worker-1385" 0x00007f6b77800ad3 in ?? ()
79 LWP 13857 "rpc worker-1385" 0x00007f6b77800ad3 in ?? ()
80 LWP 13858 "rpc worker-1385" 0x00007f6b77800ad3 in ?? ()
81 LWP 13859 "rpc worker-1385" 0x00007f6b77800ad3 in ?? ()
82 LWP 13860 "rpc worker-1386" 0x00007f6b77800ad3 in ?? ()
83 LWP 13861 "rpc worker-1386" 0x00007f6b77800ad3 in ?? ()
84 LWP 13862 "rpc worker-1386" 0x00007f6b77800ad3 in ?? ()
85 LWP 13863 "rpc worker-1386" 0x00007f6b77800ad3 in ?? ()
86 LWP 13864 "rpc worker-1386" 0x00007f6b77800ad3 in ?? ()
87 LWP 13865 "rpc worker-1386" 0x00007f6b77800ad3 in ?? ()
88 LWP 13866 "rpc worker-1386" 0x00007f6b77800ad3 in ?? ()
89 LWP 13867 "rpc worker-1386" 0x00007f6b77800ad3 in ?? ()
90 LWP 13868 "rpc worker-1386" 0x00007f6b77800ad3 in ?? ()
91 LWP 13869 "rpc worker-1386" 0x00007f6b77800ad3 in ?? ()
92 LWP 13870 "rpc worker-1387" 0x00007f6b77800ad3 in ?? ()
93 LWP 13871 "rpc worker-1387" 0x00007f6b77800ad3 in ?? ()
94 LWP 13872 "rpc worker-1387" 0x00007f6b77800ad3 in ?? ()
95 LWP 13873 "rpc worker-1387" 0x00007f6b77800ad3 in ?? ()
96 LWP 13874 "rpc worker-1387" 0x00007f6b77800ad3 in ?? ()
97 LWP 13875 "rpc worker-1387" 0x00007f6b77800ad3 in ?? ()
98 LWP 13876 "rpc worker-1387" 0x00007f6b77800ad3 in ?? ()
99 LWP 13877 "rpc worker-1387" 0x00007f6b77800ad3 in ?? ()
100 LWP 13878 "rpc worker-1387" 0x00007f6b77800ad3 in ?? ()
101 LWP 13879 "rpc worker-1387" 0x00007f6b77800ad3 in ?? ()
102 LWP 13881 "rpc worker-1388" 0x00007f6b77800ad3 in ?? ()
103 LWP 13882 "rpc worker-1388" 0x00007f6b77800ad3 in ?? ()
104 LWP 13883 "rpc worker-1388" 0x00007f6b77800ad3 in ?? ()
105 LWP 13884 "rpc worker-1388" 0x00007f6b77800ad3 in ?? ()
106 LWP 13885 "rpc worker-1388" 0x00007f6b77800ad3 in ?? ()
107 LWP 13886 "rpc worker-1388" 0x00007f6b77800ad3 in ?? ()
108 LWP 13887 "rpc worker-1388" 0x00007f6b77800ad3 in ?? ()
109 LWP 13888 "rpc worker-1388" 0x00007f6b77800ad3 in ?? ()
110 LWP 13889 "rpc worker-1388" 0x00007f6b77800ad3 in ?? ()
111 LWP 13890 "rpc worker-1389" 0x00007f6b77800ad3 in ?? ()
112 LWP 13891 "rpc worker-1389" 0x00007f6b77800ad3 in ?? ()
113 LWP 13892 "rpc worker-1389" 0x00007f6b77800ad3 in ?? ()
114 LWP 13893 "rpc worker-1389" 0x00007f6b77800ad3 in ?? ()
115 LWP 13894 "rpc worker-1389" 0x00007f6b77800ad3 in ?? ()
116 LWP 13895 "rpc worker-1389" 0x00007f6b77800ad3 in ?? ()
117 LWP 13896 "diag-logger-138" 0x00007f6b77800fb9 in ?? ()
118 LWP 13897 "result-tracker-" 0x00007f6b77800fb9 in ?? ()
119 LWP 13898 "excess-log-dele" 0x00007f6b77800fb9 in ?? ()
120 LWP 13899 "acceptor-13899" 0x00007f6b7056c0c7 in ?? ()
121 LWP 13900 "heartbeat-13900" 0x00007f6b77800fb9 in ?? ()
122 LWP 13901 "maintenance_sch" 0x00007f6b77800fb9 in ?? ()
123 LWP 14320 "prepare [worker" 0x00007f6b83de4a8b in ?? ()
124 LWP 14321 "wal-append [wor" 0x00007f6b77800fb9 in ?? ()
125 LWP 14322 "apply [worker]-" 0x00000000009f1210 in __sanitizer::theDepot ()
126 LWP 14323 "apply [worker]-" 0x00007f6b879df8d0 in ?? ()
127 LWP 14324 "apply [worker]-" 0x0000000000967448 in __sanitizer::theDepot ()
128 LWP 14325 "apply [worker]-" 0x00007f6b81ae2a98 in ?? ()
Thread 128 (LWP 14325):
#0 0x00007f6b81ae2a98 in ?? ()
#1 0x00007f6b766dc677 in ?? ()
#2 0x0000000041b58ab3 in ?? ()
#3 0x00007f6b766aba0a in ?? ()
#4 0x00007f6b766dc2a0 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 127 (LWP 14324):
#0 0x0000000000967448 in __sanitizer::theDepot ()
#1 0x00007f6b80eb1855 in ?? ()
#2 0x0000000041b58ab3 in ?? ()
#3 0x00007f6b809a1834 in ?? ()
#4 0x00007f6b80eb16b0 in ?? ()
#5 0x00007f6b21b529a0 in ?? ()
#6 0x0000000000000019 in ?? ()
#7 0x00007f6b809a762e in ?? ()
#8 0x0000000000000032 in ?? ()
#9 0x0000000000a564f3 in __sanitizer::theDepot ()
#10 0x00007f6b21b52af0 in ?? ()
#11 0x00007f6b21b52ba0 in ?? ()
#12 0x00007f6b21b52b20 in ?? ()
#13 0x00006190012a1e90 in ?? ()
#14 0x00000fed6436a560 in ?? ()
#15 0x000060c000180f40 in ?? ()
#16 0x00007f6b21b52c40 in ?? ()
#17 0x00007f6b7445bd85 in ?? ()
#18 0x0000000045e0360e in ?? ()
#19 0x00007f6b73d5a6f1 in ?? ()
#20 0x00007f6b7445bad0 in ?? ()
#21 0x00000fede43624b8 in ?? ()
#22 0x00007f6b73396720 in ?? ()
#23 0x00007f6b73396720 in ?? ()
#24 0x00007f6b73396720 in ?? ()
#25 0x00007f6b73396720 in ?? ()
#26 0x00007f6b73396720 in ?? ()
#27 0x00007f6b73396720 in ?? ()
#28 0x00007f6b73396720 in ?? ()
#29 0x00007f6b73396720 in ?? ()
#30 0x00007f6b73396720 in ?? ()
#31 0x00007f6b73396720 in ?? ()
#32 0x0000000000000000 in ?? ()
Thread 126 (LWP 14323):
#0 0x00007f6b879df8d0 in ?? ()
#1 0x00007f6b8111315f in ?? ()
#2 0x0000000050000001 in ?? ()
#3 0x00007f6b28c40550 in ?? ()
#4 0x00007f6b2a7eee90 in ?? ()
#5 0x00000fed654fddd2 in ?? ()
#6 0x00000fed654fddd2 in ?? ()
#7 0x00000fed6446a301 in ?? ()
#8 0x00007f6b22351980 in ?? ()
#9 0x00007f6b81112c16 in ?? ()
#10 0x00007f6b223519d0 in ?? ()
#11 0x00007f6b8111999b in ?? ()
#12 0x00007f6b2880d778 in ?? ()
#13 0x00007f6b2a7eee97 in ?? ()
#14 0x00007f6b2a7eee97 in ?? ()
#15 0x00007f6b223520a0 in ?? ()
#16 0x00007f6b2a7eee88 in ?? ()
#17 0x00000fed6446a33c in ?? ()
#18 0x00000fede446233c in ?? ()
#19 0x00000fed6446a340 in ?? ()
#20 0x00007f6b223521b0 in ?? ()
#21 0x00007f6b81117860 in ?? ()
#22 0x0000000041b58ab3 in ?? ()
#23 0x00007f6b809905df in ?? ()
#24 0x00007f6b811175c0 in ?? ()
#25 0x00007f6b22352e40 in ?? ()
#26 0x80007f6b2a7eee88 in ?? ()
#27 0x0000000000000039 in ?? ()
#28 0x00007f6b22351a30 in ?? ()
#29 0x000060c000036700 in ?? ()
#30 0x80007f6b2880d678 in ?? ()
#31 0x00007f6b22352e40 in ?? ()
#32 0x0000000000000008 in ?? ()
#33 0x0000000000000039 in ?? ()
#34 0x00007f6b22351a70 in ?? ()
#35 0x00007f6b22351b80 in ?? ()
#36 0x00000fede4462350 in ?? ()
#37 0x00007f6b22352e40 in ?? ()
#38 0x00007f6b22352aa0 in ?? ()
#39 0x00007f6b22351a80 in ?? ()
#40 0x00007f6b22351c10 in ?? ()
#41 0x00007f6b81116ccb in ?? ()
#42 0x0000000045e0360e in ?? ()
#43 0x00007f6b809cc241 in ?? ()
#44 0x00007f6b81116ab0 in ?? ()
#45 0x0000000000000004 in ?? ()
#46 0x0000000000000031 in ?? ()
#47 0x00007f6b22351b40 in ?? ()
#48 0x0000000000000000 in ?? ()
Thread 125 (LWP 14322):
#0 0x00000000009f1210 in __sanitizer::theDepot ()
#1 0x00000000009f0ee2 in __sanitizer::theDepot ()
#2 0x00007f6b22b54030 in ?? ()
#3 0x00007f6b810e7d3c in ?? ()
#4 0x0000000041b58ab3 in ?? ()
#5 0x00007f6b809aeb6c in ?? ()
#6 0x00007f6b810e7b90 in ?? ()
#7 0x00007f6b81022fc0 in ?? ()
#8 0x00007f6b80ff7e3c in ?? ()
#9 0x00007f6b80ffb421 in ?? ()
#10 0x00007f6b22b53b60 in ?? ()
#11 0x0000000000a71ecd in __sanitizer::theDepot ()
#12 0x0000000000000100 in ?? ()
#13 0x0000000000000048 in ?? ()
#14 0x00007f6b22b53b80 in ?? ()
#15 0x000061700044c2e8 in ?? ()
#16 0x0000000000000400 in ?? ()
#17 0x0000000000000040 in ?? ()
#18 0x0000000000000400 in ?? ()
#19 0x000061700044c2e8 in ?? ()
#20 0x00007f6b22b53c10 in ?? ()
#21 0x00007f6b71c46ed5 in ?? ()
#22 0x00000000000004b0 in ?? ()
#23 0x00007f6b00000000 in ?? ()
#24 0x0000000000ed5be0 in ?? ()
#25 0x0000000000000014 in ?? ()
#26 0x0000000000000000 in ?? ()
Thread 124 (LWP 14321):
#0 0x00007f6b77800fb9 in ?? ()
#1 0x0000602000143278 in ?? ()
#2 0x0000000000001029 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x000061a000116bb4 in ?? ()
#5 0x00007f6b23356090 in ?? ()
#6 0x0000000000002053 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f6b233560b0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 123 (LWP 14320):
#0 0x00007f6b83de4a8b in ?? ()
#1 0x00007f6b2be9a778 in ?? ()
#2 0x00007f6b2be9a420 in ?? ()
#3 0x00000fed657d3480 in ?? ()
#4 0x00007f6b2be9a3e0 in ?? ()
#5 0x00000fed657d347c in ?? ()
#6 0x00007f6b2be9a400 in ?? ()
#7 0x00007f6b2be9a490 in ?? ()
#8 0x00007f6b76250d11 in ?? ()
#9 0x0000000041b58ab3 in ?? ()
#10 0x00007f6b75f9518d in ?? ()
#11 0x00007f6b76250be0 in ?? ()
#12 0x00007f6b2be9a2e0 in ?? ()
#13 0x00000000de0b51a3 in ?? ()
#14 0x00007f6b8098c87f in ?? ()
#15 0x0000000000000000 in ?? ()
Thread 122 (LWP 13901):
#0 0x00007f6b77800fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000117 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007f6b2c6aa550 in ?? ()
#6 0x000000000000022e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 13900):
#0 0x00007f6b77800fb9 in ?? ()
#1 0x99d1b6c4365c6052 in ?? ()
#2 0x0000000000000046 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001aac0 in ?? ()
#5 0x00007f6b2cec25b0 in ?? ()
#6 0x000000000000008c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 13899):
#0 0x00007f6b7056c0c7 in ?? ()
#1 0x00006140000212a8 in ?? ()
#2 0x00007f6b2d6e9d50 in ?? ()
#3 0x00007f6b2d6e9d80 in ?? ()
#4 0x00007f6b2d6e9e80 in ?? ()
#5 0x00007f6b2d6e9d70 in ?? ()
#6 0x00007f6b2d6e9de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x0000000000000000 in ?? ()
Thread 119 (LWP 13898):
#0 0x00007f6b77800fb9 in ?? ()
#1 0x00007f6b2df02f00 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ffd2c2e19b0 in ?? ()
#5 0x00007f6b2df02ef0 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 13897):
#0 0x00007f6b77800fb9 in ?? ()
#1 0x00007f6b2e71b0c0 in ?? ()
#2 0x0000000000000046 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007f6b2e71b0b0 in ?? ()
#6 0x000000000000008c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 13896):
#0 0x00007f6b77800fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 13895):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000017 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00041688c in ?? ()
#4 0x00007f6b2f74be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f6b2f74be70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d000416840 in ?? ()
#9 0x00007f6b77800770 in ?? ()
#10 0x00007f6b2f74be70 in ?? ()
#11 0x00000fed65ee97d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 115 (LWP 13894):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000009 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0004067cc in ?? ()
#4 0x00007f6b2ff63e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f6b2ff63e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d000406780 in ?? ()
#9 0x00007f6b77800770 in ?? ()
#10 0x00007f6b2ff63e70 in ?? ()
#11 0x00000fed65fec7d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 114 (LWP 13893):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 13892):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 13891):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 13890):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 13889):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 13888):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 13887):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 13886):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 13885):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 13884):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 13883):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 13882):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 13881):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 13879):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 13878):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 13877):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 13876):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 13875):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 13874):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 13873):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 13872):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 13871):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 13870):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 13869):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 13868):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 13867):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 13866):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 13865):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 13864):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 13863):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 13862):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 13861):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 13860):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 13859):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 13858):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 13857):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 13856):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 13855):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 13854):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x00000000000001c4 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000270088 in ?? ()
#4 0x00007f6b43373e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f6b43373e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 13853):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x00000000000000a9 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0002667cc in ?? ()
#4 0x00007f6b43b8be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f6b43b8be70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d000266780 in ?? ()
#9 0x00007f6b77800770 in ?? ()
#10 0x00007f6b43b8be70 in ?? ()
#11 0x00000fed687717d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 74 (LWP 13852):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 13851):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 13850):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 13849):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 13848):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 13847):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 13846):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 13845):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 13844):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 13843):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 13842):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 13841):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 13840):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 13839):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 13838):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 13837):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 13836):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 13835):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 13834):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00017ffd8 in ?? ()
#4 0x00007f6b4d595e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f6b4d595e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 13833):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 13832):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 13831):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 13830):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 13829):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 13828):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 13827):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 13826):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 13825):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 13824):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 13823):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 13822):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 13820):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 13819):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 13818):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 13817):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 13816):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 13815):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 13814):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 13813):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x000000000000000e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000b67d8 in ?? ()
#4 0x00007f6b578a8e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f6b578a8e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 13812):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 13811):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 13810):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 13809):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 13808):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 13807):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 13806):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 13805):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 13804):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 13803):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 13802):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 13801):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 13800):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 13799):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 13798):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 13797):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 13796):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 13795):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 13794):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 13793):
#0 0x00007f6b77800fb9 in ?? ()
#1 0x00007f6b61acdc80 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ba60 in ?? ()
#5 0x00007f6b61acdc70 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 13792):
#0 0x00007f6b77800fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 13790):
#0 0x00007f6b77800fb9 in ?? ()
#1 0x00007f6b633451c0 in ?? ()
#2 0x0000000000000007 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000f9a8 in ?? ()
#5 0x00007f6b633451b0 in ?? ()
#6 0x000000000000000e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 13789):
#0 0x00007f6b77800ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 13788):
#0 0x00007f6b7056aa47 in ?? ()
#1 0x00007f6b643942e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007f6b643942d0 in ?? ()
#4 0x00007f6b643944e0 in ?? ()
#5 0x00007f6b64394320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007f6b643943a0 in ?? ()
#8 0x00007f6b72bca25d in ?? ()
#9 0x3fb974bea1806000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x00000000a6000000 in ?? ()
#14 0x00007f6b00000000 in ?? ()
#15 0x41d9e134cd666729 in ?? ()
#16 0x00000fedec86a874 in ?? ()
#17 0x00007f6b64394380 in ?? ()
#18 0x00007f6b72bceba3 in ?? ()
#19 0x00007f6b64394350 in ?? ()
#20 0x3fb974bea1806000 in ?? ()
#21 0x00000000643943a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb974bea1806000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 13787):
#0 0x00007f6b7056aa47 in ?? ()
#1 0x00007f6b64bab2e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007f6b64bab2d0 in ?? ()
#4 0x00007f6b64bab4e0 in ?? ()
#5 0x00007f6b64bab320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f6b64bab3a0 in ?? ()
#8 0x00007f6b72bca25d in ?? ()
#9 0x3fb4d8eff0d02000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x0000000019000000 in ?? ()
#14 0x00007f6b00000000 in ?? ()
#15 0x41d9e134cd666726 in ?? ()
#16 0x00000fedec96d674 in ?? ()
#17 0x00007f6b64bab380 in ?? ()
#18 0x00007f6b72bceba3 in ?? ()
#19 0x00007f6b64bab350 in ?? ()
#20 0x3fb4d8eff0d02000 in ?? ()
#21 0x0000000064bab3a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb4d8eff0d02000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 13786):
#0 0x00007f6b7056aa47 in ?? ()
#1 0x00007f6b653b42e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007f6b653b42d0 in ?? ()
#4 0x00007f6b653b44e0 in ?? ()
#5 0x00007f6b653b4320 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f6b653b43a0 in ?? ()
#8 0x00007f6b72bca25d in ?? ()
#9 0x3fb977bee4a4e000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000042000000 in ?? ()
#14 0x00007f6b00000000 in ?? ()
#15 0x41d9e134cd666727 in ?? ()
#16 0x00000fedeca6e874 in ?? ()
#17 0x00007f6b653b4380 in ?? ()
#18 0x00007f6b72bceba3 in ?? ()
#19 0x00007f6b653b4350 in ?? ()
#20 0x3fb977bee4a4e000 in ?? ()
#21 0x00000000653b43a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb977bee4a4e000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 13785):
#0 0x00007f6b7056aa47 in ?? ()
#1 0x00007f6b673ad2e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007f6b673ad2d0 in ?? ()
#4 0x00007f6b673ad4e0 in ?? ()
#5 0x00007f6b673ad320 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f6b673ad3a0 in ?? ()
#8 0x00007f6b72bca25d in ?? ()
#9 0x3f998186fe7e0000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x0000000037000000 in ?? ()
#14 0x00007f6b00000000 in ?? ()
#15 0x41d9e134cd666726 in ?? ()
#16 0x00000fedece6da74 in ?? ()
#17 0x00007f6b673ad380 in ?? ()
#18 0x00007f6b72bceba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 13782):
#0 0x00007f6b7055dcb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f6b68db7788 in ?? ()
#3 0x0000602000017cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 13781):
#0 0x00007f6b77800fb9 in ?? ()
#1 0x00007f6b685b6080 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007f6b685b6070 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 13780):
#0 0x00007f6b778049e2 in ?? ()
#1 0x00007f6b67db3b60 in ?? ()
#2 0x00007f6b67db3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f6b67db3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007f6b67db3c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007f6b67db3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007f6b7d128f0c in ?? ()
#12 0x00007f6b7d1187b9 in ?? ()
#13 0x00007f6b7d11cda6 in ?? ()
#14 0x00007f6b7d1217e0 in ?? ()
#15 0x00007f6b7d121609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007f6b744170be in ?? ()
#18 0x00007f6b777fa6db in ?? ()
#19 0x00000fedecfae67c in ?? ()
#20 0x00007f6b67db3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 13773):
#0 0x00007f6b77800fb9 in ?? ()
#1 0x00007f6b69db8c40 in ?? ()
#2 0x0000000000000166 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007f6b69db8c30 in ?? ()
#6 0x00000000000002cc in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 13772):
#0 0x00007f6b77800fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 13771):
#0 0x00007f6b77800fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 13770):
#0 0x00007f6b77800fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000004 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f6b6b5bc410 in ?? ()
#6 0x0000000000000008 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 13768):
#0 0x00007f6b77804d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250113 08:57:35.139684 11762 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 3 with UUID b7d3e63990b74f76afc0e6cd5afbc13b and pid 13157
************************ BEGIN STACKS **************************
[New LWP 13161]
[New LWP 13162]
[New LWP 13163]
[New LWP 13164]
[New LWP 13170]
[New LWP 13171]
[New LWP 13172]
[New LWP 13175]
[New LWP 13176]
[New LWP 13177]
[New LWP 13178]
[New LWP 13179]
[New LWP 13180]
[New LWP 13182]
[New LWP 13183]
[New LWP 13184]
[New LWP 13185]
[New LWP 13186]
[New LWP 13187]
[New LWP 13188]
[New LWP 13189]
[New LWP 13190]
[New LWP 13191]
[New LWP 13192]
[New LWP 13193]
[New LWP 13194]
[New LWP 13195]
[New LWP 13196]
[New LWP 13197]
[New LWP 13198]
[New LWP 13199]
[New LWP 13200]
[New LWP 13201]
[New LWP 13202]
[New LWP 13203]
[New LWP 13204]
[New LWP 13205]
[New LWP 13206]
[New LWP 13207]
[New LWP 13208]
[New LWP 13209]
[New LWP 13210]
[New LWP 13211]
[New LWP 13212]
[New LWP 13213]
[New LWP 13214]
[New LWP 13215]
[New LWP 13216]
[New LWP 13217]
[New LWP 13218]
[New LWP 13219]
[New LWP 13220]
[New LWP 13221]
[New LWP 13222]
[New LWP 13223]
[New LWP 13224]
[New LWP 13225]
[New LWP 13226]
[New LWP 13227]
[New LWP 13228]
[New LWP 13229]
[New LWP 13230]
[New LWP 13231]
[New LWP 13232]
[New LWP 13233]
[New LWP 13234]
[New LWP 13235]
[New LWP 13236]
[New LWP 13237]
[New LWP 13238]
[New LWP 13239]
[New LWP 13240]
[New LWP 13241]
[New LWP 13242]
[New LWP 13243]
[New LWP 13244]
[New LWP 13245]
[New LWP 13246]
[New LWP 13247]
[New LWP 13248]
[New LWP 13249]
[New LWP 13250]
[New LWP 13251]
[New LWP 13252]
[New LWP 13253]
[New LWP 13254]
[New LWP 13255]
[New LWP 13256]
[New LWP 13257]
[New LWP 13258]
[New LWP 13259]
[New LWP 13260]
[New LWP 13261]
[New LWP 13262]
[New LWP 13263]
[New LWP 13264]
[New LWP 13265]
[New LWP 13266]
[New LWP 13267]
[New LWP 13268]
[New LWP 13269]
[New LWP 13270]
[New LWP 13271]
[New LWP 13272]
[New LWP 13273]
[New LWP 13274]
[New LWP 13275]
[New LWP 13276]
[New LWP 13277]
[New LWP 13278]
[New LWP 13279]
[New LWP 13280]
[New LWP 13281]
[New LWP 13282]
[New LWP 13283]
[New LWP 13284]
[New LWP 13285]
[New LWP 13286]
[New LWP 13287]
[New LWP 13289]
[New LWP 13290]
[New LWP 14370]
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7469
Cannot access memory at address 0x736e753c726f7461
0x00007f9f2e99dd50 in ?? ()
Id Target Id Frame
* 1 LWP 13157 "kudu" 0x00007f9f2e99dd50 in ?? ()
2 LWP 13161 "kudu" 0x00007f9f2e999fb9 in ?? ()
3 LWP 13162 "kudu" 0x00007f9f2e999fb9 in ?? ()
4 LWP 13163 "kudu" 0x00007f9f2e999fb9 in ?? ()
5 LWP 13164 "kernel-watcher-" 0x00007f9f2e999fb9 in ?? ()
6 LWP 13170 "ntp client-1317" 0x00007f9f2e99d9e2 in ?? ()
7 LWP 13171 "file cache-evic" 0x00007f9f2e999fb9 in ?? ()
8 LWP 13172 "sq_acceptor" 0x00007f9f276f6cb9 in ?? ()
9 LWP 13175 "rpc reactor-131" 0x00007f9f27703a47 in ?? ()
10 LWP 13176 "rpc reactor-131" 0x00007f9f27703a47 in ?? ()
11 LWP 13177 "rpc reactor-131" 0x00007f9f27703a47 in ?? ()
12 LWP 13178 "rpc reactor-131" 0x00007f9f27703a47 in ?? ()
13 LWP 13179 "MaintenanceMgr " 0x00007f9f2e999ad3 in ?? ()
14 LWP 13180 "txn-status-mana" 0x00007f9f2e999fb9 in ?? ()
15 LWP 13182 "collect_and_rem" 0x00007f9f2e999fb9 in ?? ()
16 LWP 13183 "tc-session-exp-" 0x00007f9f2e999fb9 in ?? ()
17 LWP 13184 "rpc worker-1318" 0x00007f9f2e999ad3 in ?? ()
18 LWP 13185 "rpc worker-1318" 0x00007f9f2e999ad3 in ?? ()
19 LWP 13186 "rpc worker-1318" 0x00007f9f2e999ad3 in ?? ()
20 LWP 13187 "rpc worker-1318" 0x00007f9f2e999ad3 in ?? ()
21 LWP 13188 "rpc worker-1318" 0x00007f9f2e999ad3 in ?? ()
22 LWP 13189 "rpc worker-1318" 0x00007f9f2e999ad3 in ?? ()
23 LWP 13190 "rpc worker-1319" 0x00007f9f2e999ad3 in ?? ()
24 LWP 13191 "rpc worker-1319" 0x00007f9f2e999ad3 in ?? ()
25 LWP 13192 "rpc worker-1319" 0x00007f9f2e999ad3 in ?? ()
26 LWP 13193 "rpc worker-1319" 0x00007f9f2e999ad3 in ?? ()
27 LWP 13194 "rpc worker-1319" 0x00007f9f2e999ad3 in ?? ()
28 LWP 13195 "rpc worker-1319" 0x00007f9f2e999ad3 in ?? ()
29 LWP 13196 "rpc worker-1319" 0x00007f9f2e999ad3 in ?? ()
30 LWP 13197 "rpc worker-1319" 0x00007f9f2e999ad3 in ?? ()
31 LWP 13198 "rpc worker-1319" 0x00007f9f2e999ad3 in ?? ()
32 LWP 13199 "rpc worker-1319" 0x00007f9f2e999ad3 in ?? ()
33 LWP 13200 "rpc worker-1320" 0x00007f9f2e999ad3 in ?? ()
34 LWP 13201 "rpc worker-1320" 0x00007f9f2e999ad3 in ?? ()
35 LWP 13202 "rpc worker-1320" 0x00007f9f2e999ad3 in ?? ()
36 LWP 13203 "rpc worker-1320" 0x00007f9f2e999ad3 in ?? ()
37 LWP 13204 "rpc worker-1320" 0x00007f9f2e999ad3 in ?? ()
38 LWP 13205 "rpc worker-1320" 0x00007f9f2e999ad3 in ?? ()
39 LWP 13206 "rpc worker-1320" 0x00007f9f2e999ad3 in ?? ()
40 LWP 13207 "rpc worker-1320" 0x00007f9f2e999ad3 in ?? ()
41 LWP 13208 "rpc worker-1320" 0x00007f9f2e999ad3 in ?? ()
42 LWP 13209 "rpc worker-1320" 0x00007f9f2e999ad3 in ?? ()
43 LWP 13210 "rpc worker-1321" 0x00007f9f2e999ad3 in ?? ()
44 LWP 13211 "rpc worker-1321" 0x00007f9f2e999ad3 in ?? ()
45 LWP 13212 "rpc worker-1321" 0x00007f9f2e999ad3 in ?? ()
46 LWP 13213 "rpc worker-1321" 0x00007f9f2e999ad3 in ?? ()
47 LWP 13214 "rpc worker-1321" 0x00007f9f2e999ad3 in ?? ()
48 LWP 13215 "rpc worker-1321" 0x00007f9f2e999ad3 in ?? ()
49 LWP 13216 "rpc worker-1321" 0x00007f9f2e999ad3 in ?? ()
50 LWP 13217 "rpc worker-1321" 0x00007f9f2e999ad3 in ?? ()
51 LWP 13218 "rpc worker-1321" 0x00007f9f2e999ad3 in ?? ()
52 LWP 13219 "rpc worker-1321" 0x00007f9f2e999ad3 in ?? ()
53 LWP 13220 "rpc worker-1322" 0x00007f9f2e999ad3 in ?? ()
54 LWP 13221 "rpc worker-1322" 0x00007f9f2e999ad3 in ?? ()
55 LWP 13222 "rpc worker-1322" 0x00007f9f2e999ad3 in ?? ()
56 LWP 13223 "rpc worker-1322" 0x00007f9f2e999ad3 in ?? ()
57 LWP 13224 "rpc worker-1322" 0x00007f9f2e999ad3 in ?? ()
58 LWP 13225 "rpc worker-1322" 0x00007f9f2e999ad3 in ?? ()
59 LWP 13226 "rpc worker-1322" 0x00007f9f2e999ad3 in ?? ()
60 LWP 13227 "rpc worker-1322" 0x00007f9f2e999ad3 in ?? ()
61 LWP 13228 "rpc worker-1322" 0x00007f9f2e999ad3 in ?? ()
62 LWP 13229 "rpc worker-1322" 0x00007f9f2e999ad3 in ?? ()
63 LWP 13230 "rpc worker-1323" 0x00007f9f2e999ad3 in ?? ()
64 LWP 13231 "rpc worker-1323" 0x00007f9f2e999ad3 in ?? ()
65 LWP 13232 "rpc worker-1323" 0x00007f9f2e999ad3 in ?? ()
66 LWP 13233 "rpc worker-1323" 0x00007f9f2e999ad3 in ?? ()
67 LWP 13234 "rpc worker-1323" 0x00007f9f2e999ad3 in ?? ()
68 LWP 13235 "rpc worker-1323" 0x00007f9f2e999ad3 in ?? ()
69 LWP 13236 "rpc worker-1323" 0x00007f9f2e999ad3 in ?? ()
70 LWP 13237 "rpc worker-1323" 0x00007f9f2e999ad3 in ?? ()
71 LWP 13238 "rpc worker-1323" 0x00007f9f2e999ad3 in ?? ()
72 LWP 13239 "rpc worker-1323" 0x00007f9f2e999ad3 in ?? ()
73 LWP 13240 "rpc worker-1324" 0x00007f9f2e999ad3 in ?? ()
74 LWP 13241 "rpc worker-1324" 0x00007f9f2e999ad3 in ?? ()
75 LWP 13242 "rpc worker-1324" 0x00007f9f2e999ad3 in ?? ()
76 LWP 13243 "rpc worker-1324" 0x00007f9f2e999ad3 in ?? ()
77 LWP 13244 "rpc worker-1324" 0x00007f9f2e999ad3 in ?? ()
78 LWP 13245 "rpc worker-1324" 0x00007f9f2e999ad3 in ?? ()
79 LWP 13246 "rpc worker-1324" 0x00007f9f2e999ad3 in ?? ()
80 LWP 13247 "rpc worker-1324" 0x00007f9f2e999ad3 in ?? ()
81 LWP 13248 "rpc worker-1324" 0x00007f9f2e999ad3 in ?? ()
82 LWP 13249 "rpc worker-1324" 0x00007f9f2e999ad3 in ?? ()
83 LWP 13250 "rpc worker-1325" 0x00007f9f2e999ad3 in ?? ()
84 LWP 13251 "rpc worker-1325" 0x00007f9f2e999ad3 in ?? ()
85 LWP 13252 "rpc worker-1325" 0x00007f9f2e999ad3 in ?? ()
86 LWP 13253 "rpc worker-1325" 0x00007f9f2e999ad3 in ?? ()
87 LWP 13254 "rpc worker-1325" 0x00007f9f2e999ad3 in ?? ()
88 LWP 13255 "rpc worker-1325" 0x00007f9f2e999ad3 in ?? ()
89 LWP 13256 "rpc worker-1325" 0x00007f9f2e999ad3 in ?? ()
90 LWP 13257 "rpc worker-1325" 0x00007f9f2e999ad3 in ?? ()
91 LWP 13258 "rpc worker-1325" 0x00007f9f2e999ad3 in ?? ()
92 LWP 13259 "rpc worker-1325" 0x00007f9f2e999ad3 in ?? ()
93 LWP 13260 "rpc worker-1326" 0x00007f9f2e999ad3 in ?? ()
94 LWP 13261 "rpc worker-1326" 0x00007f9f2e999ad3 in ?? ()
95 LWP 13262 "rpc worker-1326" 0x00007f9f2e999ad3 in ?? ()
96 LWP 13263 "rpc worker-1326" 0x00007f9f2e999ad3 in ?? ()
97 LWP 13264 "rpc worker-1326" 0x00007f9f2e999ad3 in ?? ()
98 LWP 13265 "rpc worker-1326" 0x00007f9f2e999ad3 in ?? ()
99 LWP 13266 "rpc worker-1326" 0x00007f9f2e999ad3 in ?? ()
100 LWP 13267 "rpc worker-1326" 0x00007f9f2e999ad3 in ?? ()
101 LWP 13268 "rpc worker-1326" 0x00007f9f2e999ad3 in ?? ()
102 LWP 13269 "rpc worker-1326" 0x00007f9f2e999ad3 in ?? ()
103 LWP 13270 "rpc worker-1327" 0x00007f9f2e999ad3 in ?? ()
104 LWP 13271 "rpc worker-1327" 0x00007f9f2e999ad3 in ?? ()
105 LWP 13272 "rpc worker-1327" 0x00007f9f2e999ad3 in ?? ()
106 LWP 13273 "rpc worker-1327" 0x00007f9f2e999ad3 in ?? ()
107 LWP 13274 "rpc worker-1327" 0x00007f9f2e999ad3 in ?? ()
108 LWP 13275 "rpc worker-1327" 0x00007f9f2e999ad3 in ?? ()
109 LWP 13276 "rpc worker-1327" 0x00007f9f2e999ad3 in ?? ()
110 LWP 13277 "rpc worker-1327" 0x00007f9f2e999ad3 in ?? ()
111 LWP 13278 "rpc worker-1327" 0x00007f9f2e999ad3 in ?? ()
112 LWP 13279 "rpc worker-1327" 0x00007f9f2e999ad3 in ?? ()
113 LWP 13280 "rpc worker-1328" 0x00007f9f2e999ad3 in ?? ()
114 LWP 13281 "rpc worker-1328" 0x00007f9f2e999ad3 in ?? ()
115 LWP 13282 "rpc worker-1328" 0x00007f9f2e999ad3 in ?? ()
116 LWP 13283 "rpc worker-1328" 0x00007f9f2e999ad3 in ?? ()
117 LWP 13284 "diag-logger-132" 0x00007f9f2e999fb9 in ?? ()
118 LWP 13285 "result-tracker-" 0x00007f9f2e999fb9 in ?? ()
119 LWP 13286 "excess-log-dele" 0x00007f9f2e999fb9 in ?? ()
120 LWP 13287 "acceptor-13287" 0x00007f9f277050c7 in ?? ()
121 LWP 13289 "heartbeat-13289" 0x00007f9f2e999fb9 in ?? ()
122 LWP 13290 "maintenance_sch" 0x00007f9f2e999fb9 in ?? ()
123 LWP 14370 "raft [worker]-1" 0x00007f9f2e999fb9 in ?? ()
Thread 123 (LWP 14370):
#0 0x00007f9f2e999fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000001c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007f9ed0d27b20 in ?? ()
#5 0x00007f9ed0d277b0 in ?? ()
#6 0x0000000000000038 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 13290):
#0 0x00007f9f2e999fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000211 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007f9ee3a25550 in ?? ()
#6 0x0000000000000422 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 13289):
#0 0x00007f9f2e999fb9 in ?? ()
#1 0x99d1b6c4365c6052 in ?? ()
#2 0x0000000000000088 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x000061300001aac4 in ?? ()
#5 0x00007f9ee423d5b0 in ?? ()
#6 0x0000000000000111 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f9ee423d5d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007f9ee423d650 in ?? ()
#12 0x00000ff3dc847acc in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 120 (LWP 13287):
#0 0x00007f9f277050c7 in ?? ()
#1 0x00006140000214a8 in ?? ()
#2 0x00007f9ee4a64d50 in ?? ()
#3 0x00007f9ee4a64d80 in ?? ()
#4 0x00007f9ee4a64e80 in ?? ()
#5 0x00007f9ee4a64d70 in ?? ()
#6 0x00007f9ee4a64de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x00007f9f2b58f620 in ?? ()
#10 0x0000000000000000 in ?? ()
Thread 119 (LWP 13286):
#0 0x00007f9f2e999fb9 in ?? ()
#1 0x00007f9ee527df00 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007fff5b44ba30 in ?? ()
#5 0x00007f9ee527def0 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 118 (LWP 13285):
#0 0x00007f9f2e999fb9 in ?? ()
#1 0x00007f9ee5a970c0 in ?? ()
#2 0x0000000000000084 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000611000087c30 in ?? ()
#5 0x00007f9ee5a970b0 in ?? ()
#6 0x0000000000000108 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 13284):
#0 0x00007f9f2e999fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 13283):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000008 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002c6838 in ?? ()
#4 0x00007f9ee6ac6e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f9ee6ac6e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 115 (LWP 13282):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x000000000000002c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0002c0038 in ?? ()
#4 0x00007f9ee72dee50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f9ee72dee70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 114 (LWP 13281):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 13280):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 13279):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 13278):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 13277):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 13276):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 13275):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 13274):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 13273):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 13272):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 13271):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 13270):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 13269):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 13268):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 13267):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 13266):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 13265):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 13264):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 13263):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 13262):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 13261):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 13260):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 13259):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 13258):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 13257):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 13256):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 13255):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 13254):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 13253):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 13252):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 13251):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 13250):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 13249):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 13248):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 13247):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 13246):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 13245):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 13244):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 13243):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x000000000000029f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001c006c in ?? ()
#4 0x00007f9efa67fe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f9efa67fe70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001c0020 in ?? ()
#9 0x00007f9f2e999770 in ?? ()
#10 0x00007f9efa67fe70 in ?? ()
#11 0x00000ff3df4cffd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 13242):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x00000000000000fa in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001b6878 in ?? ()
#4 0x00007f9efae97e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f9efae97e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 13241):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001b007c in ?? ()
#4 0x00007f9efb6afe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f9efb6afe70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001b0030 in ?? ()
#9 0x00007f9f2e999770 in ?? ()
#10 0x00007f9efb6afe70 in ?? ()
#11 0x00000ff3df6d5fd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 73 (LWP 13240):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 13239):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 13238):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 13237):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 13236):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 13235):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 13234):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 13233):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 13232):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 13231):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 13230):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 13229):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 13228):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 13227):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 13226):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 13225):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 13224):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 13223):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x000000000000000d in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00013682c in ?? ()
#4 0x00007f9f04860e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f9f04860e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001367e0 in ?? ()
#9 0x00007f9f2e999770 in ?? ()
#10 0x00007f9f04860e70 in ?? ()
#11 0x00000ff3e090c1d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 13222):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 13221):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 13220):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 13219):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 13218):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 13217):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 13216):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 13215):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 13214):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 13213):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 13212):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 13211):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 13210):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 13209):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 13208):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 13207):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 13206):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 13205):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 13204):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 13203):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x00000000000007f3 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007f9f0ea48e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f9f0ea48e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007f9f2e999770 in ?? ()
#10 0x00007f9f0ea48e70 in ?? ()
#11 0x00000ff3e1d491d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 13202):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 13201):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 13200):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000a7e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000d0088 in ?? ()
#4 0x00007f9f10285e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f9f10285e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 32 (LWP 13199):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 13198):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000a44 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000c67c8 in ?? ()
#4 0x00007f9f112b3e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f9f112b3e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 30 (LWP 13197):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 13196):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 13195):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 13194):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 13193):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 13192):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 13191):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 13190):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 13189):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 13188):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 13187):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 13186):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 13185):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 13184):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 13183):
#0 0x00007f9f2e999fb9 in ?? ()
#1 0x00007f9f18ac2c80 in ?? ()
#2 0x000000000000000d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061300001ba60 in ?? ()
#5 0x00007f9f18ac2c70 in ?? ()
#6 0x000000000000001a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 15 (LWP 13182):
#0 0x00007f9f2e999fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 13180):
#0 0x00007f9f2e999fb9 in ?? ()
#1 0x00007f9f1a2c51c0 in ?? ()
#2 0x000000000000000d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061800000f9a8 in ?? ()
#5 0x00007f9f1a2c51b0 in ?? ()
#6 0x000000000000001a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 13179):
#0 0x00007f9f2e999ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 13178):
#0 0x00007f9f27703a47 in ?? ()
#1 0x00007f9f1b2c82e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007f9f1b2c82d0 in ?? ()
#4 0x00007f9f1b2c84e0 in ?? ()
#5 0x00007f9f1b2c8320 in ?? ()
#6 0x0000614000021898 in ?? ()
#7 0x00007f9f1b2c83a0 in ?? ()
#8 0x00007f9f29d6325d in ?? ()
#9 0x3fb226751af08000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x000000006c000000 in ?? ()
#14 0x00007f9f00000000 in ?? ()
#15 0x41d9e134cd666729 in ?? ()
#16 0x00000ff463651074 in ?? ()
#17 0x00007f9f1b2c8380 in ?? ()
#18 0x00007f9f29d67ba3 in ?? ()
#19 0x00007f9f1b2c8350 in ?? ()
#20 0x3fb226751af08000 in ?? ()
#21 0x000000001b2c83a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000021898 in ?? ()
#24 0x3fb226751af08000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 13177):
#0 0x00007f9f27703a47 in ?? ()
#1 0x00007f9f1bac92e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007f9f1bac92d0 in ?? ()
#4 0x00007f9f1bac94e0 in ?? ()
#5 0x00007f9f1bac9320 in ?? ()
#6 0x0000614000021698 in ?? ()
#7 0x00007f9f1bac93a0 in ?? ()
#8 0x00007f9f29d6325d in ?? ()
#9 0x3fb332ba2b4e4000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x0000000026000000 in ?? ()
#14 0x00007f9f00000000 in ?? ()
#15 0x41d9e134cd666726 in ?? ()
#16 0x00000ff463751274 in ?? ()
#17 0x00007f9f1bac9380 in ?? ()
#18 0x00007f9f29d67ba3 in ?? ()
#19 0x00007f9f1bac9350 in ?? ()
#20 0x3fb332ba2b4e4000 in ?? ()
#21 0x000000001bac93a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000021698 in ?? ()
#24 0x3fb332ba2b4e4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 13176):
#0 0x00007f9f27703a47 in ?? ()
#1 0x00007f9f1c2ca2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007f9f1c2ca2d0 in ?? ()
#4 0x00007f9f1c2ca4e0 in ?? ()
#5 0x00007f9f1c2ca320 in ?? ()
#6 0x0000614000021498 in ?? ()
#7 0x00007f9f1c2ca3a0 in ?? ()
#8 0x00007f9f29d6325d in ?? ()
#9 0x3fb34fa2aab0a000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000023000000 in ?? ()
#14 0x00007f9f00000000 in ?? ()
#15 0x41d9e134cd666725 in ?? ()
#16 0x00000ff463851474 in ?? ()
#17 0x00007f9f1c2ca380 in ?? ()
#18 0x00007f9f29d67ba3 in ?? ()
#19 0x00007f9f1c2ca350 in ?? ()
#20 0x3fb34fa2aab0a000 in ?? ()
#21 0x000000001c2ca3a0 in ?? ()
#22 0x000061a00000e480 in ?? ()
#23 0x0000614000021498 in ?? ()
#24 0x3fb34fa2aab0a000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 13175):
#0 0x00007f9f27703a47 in ?? ()
#1 0x00007f9f1deb32e0 in ?? ()
#2 0x000061a00000de80 in ?? ()
#3 0x00007f9f1deb32d0 in ?? ()
#4 0x00007f9f1deb34e0 in ?? ()
#5 0x00007f9f1deb3320 in ?? ()
#6 0x0000614000021298 in ?? ()
#7 0x00007f9f1deb33a0 in ?? ()
#8 0x00007f9f29d6325d in ?? ()
#9 0x3fb9772859f68000 in ?? ()
#10 0x000061a00000de80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000de80 in ?? ()
#13 0x00000000d4000000 in ?? ()
#14 0x00007f9f00000000 in ?? ()
#15 0x41d9e134cd666727 in ?? ()
#16 0x00000ff463bce674 in ?? ()
#17 0x00007f9f1deb3380 in ?? ()
#18 0x00007f9f29d67ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 13172):
#0 0x00007f9f276f6cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f9f1feb7788 in ?? ()
#3 0x0000602000017cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 13171):
#0 0x00007f9f2e999fb9 in ?? ()
#1 0x00007f9f1f6b6080 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000e0c0 in ?? ()
#5 0x00007f9f1f6b6070 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 6 (LWP 13170):
#0 0x00007f9f2e99d9e2 in ?? ()
#1 0x00007f9f1e6b2b60 in ?? ()
#2 0x00007f9f1e6b2c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f9f1e6b2bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007f9f1e6b2c10 in ?? ()
#8 0x0000611000088bd0 in ?? ()
#9 0x00007f9f1e6b2c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007f9f342c1f0c in ?? ()
#12 0x00007f9f342b17b9 in ?? ()
#13 0x00007f9f342b5da6 in ?? ()
#14 0x00007f9f342ba7e0 in ?? ()
#15 0x00007f9f342ba609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007f9f2b5b00be in ?? ()
#18 0x00007f9f2e9936db in ?? ()
#19 0x00000ff463cce47c in ?? ()
#20 0x00007f9f1e6b2400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 13164):
#0 0x00007f9f2e999fb9 in ?? ()
#1 0x00007f9f20eb8c40 in ?? ()
#2 0x0000000000000299 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007f9f20eb8c30 in ?? ()
#6 0x0000000000000532 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 13163):
#0 0x00007f9f2e999fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 13162):
#0 0x00007f9f2e999fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 13161):
#0 0x00007f9f2e999fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f9f226bc410 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 13157):
#0 0x00007f9f2e99dd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250113 08:57:36.253259 11762 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskqePfU0/build/asan/bin/kudu pid 12750
I20250113 08:57:36.265281 12879 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:57:36.655040 14379 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P 4a1af392ea8543f4827f17fdff2eaaac [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b7d3e63990b74f76afc0e6cd5afbc13b)
I20250113 08:57:36.655100 11762 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskqePfU0/build/asan/bin/kudu with pid 12750
W20250113 08:57:36.720124 13176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.124.129:40671 (error 108) [suppressed 22 similar messages]
W20250113 08:57:36.720136 11788 connection.cc:537] server connection from 127.11.124.129:33037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 08:57:36.720348 11762 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskqePfU0/build/asan/bin/kudu pid 12490
W20250113 08:57:36.721895 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.129:40671: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:36.727556 12619 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:57:36.857241 11762 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskqePfU0/build/asan/bin/kudu with pid 12490
I20250113 08:57:36.882244 11762 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskqePfU0/build/asan/bin/kudu pid 13768
I20250113 08:57:36.888633 13894 generic_service.cc:196] Checking for leaks (request via RPC)
W20250113 08:57:37.166548 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer 4a1af392ea8543f4827f17fdff2eaaac (127.11.124.129:40671): Couldn't send request to peer 4a1af392ea8543f4827f17fdff2eaaac. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.129:40671: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250113 08:57:37.324384 14389 raft_consensus.cc:491] T 3034bef68d124e778f0b57a39a9b152c P aa0b5c78b11d4d07bb963ae1d9a42b3d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b7d3e63990b74f76afc0e6cd5afbc13b)
I20250113 08:57:37.324484 11762 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskqePfU0/build/asan/bin/kudu with pid 13768
W20250113 08:57:37.377858 13176 connection.cc:537] client connection to 127.11.124.131:39891 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 08:57:37.378226 11762 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskqePfU0/build/asan/bin/kudu pid 13157
W20250113 08:57:37.379598 13176 consensus_peers.cc:487] T 3034bef68d124e778f0b57a39a9b152c P b7d3e63990b74f76afc0e6cd5afbc13b -> Peer aa0b5c78b11d4d07bb963ae1d9a42b3d (127.11.124.131:39891): Couldn't send request to peer aa0b5c78b11d4d07bb963ae1d9a42b3d. Status: Network error: Client connection negotiation failed: client connection to 127.11.124.131:39891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 08:57:37.384819 13282 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:57:37.858392 11762 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskqePfU0/build/asan/bin/kudu with pid 13157
I20250113 08:57:37.923197 11762 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskqePfU0/build/asan/bin/kudu pid 11770
I20250113 08:57:37.930824 11831 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 08:57:38.104507 11762 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskqePfU0/build/asan/bin/kudu with pid 11770
2025-01-13T08:57:38Z chronyd exiting
I20250113 08:57:38.154844 11762 test_util.cc:181] -----------------------------------------------
I20250113 08:57:38.154935 11762 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskqePfU0/test-tmp/maintenance_mode-itest.4.RollingRestartArgs_RollingRestartITest.TestWorkloads_0.1736758470585686-11762-0
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/0, where GetParam() = 800-byte object <01-00 00-00 01-61 73-61 04-00 00-00 BE-BE BE-BE A0-04 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-04 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-07 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 01-00 00-00 03-00 00-00 00-BE BE-BE BE-BE BE-BE> (187497 ms)
[----------] 1 test from RollingRestartArgs/RollingRestartITest (187497 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (187498 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RollingRestartArgs/RollingRestartITest.TestWorkloads/0, where GetParam() = 800-byte object <01-00 00-00 01-61 73-61 04-00 00-00 BE-BE BE-BE A0-04 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE C0-04 00-00 80-61 00-00 00-00 00-00 00-00 00-00 ... BE-BE BE-BE BE-BE BE-BE 78-07 00-00 80-61 00-00 00-00 00-00 00-00 00-00 00-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE BE-BE 00-01 BE-BE BE-BE BE-BE 01-00 00-00 03-00 00-00 00-BE BE-BE BE-BE BE-BE>
1 FAILED TEST