[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from ReplaceTabletITest
[ RUN ] ReplaceTabletITest.ReplaceTabletsWhileWriting
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250113 14:56:11.313230 28831 test_util.cc:274] Using random seed: -1523974110
2025-01-13T14:56:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-13T14:56:11Z Disabled control of system clock
I20250113 14:56:11.348377 28831 external_mini_cluster.cc:1351] Running /tmp/dist-test-task19rJWQ/build/asan/bin/kudu
/tmp/dist-test-task19rJWQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.39.254:38075
--webserver_interface=127.28.39.254
--webserver_port=0
--builtin_ntp_servers=127.28.39.212:38659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.39.254:38075 with env {}
W20250113 14:56:11.680260 28839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 14:56:11.680560 28839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 14:56:11.680703 28839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 14:56:11.688761 28839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250113 14:56:11.688876 28839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 14:56:11.688956 28839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250113 14:56:11.689034 28839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250113 14:56:11.698729 28839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.39.212:38659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.39.254:38075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.39.254:38075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.39.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 14:43:22 UTC on 24a791456cd2
build id 4159
ASAN enabled
I20250113 14:56:11.700444 28839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 14:56:11.702265 28839 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 14:56:11.715819 28845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 14:56:11.716112 28844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 14:56:11.717715 28847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 14:56:11.717829 28839 server_base.cc:1034] running on GCE node
I20250113 14:56:11.718737 28839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 14:56:11.720263 28839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 14:56:11.721496 28839 hybrid_clock.cc:648] HybridClock initialized: now 1736780171721425 us; error 49 us; skew 500 ppm
I20250113 14:56:11.721834 28839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 14:56:11.729089 28839 webserver.cc:458] Webserver started at http://127.28.39.254:46097/ using document root <none> and password file <none>
I20250113 14:56:11.729882 28839 fs_manager.cc:362] Metadata directory not provided
I20250113 14:56:11.730020 28839 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 14:56:11.730327 28839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 14:56:11.733167 28839 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/data/instance:
uuid: "05eef7a26e4b46ef8f910182694e26ba"
format_stamp: "Formatted at 2025-01-13 14:56:11 on dist-test-slave-vdd9"
I20250113 14:56:11.733970 28839 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/wal/instance:
uuid: "05eef7a26e4b46ef8f910182694e26ba"
format_stamp: "Formatted at 2025-01-13 14:56:11 on dist-test-slave-vdd9"
I20250113 14:56:11.739537 28839 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250113 14:56:11.743353 28853 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 14:56:11.744939 28839 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250113 14:56:11.745112 28839 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/data,/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/wal
uuid: "05eef7a26e4b46ef8f910182694e26ba"
format_stamp: "Formatted at 2025-01-13 14:56:11 on dist-test-slave-vdd9"
I20250113 14:56:11.745309 28839 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 14:56:11.768826 28839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 14:56:11.769755 28839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 14:56:11.770063 28839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 14:56:11.788884 28839 rpc_server.cc:307] RPC server started. Bound to: 127.28.39.254:38075
I20250113 14:56:11.788913 28904 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.39.254:38075 every 8 connection(s)
I20250113 14:56:11.790277 28839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/data/info.pb
I20250113 14:56:11.794054 28905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:11.796557 28831 external_mini_cluster.cc:1413] Started /tmp/dist-test-task19rJWQ/build/asan/bin/kudu as pid 28839
I20250113 14:56:11.796911 28831 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/master-0/wal/instance
I20250113 14:56:11.806704 28905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba: Bootstrap starting.
I20250113 14:56:11.809821 28905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:11.810869 28905 log.cc:826] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba: Log is configured to *not* fsync() on all Append() calls
I20250113 14:56:11.813644 28905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba: No bootstrap required, opened a new log
I20250113 14:56:11.820209 28905 raft_consensus.cc:357] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05eef7a26e4b46ef8f910182694e26ba" member_type: VOTER last_known_addr { host: "127.28.39.254" port: 38075 } }
I20250113 14:56:11.820636 28905 raft_consensus.cc:383] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:11.820899 28905 raft_consensus.cc:738] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05eef7a26e4b46ef8f910182694e26ba, State: Initialized, Role: FOLLOWER
I20250113 14:56:11.821715 28905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05eef7a26e4b46ef8f910182694e26ba" member_type: VOTER last_known_addr { host: "127.28.39.254" port: 38075 } }
I20250113 14:56:11.822077 28905 raft_consensus.cc:397] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250113 14:56:11.822302 28905 raft_consensus.cc:491] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250113 14:56:11.822500 28905 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:11.824822 28905 raft_consensus.cc:513] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05eef7a26e4b46ef8f910182694e26ba" member_type: VOTER last_known_addr { host: "127.28.39.254" port: 38075 } }
I20250113 14:56:11.825452 28905 leader_election.cc:304] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05eef7a26e4b46ef8f910182694e26ba; no voters:
I20250113 14:56:11.826159 28905 leader_election.cc:290] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20250113 14:56:11.826335 28910 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [term 1 FOLLOWER]: Leader election won for term 1
I20250113 14:56:11.828140 28910 raft_consensus.cc:695] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [term 1 LEADER]: Becoming Leader. State: Replica: 05eef7a26e4b46ef8f910182694e26ba, State: Running, Role: LEADER
I20250113 14:56:11.828243 28905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [sys.catalog]: configured and running, proceeding with master startup.
I20250113 14:56:11.828856 28910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05eef7a26e4b46ef8f910182694e26ba" member_type: VOTER last_known_addr { host: "127.28.39.254" port: 38075 } }
I20250113 14:56:11.833034 28911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05eef7a26e4b46ef8f910182694e26ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05eef7a26e4b46ef8f910182694e26ba" member_type: VOTER last_known_addr { host: "127.28.39.254" port: 38075 } } }
I20250113 14:56:11.833416 28911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [sys.catalog]: This master's current role is: LEADER
I20250113 14:56:11.834326 28912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05eef7a26e4b46ef8f910182694e26ba. Latest consensus state: current_term: 1 leader_uuid: "05eef7a26e4b46ef8f910182694e26ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05eef7a26e4b46ef8f910182694e26ba" member_type: VOTER last_known_addr { host: "127.28.39.254" port: 38075 } } }
I20250113 14:56:11.834678 28912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba [sys.catalog]: This master's current role is: LEADER
I20250113 14:56:11.836138 28921 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250113 14:56:11.840818 28921 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250113 14:56:11.849750 28921 catalog_manager.cc:1348] Generated new cluster ID: 412c434ad01549fc83b316388db68705
I20250113 14:56:11.849889 28921 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250113 14:56:11.868220 28921 catalog_manager.cc:1371] Generated new certificate authority record
I20250113 14:56:11.869530 28921 catalog_manager.cc:1505] Loading token signing keys...
I20250113 14:56:11.878554 28921 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba: Generated new TSK 0
I20250113 14:56:11.879403 28921 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250113 14:56:11.885660 28831 external_mini_cluster.cc:1351] Running /tmp/dist-test-task19rJWQ/build/asan/bin/kudu
/tmp/dist-test-task19rJWQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.39.193:0
--local_ip_for_outbound_sockets=127.28.39.193
--webserver_interface=127.28.39.193
--webserver_port=0
--tserver_master_addrs=127.28.39.254:38075
--builtin_ntp_servers=127.28.39.212:38659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250113 14:56:12.225829 28929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 14:56:12.226145 28929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 14:56:12.226238 28929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 14:56:12.234483 28929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 14:56:12.234707 28929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.39.193
I20250113 14:56:12.244263 28929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.39.212:38659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.39.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.39.193
--webserver_port=0
--tserver_master_addrs=127.28.39.254:38075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.39.193
--log_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 14:43:22 UTC on 24a791456cd2
build id 4159
ASAN enabled
I20250113 14:56:12.245918 28929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 14:56:12.247640 28929 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 14:56:12.259474 28934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 14:56:12.259769 28937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 14:56:12.260527 28935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 14:56:12.260938 28929 server_base.cc:1034] running on GCE node
I20250113 14:56:12.261572 28929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 14:56:12.262523 28929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 14:56:12.263747 28929 hybrid_clock.cc:648] HybridClock initialized: now 1736780172263673 us; error 56 us; skew 500 ppm
I20250113 14:56:12.264067 28929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 14:56:12.266393 28929 webserver.cc:458] Webserver started at http://127.28.39.193:33435/ using document root <none> and password file <none>
I20250113 14:56:12.267133 28929 fs_manager.cc:362] Metadata directory not provided
I20250113 14:56:12.267257 28929 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 14:56:12.267560 28929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 14:56:12.270174 28929 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/data/instance:
uuid: "43263e8341a8434891e23c68568253de"
format_stamp: "Formatted at 2025-01-13 14:56:12 on dist-test-slave-vdd9"
I20250113 14:56:12.270936 28929 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/wal/instance:
uuid: "43263e8341a8434891e23c68568253de"
format_stamp: "Formatted at 2025-01-13 14:56:12 on dist-test-slave-vdd9"
I20250113 14:56:12.276135 28929 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20250113 14:56:12.279395 28943 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 14:56:12.280711 28929 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250113 14:56:12.280879 28929 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/data,/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/wal
uuid: "43263e8341a8434891e23c68568253de"
format_stamp: "Formatted at 2025-01-13 14:56:12 on dist-test-slave-vdd9"
I20250113 14:56:12.281059 28929 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 14:56:12.308862 28929 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 14:56:12.309856 28929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 14:56:12.310163 28929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 14:56:12.311395 28929 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 14:56:12.313583 28929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 14:56:12.313683 28929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 14:56:12.313817 28929 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 14:56:12.313884 28929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 14:56:12.346585 28929 rpc_server.cc:307] RPC server started. Bound to: 127.28.39.193:38947
I20250113 14:56:12.346642 29055 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.39.193:38947 every 8 connection(s)
I20250113 14:56:12.347903 28929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/data/info.pb
I20250113 14:56:12.354108 28831 external_mini_cluster.cc:1413] Started /tmp/dist-test-task19rJWQ/build/asan/bin/kudu as pid 28929
I20250113 14:56:12.354334 28831 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/wal/instance
I20250113 14:56:12.358592 28831 external_mini_cluster.cc:1351] Running /tmp/dist-test-task19rJWQ/build/asan/bin/kudu
/tmp/dist-test-task19rJWQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.39.194:0
--local_ip_for_outbound_sockets=127.28.39.194
--webserver_interface=127.28.39.194
--webserver_port=0
--tserver_master_addrs=127.28.39.254:38075
--builtin_ntp_servers=127.28.39.212:38659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250113 14:56:12.362740 29056 heartbeater.cc:346] Connected to a master server at 127.28.39.254:38075
I20250113 14:56:12.363133 29056 heartbeater.cc:463] Registering TS with master...
I20250113 14:56:12.364154 29056 heartbeater.cc:510] Master 127.28.39.254:38075 requested a full tablet report, sending...
I20250113 14:56:12.366261 28870 ts_manager.cc:194] Registered new tserver with Master: 43263e8341a8434891e23c68568253de (127.28.39.193:38947)
I20250113 14:56:12.367787 28870 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.39.193:55899
W20250113 14:56:12.687762 29060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 14:56:12.688100 29060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 14:56:12.688284 29060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 14:56:12.698691 29060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 14:56:12.698994 29060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.39.194
I20250113 14:56:12.715649 29060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.39.212:38659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.39.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.39.194
--webserver_port=0
--tserver_master_addrs=127.28.39.254:38075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.39.194
--log_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 14:43:22 UTC on 24a791456cd2
build id 4159
ASAN enabled
I20250113 14:56:12.718044 29060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 14:56:12.720572 29060 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 14:56:12.735527 29065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 14:56:12.736311 29068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 14:56:12.738751 29066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 14:56:12.739554 29060 server_base.cc:1034] running on GCE node
I20250113 14:56:12.740233 29060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 14:56:12.741338 29060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 14:56:12.742584 29060 hybrid_clock.cc:648] HybridClock initialized: now 1736780172742506 us; error 42 us; skew 500 ppm
I20250113 14:56:12.742903 29060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 14:56:12.745637 29060 webserver.cc:458] Webserver started at http://127.28.39.194:41855/ using document root <none> and password file <none>
I20250113 14:56:12.746397 29060 fs_manager.cc:362] Metadata directory not provided
I20250113 14:56:12.746503 29060 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 14:56:12.746767 29060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 14:56:12.749543 29060 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/data/instance:
uuid: "f7ec7575f7d64455a4ecaa314f439592"
format_stamp: "Formatted at 2025-01-13 14:56:12 on dist-test-slave-vdd9"
I20250113 14:56:12.750344 29060 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/wal/instance:
uuid: "f7ec7575f7d64455a4ecaa314f439592"
format_stamp: "Formatted at 2025-01-13 14:56:12 on dist-test-slave-vdd9"
I20250113 14:56:12.755954 29060 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20250113 14:56:12.759644 29074 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 14:56:12.761404 29060 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250113 14:56:12.761601 29060 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/data,/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/wal
uuid: "f7ec7575f7d64455a4ecaa314f439592"
format_stamp: "Formatted at 2025-01-13 14:56:12 on dist-test-slave-vdd9"
I20250113 14:56:12.761816 29060 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 14:56:12.776032 29060 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 14:56:12.776974 29060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 14:56:12.777242 29060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 14:56:12.778230 29060 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 14:56:12.779917 29060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 14:56:12.779989 29060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 14:56:12.780076 29060 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 14:56:12.780119 29060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 14:56:12.812022 29060 rpc_server.cc:307] RPC server started. Bound to: 127.28.39.194:44385
I20250113 14:56:12.812044 29186 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.39.194:44385 every 8 connection(s)
I20250113 14:56:12.813727 29060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/data/info.pb
I20250113 14:56:12.817230 28831 external_mini_cluster.cc:1413] Started /tmp/dist-test-task19rJWQ/build/asan/bin/kudu as pid 29060
I20250113 14:56:12.817417 28831 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/wal/instance
I20250113 14:56:12.820648 28831 external_mini_cluster.cc:1351] Running /tmp/dist-test-task19rJWQ/build/asan/bin/kudu
/tmp/dist-test-task19rJWQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.39.195:0
--local_ip_for_outbound_sockets=127.28.39.195
--webserver_interface=127.28.39.195
--webserver_port=0
--tserver_master_addrs=127.28.39.254:38075
--builtin_ntp_servers=127.28.39.212:38659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250113 14:56:12.828161 29187 heartbeater.cc:346] Connected to a master server at 127.28.39.254:38075
I20250113 14:56:12.828509 29187 heartbeater.cc:463] Registering TS with master...
I20250113 14:56:12.829597 29187 heartbeater.cc:510] Master 127.28.39.254:38075 requested a full tablet report, sending...
I20250113 14:56:12.831424 28870 ts_manager.cc:194] Registered new tserver with Master: f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194:44385)
I20250113 14:56:12.832239 28870 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.39.194:47591
W20250113 14:56:13.155565 29190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250113 14:56:13.155892 29190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250113 14:56:13.156144 29190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250113 14:56:13.164254 29190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250113 14:56:13.164517 29190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.39.195
I20250113 14:56:13.174106 29190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.39.212:38659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.39.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.39.195
--webserver_port=0
--tserver_master_addrs=127.28.39.254:38075
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.39.195
--log_dir=/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 13 Jan 2025 14:43:22 UTC on 24a791456cd2
build id 4159
ASAN enabled
I20250113 14:56:13.175735 29190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250113 14:56:13.177515 29190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250113 14:56:13.189581 29195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 14:56:13.189643 29196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250113 14:56:13.190363 29198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250113 14:56:13.191709 29190 server_base.cc:1034] running on GCE node
I20250113 14:56:13.192348 29190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250113 14:56:13.193365 29190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250113 14:56:13.194615 29190 hybrid_clock.cc:648] HybridClock initialized: now 1736780173194547 us; error 42 us; skew 500 ppm
I20250113 14:56:13.194954 29190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250113 14:56:13.197476 29190 webserver.cc:458] Webserver started at http://127.28.39.195:41291/ using document root <none> and password file <none>
I20250113 14:56:13.198238 29190 fs_manager.cc:362] Metadata directory not provided
I20250113 14:56:13.198369 29190 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250113 14:56:13.198662 29190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250113 14:56:13.201520 29190 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/data/instance:
uuid: "12d6ff88cee74893a7e808a1c5b8bba2"
format_stamp: "Formatted at 2025-01-13 14:56:13 on dist-test-slave-vdd9"
I20250113 14:56:13.202433 29190 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/wal/instance:
uuid: "12d6ff88cee74893a7e808a1c5b8bba2"
format_stamp: "Formatted at 2025-01-13 14:56:13 on dist-test-slave-vdd9"
I20250113 14:56:13.208190 29190 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250113 14:56:13.211820 29204 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250113 14:56:13.213243 29190 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250113 14:56:13.213434 29190 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/data,/tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/wal
uuid: "12d6ff88cee74893a7e808a1c5b8bba2"
format_stamp: "Formatted at 2025-01-13 14:56:13 on dist-test-slave-vdd9"
I20250113 14:56:13.213655 29190 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250113 14:56:13.232141 29190 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250113 14:56:13.233033 29190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250113 14:56:13.233318 29190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250113 14:56:13.234283 29190 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250113 14:56:13.235813 29190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250113 14:56:13.235890 29190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250113 14:56:13.236021 29190 ts_tablet_manager.cc:610] Registered 0 tablets
I20250113 14:56:13.236078 29190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250113 14:56:13.267946 29190 rpc_server.cc:307] RPC server started. Bound to: 127.28.39.195:39003
I20250113 14:56:13.268016 29316 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.39.195:39003 every 8 connection(s)
I20250113 14:56:13.269330 29190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/data/info.pb
I20250113 14:56:13.270226 28831 external_mini_cluster.cc:1413] Started /tmp/dist-test-task19rJWQ/build/asan/bin/kudu as pid 29190
I20250113 14:56:13.270407 28831 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/wal/instance
I20250113 14:56:13.282866 29317 heartbeater.cc:346] Connected to a master server at 127.28.39.254:38075
I20250113 14:56:13.283179 29317 heartbeater.cc:463] Registering TS with master...
I20250113 14:56:13.283941 29317 heartbeater.cc:510] Master 127.28.39.254:38075 requested a full tablet report, sending...
I20250113 14:56:13.285516 28870 ts_manager.cc:194] Registered new tserver with Master: 12d6ff88cee74893a7e808a1c5b8bba2 (127.28.39.195:39003)
I20250113 14:56:13.286705 28870 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.39.195:44135
I20250113 14:56:13.288220 28831 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250113 14:56:13.318424 28831 test_util.cc:274] Using random seed: -1521968891
I20250113 14:56:13.340479 28870 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54934:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250113 14:56:13.342474 28870 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250113 14:56:13.370784 29056 heartbeater.cc:502] Master 127.28.39.254:38075 was elected leader, sending a full tablet report...
I20250113 14:56:13.391492 29251 tablet_service.cc:1467] Processing CreateTablet for tablet b4523203195c4eaa84328c755fae8d0f (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250113 14:56:13.391937 28988 tablet_service.cc:1467] Processing CreateTablet for tablet 596da1f3892f4daf9a4f308ac6f21949 (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250113 14:56:13.393376 28989 tablet_service.cc:1467] Processing CreateTablet for tablet 45bb4becce044b429b4235b7062b6e3e (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250113 14:56:13.393675 29251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4523203195c4eaa84328c755fae8d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:13.393673 28988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 596da1f3892f4daf9a4f308ac6f21949. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:13.394156 28989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45bb4becce044b429b4235b7062b6e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:13.394536 29249 tablet_service.cc:1467] Processing CreateTablet for tablet 596da1f3892f4daf9a4f308ac6f21949 (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250113 14:56:13.393900 29119 tablet_service.cc:1467] Processing CreateTablet for tablet 596da1f3892f4daf9a4f308ac6f21949 (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250113 14:56:13.396570 28990 tablet_service.cc:1467] Processing CreateTablet for tablet b4523203195c4eaa84328c755fae8d0f (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250113 14:56:13.397128 28990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4523203195c4eaa84328c755fae8d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:13.397061 29250 tablet_service.cc:1467] Processing CreateTablet for tablet 45bb4becce044b429b4235b7062b6e3e (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250113 14:56:13.397760 29249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 596da1f3892f4daf9a4f308ac6f21949. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:13.397874 28991 tablet_service.cc:1467] Processing CreateTablet for tablet 39bd09a8e75e4f41ac24b72633514cb5 (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250113 14:56:13.398329 28991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39bd09a8e75e4f41ac24b72633514cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:13.398806 29119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 596da1f3892f4daf9a4f308ac6f21949. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:13.399251 29252 tablet_service.cc:1467] Processing CreateTablet for tablet 39bd09a8e75e4f41ac24b72633514cb5 (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250113 14:56:13.399890 29252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39bd09a8e75e4f41ac24b72633514cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:13.395328 29120 tablet_service.cc:1467] Processing CreateTablet for tablet 45bb4becce044b429b4235b7062b6e3e (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250113 14:56:13.400938 29250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45bb4becce044b429b4235b7062b6e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:13.401324 29120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45bb4becce044b429b4235b7062b6e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:13.395872 29121 tablet_service.cc:1467] Processing CreateTablet for tablet b4523203195c4eaa84328c755fae8d0f (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250113 14:56:13.405148 29121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4523203195c4eaa84328c755fae8d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:13.396267 29122 tablet_service.cc:1467] Processing CreateTablet for tablet 39bd09a8e75e4f41ac24b72633514cb5 (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250113 14:56:13.406157 29122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39bd09a8e75e4f41ac24b72633514cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:13.431682 29345 tablet_bootstrap.cc:492] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2: Bootstrap starting.
I20250113 14:56:13.435618 29345 tablet_bootstrap.cc:654] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:13.437026 29345 log.cc:826] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2: Log is configured to *not* fsync() on all Append() calls
I20250113 14:56:13.437381 29347 tablet_bootstrap.cc:492] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592: Bootstrap starting.
I20250113 14:56:13.440331 29346 tablet_bootstrap.cc:492] T 596da1f3892f4daf9a4f308ac6f21949 P 43263e8341a8434891e23c68568253de: Bootstrap starting.
I20250113 14:56:13.441242 29345 tablet_bootstrap.cc:492] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2: No bootstrap required, opened a new log
I20250113 14:56:13.441457 29347 tablet_bootstrap.cc:654] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:13.441540 29345 ts_tablet_manager.cc:1397] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20250113 14:56:13.442484 29347 log.cc:826] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592: Log is configured to *not* fsync() on all Append() calls
I20250113 14:56:13.443913 29346 tablet_bootstrap.cc:654] T 596da1f3892f4daf9a4f308ac6f21949 P 43263e8341a8434891e23c68568253de: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:13.444846 29347 tablet_bootstrap.cc:492] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592: No bootstrap required, opened a new log
I20250113 14:56:13.445134 29347 ts_tablet_manager.cc:1397] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20250113 14:56:13.445235 29346 log.cc:826] T 596da1f3892f4daf9a4f308ac6f21949 P 43263e8341a8434891e23c68568253de: Log is configured to *not* fsync() on all Append() calls
I20250113 14:56:13.451411 29346 tablet_bootstrap.cc:492] T 596da1f3892f4daf9a4f308ac6f21949 P 43263e8341a8434891e23c68568253de: No bootstrap required, opened a new log
I20250113 14:56:13.451706 29346 ts_tablet_manager.cc:1397] T 596da1f3892f4daf9a4f308ac6f21949 P 43263e8341a8434891e23c68568253de: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20250113 14:56:13.451375 29345 raft_consensus.cc:357] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.451864 29345 raft_consensus.cc:383] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:13.451977 29345 raft_consensus.cc:738] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d6ff88cee74893a7e808a1c5b8bba2, State: Initialized, Role: FOLLOWER
I20250113 14:56:13.451851 29347 raft_consensus.cc:357] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } } peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } }
I20250113 14:56:13.452251 29347 raft_consensus.cc:383] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:13.452347 29347 raft_consensus.cc:738] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ec7575f7d64455a4ecaa314f439592, State: Initialized, Role: FOLLOWER
I20250113 14:56:13.452824 29345 consensus_queue.cc:260] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.452916 29347 consensus_queue.cc:260] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } } peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } }
I20250113 14:56:13.454035 29187 heartbeater.cc:502] Master 127.28.39.254:38075 was elected leader, sending a full tablet report...
I20250113 14:56:13.454505 29317 heartbeater.cc:502] Master 127.28.39.254:38075 was elected leader, sending a full tablet report...
I20250113 14:56:13.454603 29347 ts_tablet_manager.cc:1428] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20250113 14:56:13.454933 29347 tablet_bootstrap.cc:492] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592: Bootstrap starting.
I20250113 14:56:13.456661 29347 tablet_bootstrap.cc:654] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:13.456779 29345 ts_tablet_manager.cc:1428] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20250113 14:56:13.457190 29345 tablet_bootstrap.cc:492] T b4523203195c4eaa84328c755fae8d0f P 12d6ff88cee74893a7e808a1c5b8bba2: Bootstrap starting.
I20250113 14:56:13.459065 29345 tablet_bootstrap.cc:654] T b4523203195c4eaa84328c755fae8d0f P 12d6ff88cee74893a7e808a1c5b8bba2: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:13.459194 29347 tablet_bootstrap.cc:492] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592: No bootstrap required, opened a new log
I20250113 14:56:13.459384 29347 ts_tablet_manager.cc:1397] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250113 14:56:13.460392 29347 raft_consensus.cc:357] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.460644 29347 raft_consensus.cc:383] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:13.460800 29347 raft_consensus.cc:738] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ec7575f7d64455a4ecaa314f439592, State: Initialized, Role: FOLLOWER
I20250113 14:56:13.461088 29347 consensus_queue.cc:260] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.461753 29347 ts_tablet_manager.cc:1428] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 14:56:13.462065 29347 tablet_bootstrap.cc:492] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592: Bootstrap starting.
I20250113 14:56:13.463938 29347 tablet_bootstrap.cc:654] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:13.465740 29346 raft_consensus.cc:357] T 596da1f3892f4daf9a4f308ac6f21949 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.466445 29346 raft_consensus.cc:383] T 596da1f3892f4daf9a4f308ac6f21949 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:13.466621 29346 raft_consensus.cc:738] T 596da1f3892f4daf9a4f308ac6f21949 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43263e8341a8434891e23c68568253de, State: Initialized, Role: FOLLOWER
I20250113 14:56:13.467267 29345 tablet_bootstrap.cc:492] T b4523203195c4eaa84328c755fae8d0f P 12d6ff88cee74893a7e808a1c5b8bba2: No bootstrap required, opened a new log
I20250113 14:56:13.467434 29345 ts_tablet_manager.cc:1397] T b4523203195c4eaa84328c755fae8d0f P 12d6ff88cee74893a7e808a1c5b8bba2: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20250113 14:56:13.467832 29346 consensus_queue.cc:260] T 596da1f3892f4daf9a4f308ac6f21949 P 43263e8341a8434891e23c68568253de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.468495 29345 raft_consensus.cc:357] T b4523203195c4eaa84328c755fae8d0f P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.468736 29345 raft_consensus.cc:383] T b4523203195c4eaa84328c755fae8d0f P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:13.468854 29345 raft_consensus.cc:738] T b4523203195c4eaa84328c755fae8d0f P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d6ff88cee74893a7e808a1c5b8bba2, State: Initialized, Role: FOLLOWER
I20250113 14:56:13.469166 29345 consensus_queue.cc:260] T b4523203195c4eaa84328c755fae8d0f P 12d6ff88cee74893a7e808a1c5b8bba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.469863 29345 ts_tablet_manager.cc:1428] T b4523203195c4eaa84328c755fae8d0f P 12d6ff88cee74893a7e808a1c5b8bba2: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250113 14:56:13.470149 29345 tablet_bootstrap.cc:492] T 39bd09a8e75e4f41ac24b72633514cb5 P 12d6ff88cee74893a7e808a1c5b8bba2: Bootstrap starting.
I20250113 14:56:13.472002 29345 tablet_bootstrap.cc:654] T 39bd09a8e75e4f41ac24b72633514cb5 P 12d6ff88cee74893a7e808a1c5b8bba2: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:13.474066 29347 tablet_bootstrap.cc:492] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592: No bootstrap required, opened a new log
I20250113 14:56:13.474185 29345 tablet_bootstrap.cc:492] T 39bd09a8e75e4f41ac24b72633514cb5 P 12d6ff88cee74893a7e808a1c5b8bba2: No bootstrap required, opened a new log
I20250113 14:56:13.474262 29347 ts_tablet_manager.cc:1397] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20250113 14:56:13.474341 29345 ts_tablet_manager.cc:1397] T 39bd09a8e75e4f41ac24b72633514cb5 P 12d6ff88cee74893a7e808a1c5b8bba2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250113 14:56:13.474556 29346 ts_tablet_manager.cc:1428] T 596da1f3892f4daf9a4f308ac6f21949 P 43263e8341a8434891e23c68568253de: Time spent starting tablet: real 0.023s user 0.015s sys 0.004s
I20250113 14:56:13.475068 29345 raft_consensus.cc:357] T 39bd09a8e75e4f41ac24b72633514cb5 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } } peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } }
I20250113 14:56:13.475077 29347 raft_consensus.cc:357] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.475308 29345 raft_consensus.cc:383] T 39bd09a8e75e4f41ac24b72633514cb5 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:13.475308 29347 raft_consensus.cc:383] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:13.475391 29345 raft_consensus.cc:738] T 39bd09a8e75e4f41ac24b72633514cb5 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d6ff88cee74893a7e808a1c5b8bba2, State: Initialized, Role: FOLLOWER
I20250113 14:56:13.475391 29347 raft_consensus.cc:738] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ec7575f7d64455a4ecaa314f439592, State: Initialized, Role: FOLLOWER
I20250113 14:56:13.475425 29346 tablet_bootstrap.cc:492] T b4523203195c4eaa84328c755fae8d0f P 43263e8341a8434891e23c68568253de: Bootstrap starting.
I20250113 14:56:13.475648 29347 consensus_queue.cc:260] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.475685 29345 consensus_queue.cc:260] T 39bd09a8e75e4f41ac24b72633514cb5 P 12d6ff88cee74893a7e808a1c5b8bba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } } peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } }
I20250113 14:56:13.476213 29347 ts_tablet_manager.cc:1428] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 14:56:13.476614 29347 tablet_bootstrap.cc:492] T 596da1f3892f4daf9a4f308ac6f21949 P f7ec7575f7d64455a4ecaa314f439592: Bootstrap starting.
I20250113 14:56:13.476910 29345 ts_tablet_manager.cc:1428] T 39bd09a8e75e4f41ac24b72633514cb5 P 12d6ff88cee74893a7e808a1c5b8bba2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250113 14:56:13.477226 29345 tablet_bootstrap.cc:492] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2: Bootstrap starting.
I20250113 14:56:13.479038 29345 tablet_bootstrap.cc:654] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:13.479879 29346 tablet_bootstrap.cc:654] T b4523203195c4eaa84328c755fae8d0f P 43263e8341a8434891e23c68568253de: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:13.481681 29347 tablet_bootstrap.cc:654] T 596da1f3892f4daf9a4f308ac6f21949 P f7ec7575f7d64455a4ecaa314f439592: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:13.481773 29352 raft_consensus.cc:491] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 14:56:13.482106 29352 raft_consensus.cc:513] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.488186 29345 tablet_bootstrap.cc:492] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2: No bootstrap required, opened a new log
I20250113 14:56:13.488453 29345 ts_tablet_manager.cc:1397] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20250113 14:56:13.489352 29345 raft_consensus.cc:357] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.489567 29345 raft_consensus.cc:383] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:13.489661 29345 raft_consensus.cc:738] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d6ff88cee74893a7e808a1c5b8bba2, State: Initialized, Role: FOLLOWER
I20250113 14:56:13.489929 29345 consensus_queue.cc:260] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.490602 29345 ts_tablet_manager.cc:1428] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250113 14:56:13.492957 29352 leader_election.cc:290] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43263e8341a8434891e23c68568253de (127.28.39.193:38947), 12d6ff88cee74893a7e808a1c5b8bba2 (127.28.39.195:39003)
I20250113 14:56:13.495924 29347 tablet_bootstrap.cc:492] T 596da1f3892f4daf9a4f308ac6f21949 P f7ec7575f7d64455a4ecaa314f439592: No bootstrap required, opened a new log
I20250113 14:56:13.496121 29346 tablet_bootstrap.cc:492] T b4523203195c4eaa84328c755fae8d0f P 43263e8341a8434891e23c68568253de: No bootstrap required, opened a new log
I20250113 14:56:13.496136 29347 ts_tablet_manager.cc:1397] T 596da1f3892f4daf9a4f308ac6f21949 P f7ec7575f7d64455a4ecaa314f439592: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.009s
I20250113 14:56:13.496292 29346 ts_tablet_manager.cc:1397] T b4523203195c4eaa84328c755fae8d0f P 43263e8341a8434891e23c68568253de: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.012s
I20250113 14:56:13.497102 29347 raft_consensus.cc:357] T 596da1f3892f4daf9a4f308ac6f21949 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.497323 29347 raft_consensus.cc:383] T 596da1f3892f4daf9a4f308ac6f21949 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:13.497421 29347 raft_consensus.cc:738] T 596da1f3892f4daf9a4f308ac6f21949 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ec7575f7d64455a4ecaa314f439592, State: Initialized, Role: FOLLOWER
I20250113 14:56:13.497434 29346 raft_consensus.cc:357] T b4523203195c4eaa84328c755fae8d0f P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.497681 29346 raft_consensus.cc:383] T b4523203195c4eaa84328c755fae8d0f P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:13.497797 29346 raft_consensus.cc:738] T b4523203195c4eaa84328c755fae8d0f P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43263e8341a8434891e23c68568253de, State: Initialized, Role: FOLLOWER
I20250113 14:56:13.497707 29347 consensus_queue.cc:260] T 596da1f3892f4daf9a4f308ac6f21949 P f7ec7575f7d64455a4ecaa314f439592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.498062 29346 consensus_queue.cc:260] T b4523203195c4eaa84328c755fae8d0f P 43263e8341a8434891e23c68568253de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.498426 29347 ts_tablet_manager.cc:1428] T 596da1f3892f4daf9a4f308ac6f21949 P f7ec7575f7d64455a4ecaa314f439592: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250113 14:56:13.498658 29346 ts_tablet_manager.cc:1428] T b4523203195c4eaa84328c755fae8d0f P 43263e8341a8434891e23c68568253de: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250113 14:56:13.498981 29346 tablet_bootstrap.cc:492] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de: Bootstrap starting.
I20250113 14:56:13.500941 29346 tablet_bootstrap.cc:654] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:13.504581 29346 tablet_bootstrap.cc:492] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de: No bootstrap required, opened a new log
I20250113 14:56:13.504554 29011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4523203195c4eaa84328c755fae8d0f" candidate_uuid: "f7ec7575f7d64455a4ecaa314f439592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43263e8341a8434891e23c68568253de" is_pre_election: true
I20250113 14:56:13.504935 29346 ts_tablet_manager.cc:1397] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20250113 14:56:13.505288 29011 raft_consensus.cc:2463] T b4523203195c4eaa84328c755fae8d0f P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ec7575f7d64455a4ecaa314f439592 in term 0.
I20250113 14:56:13.505754 29346 raft_consensus.cc:357] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } } peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } }
I20250113 14:56:13.505883 29077 leader_election.cc:304] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43263e8341a8434891e23c68568253de, f7ec7575f7d64455a4ecaa314f439592; no voters:
I20250113 14:56:13.505968 29346 raft_consensus.cc:383] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:13.506047 29346 raft_consensus.cc:738] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43263e8341a8434891e23c68568253de, State: Initialized, Role: FOLLOWER
I20250113 14:56:13.506327 29352 raft_consensus.cc:2798] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 14:56:13.506371 29346 consensus_queue.cc:260] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } } peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } }
I20250113 14:56:13.506474 29352 raft_consensus.cc:491] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 14:56:13.506587 29352 raft_consensus.cc:3054] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:13.506892 29346 ts_tablet_manager.cc:1428] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250113 14:56:13.507148 29346 tablet_bootstrap.cc:492] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de: Bootstrap starting.
I20250113 14:56:13.508384 29346 tablet_bootstrap.cc:654] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:13.508710 29352 raft_consensus.cc:513] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.509330 29352 leader_election.cc:290] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [CANDIDATE]: Term 1 election: Requested vote from peers 43263e8341a8434891e23c68568253de (127.28.39.193:38947), 12d6ff88cee74893a7e808a1c5b8bba2 (127.28.39.195:39003)
I20250113 14:56:13.509816 29011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4523203195c4eaa84328c755fae8d0f" candidate_uuid: "f7ec7575f7d64455a4ecaa314f439592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43263e8341a8434891e23c68568253de"
I20250113 14:56:13.510011 29011 raft_consensus.cc:3054] T b4523203195c4eaa84328c755fae8d0f P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:13.510588 29271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4523203195c4eaa84328c755fae8d0f" candidate_uuid: "f7ec7575f7d64455a4ecaa314f439592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d6ff88cee74893a7e808a1c5b8bba2"
I20250113 14:56:13.511165 29271 raft_consensus.cc:3054] T b4523203195c4eaa84328c755fae8d0f P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:13.511986 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4523203195c4eaa84328c755fae8d0f" candidate_uuid: "f7ec7575f7d64455a4ecaa314f439592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" is_pre_election: true
I20250113 14:56:13.512969 29346 tablet_bootstrap.cc:492] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de: No bootstrap required, opened a new log
I20250113 14:56:13.513136 29346 ts_tablet_manager.cc:1397] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20250113 14:56:13.513216 29011 raft_consensus.cc:2463] T b4523203195c4eaa84328c755fae8d0f P 43263e8341a8434891e23c68568253de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ec7575f7d64455a4ecaa314f439592 in term 1.
I20250113 14:56:13.513785 29077 leader_election.cc:304] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43263e8341a8434891e23c68568253de, f7ec7575f7d64455a4ecaa314f439592; no voters:
I20250113 14:56:13.513938 29346 raft_consensus.cc:357] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.514138 29346 raft_consensus.cc:383] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:13.514130 29352 raft_consensus.cc:2798] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 14:56:13.514237 29346 raft_consensus.cc:738] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43263e8341a8434891e23c68568253de, State: Initialized, Role: FOLLOWER
I20250113 14:56:13.514457 29346 consensus_queue.cc:260] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.514583 29271 raft_consensus.cc:2463] T b4523203195c4eaa84328c755fae8d0f P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ec7575f7d64455a4ecaa314f439592 in term 1.
I20250113 14:56:13.514999 29346 ts_tablet_manager.cc:1428] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250113 14:56:13.515492 29352 raft_consensus.cc:695] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [term 1 LEADER]: Becoming Leader. State: Replica: f7ec7575f7d64455a4ecaa314f439592, State: Running, Role: LEADER
I20250113 14:56:13.516048 29352 consensus_queue.cc:237] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.516587 29356 raft_consensus.cc:491] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 14:56:13.516747 29356 raft_consensus.cc:513] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } } peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } }
I20250113 14:56:13.518285 29356 leader_election.cc:290] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194:44385), 12d6ff88cee74893a7e808a1c5b8bba2 (127.28.39.195:39003)
W20250113 14:56:13.521045 29318 tablet.cc:2367] T 39bd09a8e75e4f41ac24b72633514cb5 P 12d6ff88cee74893a7e808a1c5b8bba2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 14:56:13.521461 28868 catalog_manager.cc:5526] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194). New cstate: current_term: 1 leader_uuid: "f7ec7575f7d64455a4ecaa314f439592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20250113 14:56:13.535341 29271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bd09a8e75e4f41ac24b72633514cb5" candidate_uuid: "43263e8341a8434891e23c68568253de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" is_pre_election: true
I20250113 14:56:13.535604 29271 raft_consensus.cc:2463] T 39bd09a8e75e4f41ac24b72633514cb5 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43263e8341a8434891e23c68568253de in term 0.
I20250113 14:56:13.535655 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bd09a8e75e4f41ac24b72633514cb5" candidate_uuid: "43263e8341a8434891e23c68568253de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ec7575f7d64455a4ecaa314f439592" is_pre_election: true
I20250113 14:56:13.536080 29142 raft_consensus.cc:2463] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43263e8341a8434891e23c68568253de in term 0.
I20250113 14:56:13.536099 28945 leader_election.cc:304] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d6ff88cee74893a7e808a1c5b8bba2, 43263e8341a8434891e23c68568253de; no voters:
I20250113 14:56:13.536541 29356 raft_consensus.cc:2798] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 14:56:13.536702 29356 raft_consensus.cc:491] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 14:56:13.536828 29356 raft_consensus.cc:3054] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:13.538787 29356 raft_consensus.cc:513] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } } peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } }
I20250113 14:56:13.539342 29356 leader_election.cc:290] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [CANDIDATE]: Term 1 election: Requested vote from peers f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194:44385), 12d6ff88cee74893a7e808a1c5b8bba2 (127.28.39.195:39003)
I20250113 14:56:13.539805 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bd09a8e75e4f41ac24b72633514cb5" candidate_uuid: "43263e8341a8434891e23c68568253de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ec7575f7d64455a4ecaa314f439592"
I20250113 14:56:13.539875 29271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39bd09a8e75e4f41ac24b72633514cb5" candidate_uuid: "43263e8341a8434891e23c68568253de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d6ff88cee74893a7e808a1c5b8bba2"
I20250113 14:56:13.540062 29142 raft_consensus.cc:3054] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:13.540194 29271 raft_consensus.cc:3054] T 39bd09a8e75e4f41ac24b72633514cb5 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:13.542152 29271 raft_consensus.cc:2463] T 39bd09a8e75e4f41ac24b72633514cb5 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43263e8341a8434891e23c68568253de in term 1.
I20250113 14:56:13.542213 29142 raft_consensus.cc:2463] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43263e8341a8434891e23c68568253de in term 1.
I20250113 14:56:13.542538 28945 leader_election.cc:304] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d6ff88cee74893a7e808a1c5b8bba2, 43263e8341a8434891e23c68568253de; no voters:
I20250113 14:56:13.542837 29356 raft_consensus.cc:2798] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [term 1 FOLLOWER]: Leader election won for term 1
I20250113 14:56:13.543263 29356 raft_consensus.cc:695] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [term 1 LEADER]: Becoming Leader. State: Replica: 43263e8341a8434891e23c68568253de, State: Running, Role: LEADER
I20250113 14:56:13.543753 29356 consensus_queue.cc:237] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } } peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } }
I20250113 14:56:13.548204 28868 catalog_manager.cc:5526] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de reported cstate change: term changed from 0 to 1, leader changed from <none> to 43263e8341a8434891e23c68568253de (127.28.39.193). New cstate: current_term: 1 leader_uuid: "43263e8341a8434891e23c68568253de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } health_report { overall_health: HEALTHY } } }
W20250113 14:56:13.565929 29188 tablet.cc:2367] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 14:56:13.574369 29351 raft_consensus.cc:491] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 14:56:13.574587 29351 raft_consensus.cc:513] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.576239 29351 leader_election.cc:290] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43263e8341a8434891e23c68568253de (127.28.39.193:38947), f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194:44385)
I20250113 14:56:13.582953 29011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596da1f3892f4daf9a4f308ac6f21949" candidate_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43263e8341a8434891e23c68568253de" is_pre_election: true
I20250113 14:56:13.583209 29011 raft_consensus.cc:2463] T 596da1f3892f4daf9a4f308ac6f21949 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d6ff88cee74893a7e808a1c5b8bba2 in term 0.
I20250113 14:56:13.583236 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596da1f3892f4daf9a4f308ac6f21949" candidate_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ec7575f7d64455a4ecaa314f439592" is_pre_election: true
I20250113 14:56:13.583488 29142 raft_consensus.cc:2463] T 596da1f3892f4daf9a4f308ac6f21949 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d6ff88cee74893a7e808a1c5b8bba2 in term 0.
I20250113 14:56:13.583711 29207 leader_election.cc:304] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d6ff88cee74893a7e808a1c5b8bba2, 43263e8341a8434891e23c68568253de; no voters:
I20250113 14:56:13.584082 29351 raft_consensus.cc:2798] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 14:56:13.584213 29351 raft_consensus.cc:491] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 14:56:13.584302 29351 raft_consensus.cc:3054] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:13.586580 29351 raft_consensus.cc:513] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.587210 29351 leader_election.cc:290] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [CANDIDATE]: Term 1 election: Requested vote from peers 43263e8341a8434891e23c68568253de (127.28.39.193:38947), f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194:44385)
I20250113 14:56:13.587662 29011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596da1f3892f4daf9a4f308ac6f21949" candidate_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43263e8341a8434891e23c68568253de"
I20250113 14:56:13.587715 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596da1f3892f4daf9a4f308ac6f21949" candidate_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ec7575f7d64455a4ecaa314f439592"
I20250113 14:56:13.587884 29011 raft_consensus.cc:3054] T 596da1f3892f4daf9a4f308ac6f21949 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:13.587949 29142 raft_consensus.cc:3054] T 596da1f3892f4daf9a4f308ac6f21949 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:13.589864 29142 raft_consensus.cc:2463] T 596da1f3892f4daf9a4f308ac6f21949 P f7ec7575f7d64455a4ecaa314f439592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d6ff88cee74893a7e808a1c5b8bba2 in term 1.
I20250113 14:56:13.589883 29011 raft_consensus.cc:2463] T 596da1f3892f4daf9a4f308ac6f21949 P 43263e8341a8434891e23c68568253de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d6ff88cee74893a7e808a1c5b8bba2 in term 1.
I20250113 14:56:13.590288 29207 leader_election.cc:304] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12d6ff88cee74893a7e808a1c5b8bba2, 43263e8341a8434891e23c68568253de, f7ec7575f7d64455a4ecaa314f439592; no voters:
I20250113 14:56:13.590615 29351 raft_consensus.cc:2798] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 14:56:13.591096 29351 raft_consensus.cc:695] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 LEADER]: Becoming Leader. State: Replica: 12d6ff88cee74893a7e808a1c5b8bba2, State: Running, Role: LEADER
I20250113 14:56:13.591553 29351 consensus_queue.cc:237] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.595992 28868 catalog_manager.cc:5526] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12d6ff88cee74893a7e808a1c5b8bba2 (127.28.39.195). New cstate: current_term: 1 leader_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } health_report { overall_health: HEALTHY } } }
W20250113 14:56:13.600332 29057 tablet.cc:2367] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250113 14:56:13.704041 29351 raft_consensus.cc:491] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 14:56:13.704274 29351 raft_consensus.cc:513] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.704962 29351 leader_election.cc:290] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43263e8341a8434891e23c68568253de (127.28.39.193:38947), f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194:44385)
I20250113 14:56:13.705531 29011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bb4becce044b429b4235b7062b6e3e" candidate_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43263e8341a8434891e23c68568253de" is_pre_election: true
I20250113 14:56:13.705550 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bb4becce044b429b4235b7062b6e3e" candidate_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ec7575f7d64455a4ecaa314f439592" is_pre_election: true
I20250113 14:56:13.705750 29011 raft_consensus.cc:2463] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d6ff88cee74893a7e808a1c5b8bba2 in term 0.
I20250113 14:56:13.705782 29142 raft_consensus.cc:2463] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d6ff88cee74893a7e808a1c5b8bba2 in term 0.
I20250113 14:56:13.706202 29207 leader_election.cc:304] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d6ff88cee74893a7e808a1c5b8bba2, 43263e8341a8434891e23c68568253de; no voters:
I20250113 14:56:13.706496 29351 raft_consensus.cc:2798] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 14:56:13.706598 29351 raft_consensus.cc:491] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 14:56:13.706712 29351 raft_consensus.cc:3054] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:13.708653 29351 raft_consensus.cc:513] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.709329 29351 leader_election.cc:290] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [CANDIDATE]: Term 1 election: Requested vote from peers 43263e8341a8434891e23c68568253de (127.28.39.193:38947), f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194:44385)
I20250113 14:56:13.709746 29011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bb4becce044b429b4235b7062b6e3e" candidate_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43263e8341a8434891e23c68568253de"
I20250113 14:56:13.709816 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45bb4becce044b429b4235b7062b6e3e" candidate_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ec7575f7d64455a4ecaa314f439592"
I20250113 14:56:13.709968 29011 raft_consensus.cc:3054] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:13.710008 29142 raft_consensus.cc:3054] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:13.712877 29142 raft_consensus.cc:2463] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d6ff88cee74893a7e808a1c5b8bba2 in term 1.
I20250113 14:56:13.712877 29011 raft_consensus.cc:2463] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d6ff88cee74893a7e808a1c5b8bba2 in term 1.
I20250113 14:56:13.713351 29207 leader_election.cc:304] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12d6ff88cee74893a7e808a1c5b8bba2, 43263e8341a8434891e23c68568253de, f7ec7575f7d64455a4ecaa314f439592; no voters:
I20250113 14:56:13.713661 29351 raft_consensus.cc:2798] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 14:56:13.713840 29351 raft_consensus.cc:695] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 LEADER]: Becoming Leader. State: Replica: 12d6ff88cee74893a7e808a1c5b8bba2, State: Running, Role: LEADER
I20250113 14:56:13.714114 29351 consensus_queue.cc:237] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:13.716759 28868 catalog_manager.cc:5526] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12d6ff88cee74893a7e808a1c5b8bba2 (127.28.39.195). New cstate: current_term: 1 leader_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } health_report { overall_health: HEALTHY } } }
I20250113 14:56:13.806005 29271 raft_consensus.cc:1270] T 39bd09a8e75e4f41ac24b72633514cb5 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 FOLLOWER]: Refusing update from remote peer 43263e8341a8434891e23c68568253de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 14:56:13.807832 29356 consensus_queue.cc:1035] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 14:56:13.808106 29142 raft_consensus.cc:1270] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592 [term 1 FOLLOWER]: Refusing update from remote peer 12d6ff88cee74893a7e808a1c5b8bba2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 14:56:13.809037 29351 consensus_queue.cc:1035] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 14:56:13.814752 29011 raft_consensus.cc:1270] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de [term 1 FOLLOWER]: Refusing update from remote peer 12d6ff88cee74893a7e808a1c5b8bba2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 14:56:13.815575 29351 consensus_queue.cc:1035] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 14:56:13.815780 29141 raft_consensus.cc:1270] T 39bd09a8e75e4f41ac24b72633514cb5 P f7ec7575f7d64455a4ecaa314f439592 [term 1 FOLLOWER]: Refusing update from remote peer 43263e8341a8434891e23c68568253de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 14:56:13.816519 29356 consensus_queue.cc:1035] T 39bd09a8e75e4f41ac24b72633514cb5 P 43263e8341a8434891e23c68568253de [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 14:56:13.829456 29140 raft_consensus.cc:1270] T 596da1f3892f4daf9a4f308ac6f21949 P f7ec7575f7d64455a4ecaa314f439592 [term 1 FOLLOWER]: Refusing update from remote peer 12d6ff88cee74893a7e808a1c5b8bba2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 14:56:13.830334 29010 raft_consensus.cc:1270] T 596da1f3892f4daf9a4f308ac6f21949 P 43263e8341a8434891e23c68568253de [term 1 FOLLOWER]: Refusing update from remote peer 12d6ff88cee74893a7e808a1c5b8bba2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 14:56:13.830669 29009 raft_consensus.cc:1270] T b4523203195c4eaa84328c755fae8d0f P 43263e8341a8434891e23c68568253de [term 1 FOLLOWER]: Refusing update from remote peer f7ec7575f7d64455a4ecaa314f439592: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 14:56:13.832583 29372 consensus_queue.cc:1035] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 14:56:13.833897 29372 consensus_queue.cc:1035] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 14:56:13.834988 29272 raft_consensus.cc:1270] T b4523203195c4eaa84328c755fae8d0f P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 FOLLOWER]: Refusing update from remote peer f7ec7575f7d64455a4ecaa314f439592: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250113 14:56:13.831602 29365 consensus_queue.cc:1035] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250113 14:56:13.837332 29365 consensus_queue.cc:1035] T b4523203195c4eaa84328c755fae8d0f P f7ec7575f7d64455a4ecaa314f439592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 14:56:13.894692 29385 mvcc.cc:204] Tried to move back new op lower bound from 7113851592020455424 to 7113851590726877184. Current Snapshot: MvccSnapshot[applied={T|T < 7113851591985737728}]
I20250113 14:56:13.899107 29387 mvcc.cc:204] Tried to move back new op lower bound from 7113851591897325568 to 7113851590839169024. Current Snapshot: MvccSnapshot[applied={T|T < 7113851591897325568 or (T in {7113851591897325568})}]
I20250113 14:56:13.892269 29384 mvcc.cc:204] Tried to move back new op lower bound from 7113851591993921536 to 7113851591035031552. Current Snapshot: MvccSnapshot[applied={T|T < 7113851591993921536}]
I20250113 14:56:16.990357 28862 master_service.cc:946] ReplaceTablet: received request to replace tablet 45bb4becce044b429b4235b7062b6e3e from {username='slave'} at 127.0.0.1:54930
I20250113 14:56:17.083333 28862 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 05eef7a26e4b46ef8f910182694e26ba: Sending DeleteTablet for 3 replicas of tablet 45bb4becce044b429b4235b7062b6e3e
I20250113 14:56:17.084240 28862 catalog_manager.cc:6596] ReplaceTablet: tablet 45bb4becce044b429b4235b7062b6e3e deleted and replaced by tablet b46a2c13dda4486892efa9475b7b3032
I20250113 14:56:17.084450 28988 tablet_service.cc:1514] Processing DeleteTablet for tablet 45bb4becce044b429b4235b7062b6e3e with delete_type TABLET_DATA_DELETED (replaced by tablet b46a2c13dda4486892efa9475b7b3032) from {username='slave'} at 127.0.0.1:44558
I20250113 14:56:17.085443 29122 tablet_service.cc:1514] Processing DeleteTablet for tablet 45bb4becce044b429b4235b7062b6e3e with delete_type TABLET_DATA_DELETED (replaced by tablet b46a2c13dda4486892efa9475b7b3032) from {username='slave'} at 127.0.0.1:36310
I20250113 14:56:17.087636 29249 tablet_service.cc:1514] Processing DeleteTablet for tablet 45bb4becce044b429b4235b7062b6e3e with delete_type TABLET_DATA_DELETED (replaced by tablet b46a2c13dda4486892efa9475b7b3032) from {username='slave'} at 127.0.0.1:52706
I20250113 14:56:17.106086 28990 tablet_service.cc:1467] Processing CreateTablet for tablet b46a2c13dda4486892efa9475b7b3032 (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250113 14:56:17.106595 28990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b46a2c13dda4486892efa9475b7b3032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:17.106623 29119 tablet_service.cc:1467] Processing CreateTablet for tablet b46a2c13dda4486892efa9475b7b3032 (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250113 14:56:17.107079 29119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b46a2c13dda4486892efa9475b7b3032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:17.108080 29252 tablet_service.cc:1467] Processing CreateTablet for tablet b46a2c13dda4486892efa9475b7b3032 (DEFAULT_TABLE table=test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250113 14:56:17.108494 29252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b46a2c13dda4486892efa9475b7b3032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250113 14:56:17.126921 29436 tablet_bootstrap.cc:492] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2: Bootstrap starting.
I20250113 14:56:17.129989 29436 tablet_bootstrap.cc:654] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:17.144731 29431 tablet_replica.cc:331] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592: stopping tablet replica
I20250113 14:56:17.146314 29431 raft_consensus.cc:2238] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250113 14:56:17.147646 29208 consensus_peers.cc:487] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 -> Peer f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194:44385): Couldn't send request to peer f7ec7575f7d64455a4ecaa314f439592. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250113 14:56:17.158561 29432 tablet_replica.cc:331] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2: stopping tablet replica
I20250113 14:56:17.159103 29432 raft_consensus.cc:2238] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 LEADER]: Raft consensus shutting down.
W20250113 14:56:17.159386 29411 consensus_queue.cc:1159] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "43263e8341a8434891e23c68568253de" responder_term: 1 status { last_received { term: 1 index: 220 } last_committed_idx: 219 last_received_current_leader { term: 1 index: 220 } }
I20250113 14:56:17.159936 29432 pending_rounds.cc:62] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2: Trying to abort 3 pending ops.
I20250113 14:56:17.160086 29432 pending_rounds.cc:69] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2: Aborting op as it isn't in flight: id { term: 1 index: 222 } timestamp: 7113851605635293184 op_type: WRITE_OP write_request { tablet_id: "45bb4becce044b429b4235b7062b6e3e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000b>3E\230\'f\020\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\326f\002We8\345V\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\223\\WI\311\037\277\362 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000]\225\336W\320\221\256]0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0007\207\'F\022\034\r\001@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000j\025\031QI\216\263\344P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\211p\257W\272\252\266y`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000d\017\322Re\004\2727p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\026P+B\232\211\003v\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0001\200\350D\307g\313c\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\270\244\261V\272\016\305\231\240\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""d82df6b1633dd5bbde3687af9c0c44f7021f33491cbffd6bd7498104b56ab4c2c671736d84a23a7929c1f8627b33feb1ef91b3f10d79573a4f6fc52565f0101b4e61b1b7dfb51b2962bd1daadbcd6d15ad5c6221985d393d" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7113851605505011712 authz_token { token_data: "\010\271\325\224\274\006\"3\n\005slave\022*\n b2b4c51b1bd6475ba8e5bb3f9e211634\020\001\030\001 \001(\001" signature: "<redacted>""\254jd\000\330\350\264\016\207,/y\307\373o\214]\355\336\000<\336\000\3552\274\000\036M\210\300\322\254\300\313Ne\350\212|V\253\306\3638\2715D\212\026\321%\375\332\340;\356b\245\376\2348\325\036" signing_key_seq_num: 0 } } request_id { client_id: "5a3392a41e264d8dbd2310f89bb498ac" seq_no: 883 first_incomplete_seq_no: 872 attempt_no: 0 }
W20250113 14:56:17.160845 29432 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52718: Aborted: Op aborted
I20250113 14:56:17.162825 29432 pending_rounds.cc:69] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2: Aborting op as it isn't in flight: id { term: 1 index: 221 } timestamp: 7113851605631623168 op_type: WRITE_OP write_request { tablet_id: "45bb4becce044b429b4235b7062b6e3e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\233.#H\310\037\275\020\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\3450\320_\342K\032\232\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\301\364\225E\tY\034\256 \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\030\261\377^\365\034d\3630\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000}\277\344E<)j\013@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\274#\303Om\037s\366P\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\300N\351_\322R\230\207`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\031tKQ\241s\355\307p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\0004\344_P\316\036\004\362\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""1750f19386482d5c0d26ce9d5d8d58536c39499d4a1ff23371e534637da4d258c2f864b1033543fade18f547711e2f8142daa61b419e7027a2643290c0677ccefa07819585993794" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7113851605505011712 authz_token { token_data: "\010\271\325\224\274\006\"3\n\005slave\022*\n b2b4c51b1bd6475ba8e5bb3f9e211634\020\001\030\001 \001(\001" signature: "<redacted>""\254jd\000\330\350\264\016\207,/y\307\373o\214]\355\336\000<\336\000\3552\274\000\036M\210\300\322\254\300\313Ne\350\212|V\253\306\3638\2715D\212\026\321%\375\332\340;\356b\245\376\2348\325\036" signing_key_seq_num: 0 } } request_id { client_id: "5a3392a41e264d8dbd2310f89bb498ac" seq_no: 882 first_incomplete_seq_no: 872 attempt_no: 0 }
W20250113 14:56:17.164317 29432 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52718: Aborted: Op aborted
W20250113 14:56:17.164566 28968 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44560: Illegal state: replica 43263e8341a8434891e23c68568253de is not leader of this config: current role FOLLOWER
I20250113 14:56:17.165750 29432 pending_rounds.cc:69] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2: Aborting op as it isn't in flight: id { term: 1 index: 220 } timestamp: 7113851605608660992 op_type: WRITE_OP write_request { tablet_id: "45bb4becce044b429b4235b7062b6e3e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\310\377\254P\243\301\234\200\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\034\353\263BC\316\322`\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\034<\347V|H$> \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\374&`W\030\r\200l0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\351N\221I\006^\037>@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\322\264AX\312\272\372KP\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\203e^QM:\023\223`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\r\021\366K\275\343^\335p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\366Y&C\337\232\365\203\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000fYwR\2655\271R\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""12e3eb81e2f51955c8f6e0f11053e5a08c346f890d0d234353ecaea3af7af4c14c026239e0fd6656121b0ed85cd720626f97d45a31ecf23e5119ac84591a2dc2231e52de37a4fdaa8b5b15ec91f8bf35" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7113851605505011712 authz_token { token_data: "\010\271\325\224\274\006\"3\n\005slave\022*\n b2b4c51b1bd6475ba8e5bb3f9e211634\020\001\030\001 \001(\001" signature: "<redacted>""\254jd\000\330\350\264\016\207,/y\307\373o\214]\355\336\000<\336\000\3552\274\000\036M\210\300\322\254\300\313Ne\350\212|V\253\306\3638\2715D\212\026\321%\375\332\340;\356b\245\376\2348\325\036" signing_key_seq_num: 0 } } request_id { client_id: "5a3392a41e264d8dbd2310f89bb498ac" seq_no: 877 first_incomplete_seq_no: 872 attempt_no: 0 }
W20250113 14:56:17.166235 29432 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52718: Aborted: Op aborted
I20250113 14:56:17.167678 29432 raft_consensus.cc:2267] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250113 14:56:17.168752 29382 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52718: Illegal state: RaftConsensus is not running: State = Stopped
W20250113 14:56:17.169555 28968 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44560: Illegal state: replica 43263e8341a8434891e23c68568253de is not leader of this config: current role FOLLOWER
I20250113 14:56:17.171902 29432 ts_tablet_manager.cc:1905] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 14:56:17.186071 29434 tablet_bootstrap.cc:492] T b46a2c13dda4486892efa9475b7b3032 P 43263e8341a8434891e23c68568253de: Bootstrap starting.
I20250113 14:56:17.188122 29432 ts_tablet_manager.cc:1918] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.222
I20250113 14:56:17.188351 29432 log.cc:1198] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2: Deleting WAL directory at /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-2/wal/wals/45bb4becce044b429b4235b7062b6e3e
I20250113 14:56:17.189011 29432 ts_tablet_manager.cc:1939] T 45bb4becce044b429b4235b7062b6e3e P 12d6ff88cee74893a7e808a1c5b8bba2: Deleting consensus metadata
I20250113 14:56:17.190474 29434 tablet_bootstrap.cc:654] T b46a2c13dda4486892efa9475b7b3032 P 43263e8341a8434891e23c68568253de: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:17.193260 28855 catalog_manager.cc:4872] TS 12d6ff88cee74893a7e808a1c5b8bba2 (127.28.39.195:39003): tablet 45bb4becce044b429b4235b7062b6e3e (table test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]) successfully deleted
I20250113 14:56:17.193452 29436 tablet_bootstrap.cc:492] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2: No bootstrap required, opened a new log
I20250113 14:56:17.193634 29436 ts_tablet_manager.cc:1397] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2: Time spent bootstrapping tablet: real 0.067s user 0.013s sys 0.026s
I20250113 14:56:17.194183 29435 tablet_bootstrap.cc:492] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592: Bootstrap starting.
I20250113 14:56:17.194747 29436 raft_consensus.cc:357] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:17.194999 29436 raft_consensus.cc:383] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:17.195093 29436 raft_consensus.cc:738] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d6ff88cee74893a7e808a1c5b8bba2, State: Initialized, Role: FOLLOWER
I20250113 14:56:17.195307 29436 consensus_queue.cc:260] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:17.195923 29430 tablet_replica.cc:331] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de: stopping tablet replica
I20250113 14:56:17.195961 29436 ts_tablet_manager.cc:1428] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250113 14:56:17.196467 29430 raft_consensus.cc:2238] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de [term 1 FOLLOWER]: Raft consensus shutting down.
I20250113 14:56:17.197126 29430 pending_rounds.cc:62] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de: Trying to abort 1 pending ops.
I20250113 14:56:17.198734 29430 pending_rounds.cc:69] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de: Aborting op as it isn't in flight: id { term: 1 index: 220 } timestamp: 7113851605608660992 op_type: WRITE_OP write_request { tablet_id: "45bb4becce044b429b4235b7062b6e3e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\310\377\254P\243\301\234\200\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\034\353\263BC\316\322`\020\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\034<\347V|H$> \000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\374&`W\030\r\200l0\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\351N\221I\006^\037>@\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\322\264AX\312\272\372KP\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\203e^QM:\023\223`\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\r\021\366K\275\343^\335p\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000\366Y&C\337\232\365\203\200\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000\001\007\000fYwR\2655\271R\220\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""12e3eb81e2f51955c8f6e0f11053e5a08c346f890d0d234353ecaea3af7af4c14c026239e0fd6656121b0ed85cd720626f97d45a31ecf23e5119ac84591a2dc2231e52de37a4fdaa8b5b15ec91f8bf35" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7113851605505011712 authz_token { token_data: "\010\271\325\224\274\006\"3\n\005slave\022*\n b2b4c51b1bd6475ba8e5bb3f9e211634\020\001\030\001 \001(\001" signature: "<redacted>""\254jd\000\330\350\264\016\207,/y\307\373o\214]\355\336\000<\336\000\3552\274\000\036M\210\300\322\254\300\313Ne\350\212|V\253\306\3638\2715D\212\026\321%\375\332\340;\356b\245\376\2348\325\036" signing_key_seq_num: 0 } } request_id { client_id: "5a3392a41e264d8dbd2310f89bb498ac" seq_no: 877 first_incomplete_seq_no: 872 attempt_no: 0 }
I20250113 14:56:17.199818 29430 raft_consensus.cc:2267] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 14:56:17.202008 29430 ts_tablet_manager.cc:1905] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 14:56:17.202952 28862 master_service.cc:946] ReplaceTablet: received request to replace tablet 45bb4becce044b429b4235b7062b6e3e from {username='slave'} at 127.0.0.1:54930
I20250113 14:56:17.203685 29435 tablet_bootstrap.cc:654] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592: Neither blocks nor log segments found. Creating new log.
I20250113 14:56:17.203856 29434 tablet_bootstrap.cc:492] T b46a2c13dda4486892efa9475b7b3032 P 43263e8341a8434891e23c68568253de: No bootstrap required, opened a new log
I20250113 14:56:17.204016 29434 ts_tablet_manager.cc:1397] T b46a2c13dda4486892efa9475b7b3032 P 43263e8341a8434891e23c68568253de: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/replace_tablet-itest.cc:133: Failure
Failed
Bad status: Not found: Tablet 45bb4becce044b429b4235b7062b6e3e already deleted
I20250113 14:56:17.204893 29434 raft_consensus.cc:357] T b46a2c13dda4486892efa9475b7b3032 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:17.205096 29434 raft_consensus.cc:383] T b46a2c13dda4486892efa9475b7b3032 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:17.206398 29434 raft_consensus.cc:738] T b46a2c13dda4486892efa9475b7b3032 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43263e8341a8434891e23c68568253de, State: Initialized, Role: FOLLOWER
I20250113 14:56:17.206655 29434 consensus_queue.cc:260] T b46a2c13dda4486892efa9475b7b3032 P 43263e8341a8434891e23c68568253de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:17.207273 29434 ts_tablet_manager.cc:1428] T b46a2c13dda4486892efa9475b7b3032 P 43263e8341a8434891e23c68568253de: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250113 14:56:17.215346 29431 raft_consensus.cc:2267] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250113 14:56:17.218468 29438 raft_consensus.cc:491] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 14:56:17.218693 29431 ts_tablet_manager.cc:1905] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250113 14:56:17.218663 29438 raft_consensus.cc:513] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:17.219570 29438 leader_election.cc:290] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43263e8341a8434891e23c68568253de (127.28.39.193:38947), f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194:44385)
I20250113 14:56:17.220294 29011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b46a2c13dda4486892efa9475b7b3032" candidate_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43263e8341a8434891e23c68568253de" is_pre_election: true
I20250113 14:56:17.220535 29011 raft_consensus.cc:2463] T b46a2c13dda4486892efa9475b7b3032 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d6ff88cee74893a7e808a1c5b8bba2 in term 0.
I20250113 14:56:17.221153 29207 leader_election.cc:304] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d6ff88cee74893a7e808a1c5b8bba2, 43263e8341a8434891e23c68568253de; no voters:
I20250113 14:56:17.221588 29438 raft_consensus.cc:2798] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250113 14:56:17.221731 29438 raft_consensus.cc:491] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250113 14:56:17.221832 29438 raft_consensus.cc:3054] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:17.222221 29139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b46a2c13dda4486892efa9475b7b3032" candidate_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ec7575f7d64455a4ecaa314f439592" is_pre_election: true
W20250113 14:56:17.222823 29208 leader_election.cc:343] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194:44385): Illegal state: must be running to vote when last-logged opid is not known
I20250113 14:56:17.224934 29438 raft_consensus.cc:513] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:17.226524 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b46a2c13dda4486892efa9475b7b3032" candidate_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ec7575f7d64455a4ecaa314f439592"
W20250113 14:56:17.227017 29208 leader_election.cc:343] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194:44385): Illegal state: must be running to vote when last-logged opid is not known
I20250113 14:56:17.227115 29008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b46a2c13dda4486892efa9475b7b3032" candidate_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43263e8341a8434891e23c68568253de"
I20250113 14:56:17.227345 29008 raft_consensus.cc:3054] T b46a2c13dda4486892efa9475b7b3032 P 43263e8341a8434891e23c68568253de [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:17.227717 29435 tablet_bootstrap.cc:492] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592: No bootstrap required, opened a new log
I20250113 14:56:17.227877 29435 ts_tablet_manager.cc:1397] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.006s
I20250113 14:56:17.225665 29438 leader_election.cc:290] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [CANDIDATE]: Term 1 election: Requested vote from peers 43263e8341a8434891e23c68568253de (127.28.39.193:38947), f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194:44385)
I20250113 14:56:17.229163 29430 ts_tablet_manager.cc:1918] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.220
I20250113 14:56:17.229219 29435 raft_consensus.cc:357] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:17.229353 29430 log.cc:1198] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de: Deleting WAL directory at /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-0/wal/wals/45bb4becce044b429b4235b7062b6e3e
I20250113 14:56:17.229441 29435 raft_consensus.cc:383] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250113 14:56:17.229544 29435 raft_consensus.cc:738] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ec7575f7d64455a4ecaa314f439592, State: Initialized, Role: FOLLOWER
I20250113 14:56:17.229779 29435 consensus_queue.cc:260] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:17.230095 29430 ts_tablet_manager.cc:1939] T 45bb4becce044b429b4235b7062b6e3e P 43263e8341a8434891e23c68568253de: Deleting consensus metadata
I20250113 14:56:17.230311 29008 raft_consensus.cc:2463] T b46a2c13dda4486892efa9475b7b3032 P 43263e8341a8434891e23c68568253de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d6ff88cee74893a7e808a1c5b8bba2 in term 1.
I20250113 14:56:17.230819 29207 leader_election.cc:304] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12d6ff88cee74893a7e808a1c5b8bba2, 43263e8341a8434891e23c68568253de; no voters: f7ec7575f7d64455a4ecaa314f439592
I20250113 14:56:17.232213 29438 raft_consensus.cc:2798] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 FOLLOWER]: Leader election won for term 1
I20250113 14:56:17.232569 28856 catalog_manager.cc:4872] TS 43263e8341a8434891e23c68568253de (127.28.39.193:38947): tablet 45bb4becce044b429b4235b7062b6e3e (table test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]) successfully deleted
I20250113 14:56:17.232860 29438 raft_consensus.cc:695] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [term 1 LEADER]: Becoming Leader. State: Replica: 12d6ff88cee74893a7e808a1c5b8bba2, State: Running, Role: LEADER
I20250113 14:56:17.233289 29435 ts_tablet_manager.cc:1428] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250113 14:56:17.233162 29438 consensus_queue.cc:237] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:17.237166 28862 catalog_manager.cc:5526] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12d6ff88cee74893a7e808a1c5b8bba2 (127.28.39.195). New cstate: current_term: 1 leader_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } health_report { overall_health: HEALTHY } } }
I20250113 14:56:17.240095 29431 ts_tablet_manager.cc:1918] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.215
I20250113 14:56:17.240437 29431 log.cc:1198] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592: Deleting WAL directory at /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0/minicluster-data/ts-1/wal/wals/45bb4becce044b429b4235b7062b6e3e
I20250113 14:56:17.241088 29431 ts_tablet_manager.cc:1939] T 45bb4becce044b429b4235b7062b6e3e P f7ec7575f7d64455a4ecaa314f439592: Deleting consensus metadata
I20250113 14:56:17.241209 29330 meta_cache.cc:557] tablet seems to be replaced: former ID 45bb4becce044b429b4235b7062b6e3e, new ID b46a2c13dda4486892efa9475b7b3032 (table test-workload)
W20250113 14:56:17.241459 29330 batcher.cc:441] Invalid argument: Failed to write batch of 9 ops to tablet 45bb4becce044b429b4235b7062b6e3e after 4 attempt(s): tablet ID 45bb4becce044b429b4235b7062b6e3e is not valid
I20250113 14:56:17.242738 28857 catalog_manager.cc:4872] TS f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194:44385): tablet 45bb4becce044b429b4235b7062b6e3e (table test-workload [id=b2b4c51b1bd6475ba8e5bb3f9e211634]) successfully deleted
I20250113 14:56:17.243285 29330 meta_cache.cc:557] tablet seems to be replaced: former ID 45bb4becce044b429b4235b7062b6e3e, new ID b46a2c13dda4486892efa9475b7b3032 (table test-workload)
I20250113 14:56:17.243657 29330 meta_cache.cc:557] tablet seems to be replaced: former ID 45bb4becce044b429b4235b7062b6e3e, new ID b46a2c13dda4486892efa9475b7b3032 (table test-workload)
I20250113 14:56:17.244026 29330 meta_cache.cc:557] tablet seems to be replaced: former ID 45bb4becce044b429b4235b7062b6e3e, new ID b46a2c13dda4486892efa9475b7b3032 (table test-workload)
I20250113 14:56:17.245889 28831 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20250113 14:56:17.246099 28831 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 43263e8341a8434891e23c68568253de and pid 28929
************************ BEGIN STACKS **************************
I20250113 14:56:17.601696 29419 raft_consensus.cc:491] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250113 14:56:17.601907 29419 raft_consensus.cc:513] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 } } peers { permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 } } peers { permanent_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" member_type: VOTER last_known_addr { host: "127.28.39.195" port: 39003 } }
I20250113 14:56:17.602557 29419 leader_election.cc:290] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43263e8341a8434891e23c68568253de (127.28.39.193:38947), 12d6ff88cee74893a7e808a1c5b8bba2 (127.28.39.195:39003)
I20250113 14:56:17.603152 29272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b46a2c13dda4486892efa9475b7b3032" candidate_uuid: "f7ec7575f7d64455a4ecaa314f439592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d6ff88cee74893a7e808a1c5b8bba2" is_pre_election: true
I20250113 14:56:17.603271 29008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b46a2c13dda4486892efa9475b7b3032" candidate_uuid: "f7ec7575f7d64455a4ecaa314f439592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43263e8341a8434891e23c68568253de" is_pre_election: true
I20250113 14:56:17.603576 29008 raft_consensus.cc:2388] T b46a2c13dda4486892efa9475b7b3032 P 43263e8341a8434891e23c68568253de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7ec7575f7d64455a4ecaa314f439592 in current term 1: Already voted for candidate 12d6ff88cee74893a7e808a1c5b8bba2 in this term.
I20250113 14:56:17.604132 29077 leader_election.cc:304] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7ec7575f7d64455a4ecaa314f439592; no voters: 12d6ff88cee74893a7e808a1c5b8bba2, 43263e8341a8434891e23c68568253de
I20250113 14:56:17.604418 29419 raft_consensus.cc:3054] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592 [term 0 FOLLOWER]: Advancing to term 1
I20250113 14:56:17.606424 29419 raft_consensus.cc:2743] T b46a2c13dda4486892efa9475b7b3032 P f7ec7575f7d64455a4ecaa314f439592 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250113 14:56:17.648030 29438 consensus_queue.cc:1035] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43263e8341a8434891e23c68568253de" member_type: VOTER last_known_addr { host: "127.28.39.193" port: 38947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250113 14:56:17.652444 29372 consensus_queue.cc:1035] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ec7575f7d64455a4ecaa314f439592" member_type: VOTER last_known_addr { host: "127.28.39.194" port: 44385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
[New LWP 28930]
[New LWP 28931]
[New LWP 28932]
[New LWP 28933]
[New LWP 28939]
[New LWP 28940]
[New LWP 28941]
[New LWP 28944]
[New LWP 28945]
[New LWP 28946]
[New LWP 28947]
[New LWP 28948]
[New LWP 28949]
[New LWP 28950]
[New LWP 28951]
[New LWP 28952]
[New LWP 28953]
[New LWP 28954]
[New LWP 28955]
[New LWP 28956]
[New LWP 28957]
[New LWP 28958]
[New LWP 28959]
[New LWP 28960]
[New LWP 28961]
[New LWP 28962]
[New LWP 28963]
[New LWP 28964]
[New LWP 28965]
[New LWP 28966]
[New LWP 28967]
[New LWP 28968]
[New LWP 28969]
[New LWP 28970]
[New LWP 28971]
[New LWP 28972]
[New LWP 28973]
[New LWP 28974]
[New LWP 28975]
[New LWP 28976]
[New LWP 28977]
[New LWP 28978]
[New LWP 28979]
[New LWP 28980]
[New LWP 28981]
[New LWP 28982]
[New LWP 28983]
[New LWP 28984]
[New LWP 28985]
[New LWP 28986]
[New LWP 28987]
[New LWP 28988]
[New LWP 28989]
[New LWP 28990]
[New LWP 28991]
[New LWP 28992]
[New LWP 28993]
[New LWP 28994]
[New LWP 28995]
[New LWP 28996]
[New LWP 28997]
[New LWP 28998]
[New LWP 28999]
[New LWP 29000]
[New LWP 29001]
[New LWP 29002]
[New LWP 29003]
[New LWP 29004]
[New LWP 29005]
[New LWP 29006]
[New LWP 29007]
[New LWP 29008]
[New LWP 29009]
[New LWP 29010]
[New LWP 29011]
[New LWP 29012]
[New LWP 29013]
[New LWP 29014]
[New LWP 29015]
[New LWP 29016]
[New LWP 29017]
[New LWP 29018]
[New LWP 29019]
[New LWP 29020]
[New LWP 29021]
[New LWP 29022]
[New LWP 29023]
[New LWP 29024]
[New LWP 29025]
[New LWP 29026]
[New LWP 29027]
[New LWP 29028]
[New LWP 29029]
[New LWP 29030]
[New LWP 29031]
[New LWP 29032]
[New LWP 29033]
[New LWP 29034]
[New LWP 29035]
[New LWP 29036]
[New LWP 29037]
[New LWP 29038]
[New LWP 29039]
[New LWP 29040]
[New LWP 29041]
[New LWP 29042]
[New LWP 29043]
[New LWP 29044]
[New LWP 29045]
[New LWP 29046]
[New LWP 29047]
[New LWP 29048]
[New LWP 29049]
[New LWP 29050]
[New LWP 29051]
[New LWP 29052]
[New LWP 29053]
[New LWP 29054]
[New LWP 29055]
[New LWP 29056]
[New LWP 29057]
[New LWP 29370]
[New LWP 29393]
[New LWP 29394]
[New LWP 29399]
[New LWP 29420]
[New LWP 29447]
Cannot access memory at address 0x5f35535f35534b51
Cannot access memory at address 0x5f35535f35534b49
Cannot access memory at address 0x5f35535f35534b51
Cannot access memory at address 0x5f35535f35534b51
Cannot access memory at address 0x5f35535f35534b49
0x00007ff6bb7bdd50 in ?? ()
Id Target Id Frame
* 1 LWP 28929 "kudu" 0x00007ff6bb7bdd50 in ?? ()
2 LWP 28930 "kudu" 0x00007ff6bb7b9fb9 in ?? ()
3 LWP 28931 "kudu" 0x00007ff6bb7b9fb9 in ?? ()
4 LWP 28932 "kudu" 0x00007ff6bb7b9fb9 in ?? ()
5 LWP 28933 "kernel-watcher-" 0x00007ff6bb7b9fb9 in ?? ()
6 LWP 28939 "ntp client-2893" 0x00007ff6bb7bd9e2 in ?? ()
7 LWP 28940 "file cache-evic" 0x00007ff6bb7b9fb9 in ?? ()
8 LWP 28941 "sq_acceptor" 0x00007ff6b4516cb9 in ?? ()
9 LWP 28944 "rpc reactor-289" 0x00007ff6b4523a47 in ?? ()
10 LWP 28945 "rpc reactor-289" 0x00007ff6b4523a47 in ?? ()
11 LWP 28946 "rpc reactor-289" 0x00007ff6b4523a47 in ?? ()
12 LWP 28947 "rpc reactor-289" 0x00007ff6b4523a47 in ?? ()
13 LWP 28948 "MaintenanceMgr " 0x00007ff6bb7b9ad3 in ?? ()
14 LWP 28949 "txn-status-mana" 0x00007ff6bb7b9fb9 in ?? ()
15 LWP 28950 "collect_and_rem" 0x00007ff6bb7b9fb9 in ?? ()
16 LWP 28951 "tc-session-exp-" 0x00007ff6bb7b9fb9 in ?? ()
17 LWP 28952 "rpc worker-2895" 0x00007ff6bb7b9ad3 in ?? ()
18 LWP 28953 "rpc worker-2895" 0x00007ff6bb7b9ad3 in ?? ()
19 LWP 28954 "rpc worker-2895" 0x00007ff6bb7b9ad3 in ?? ()
20 LWP 28955 "rpc worker-2895" 0x00007ff6bb7b9ad3 in ?? ()
21 LWP 28956 "rpc worker-2895" 0x00007ff6bb7b9ad3 in ?? ()
22 LWP 28957 "rpc worker-2895" 0x00007ff6bb7b9ad3 in ?? ()
23 LWP 28958 "rpc worker-2895" 0x00007ff6bb7b9ad3 in ?? ()
24 LWP 28959 "rpc worker-2895" 0x00007ff6bb7b9ad3 in ?? ()
25 LWP 28960 "rpc worker-2896" 0x00007ff6bb7b9ad3 in ?? ()
26 LWP 28961 "rpc worker-2896" 0x00007ff6bb7b9ad3 in ?? ()
27 LWP 28962 "rpc worker-2896" 0x00007ff6bb7b9ad3 in ?? ()
28 LWP 28963 "rpc worker-2896" 0x00007ff6bb7b9ad3 in ?? ()
29 LWP 28964 "rpc worker-2896" 0x00007ff6bb7b9ad3 in ?? ()
30 LWP 28965 "rpc worker-2896" 0x00007ff6bb7b9ad3 in ?? ()
31 LWP 28966 "rpc worker-2896" 0x00007ff6bb7b9ad3 in ?? ()
32 LWP 28967 "rpc worker-2896" 0x00007ff6bb7b9ad3 in ?? ()
33 LWP 28968 "rpc worker-2896" 0x00007ff6bb7b9ad3 in ?? ()
34 LWP 28969 "rpc worker-2896" 0x00007ff6bb7b9ad3 in ?? ()
35 LWP 28970 "rpc worker-2897" 0x00007ff6bb7b9ad3 in ?? ()
36 LWP 28971 "rpc worker-2897" 0x00007ff6bb7b9ad3 in ?? ()
37 LWP 28972 "rpc worker-2897" 0x00007ff6bb7b9ad3 in ?? ()
38 LWP 28973 "rpc worker-2897" 0x00007ff6bb7b9ad3 in ?? ()
39 LWP 28974 "rpc worker-2897" 0x00007ff6bb7b9ad3 in ?? ()
40 LWP 28975 "rpc worker-2897" 0x00007ff6bb7b9ad3 in ?? ()
41 LWP 28976 "rpc worker-2897" 0x00007ff6bb7b9ad3 in ?? ()
42 LWP 28977 "rpc worker-2897" 0x00007ff6bb7b9ad3 in ?? ()
43 LWP 28978 "rpc worker-2897" 0x00007ff6bb7b9ad3 in ?? ()
44 LWP 28979 "rpc worker-2897" 0x00007ff6bb7b9ad3 in ?? ()
45 LWP 28980 "rpc worker-2898" 0x00007ff6bb7b9ad3 in ?? ()
46 LWP 28981 "rpc worker-2898" 0x00007ff6bb7b9ad3 in ?? ()
47 LWP 28982 "rpc worker-2898" 0x00007ff6bb7b9ad3 in ?? ()
48 LWP 28983 "rpc worker-2898" 0x00007ff6bb7b9ad3 in ?? ()
49 LWP 28984 "rpc worker-2898" 0x00007ff6bb7b9ad3 in ?? ()
50 LWP 28985 "rpc worker-2898" 0x00007ff6bb7b9ad3 in ?? ()
51 LWP 28986 "rpc worker-2898" 0x00007ff6bb7b9ad3 in ?? ()
52 LWP 28987 "rpc worker-2898" 0x00007ff6bb7b9ad3 in ?? ()
53 LWP 28988 "rpc worker-2898" 0x00007ff6bb7b9ad3 in ?? ()
54 LWP 28989 "rpc worker-2898" 0x00007ff6bb7b9ad3 in ?? ()
55 LWP 28990 "rpc worker-2899" 0x00007ff6bb7b9ad3 in ?? ()
56 LWP 28991 "rpc worker-2899" 0x00007ff6bb7b9ad3 in ?? ()
57 LWP 28992 "rpc worker-2899" 0x00007ff6bb7b9ad3 in ?? ()
58 LWP 28993 "rpc worker-2899" 0x00007ff6bb7b9ad3 in ?? ()
59 LWP 28994 "rpc worker-2899" 0x00007ff6bb7b9ad3 in ?? ()
60 LWP 28995 "rpc worker-2899" 0x00007ff6bb7b9ad3 in ?? ()
61 LWP 28996 "rpc worker-2899" 0x00007ff6bb7b9ad3 in ?? ()
62 LWP 28997 "rpc worker-2899" 0x00007ff6bb7b9ad3 in ?? ()
63 LWP 28998 "rpc worker-2899" 0x00007ff6bb7b9ad3 in ?? ()
64 LWP 28999 "rpc worker-2899" 0x00007ff6bb7b9ad3 in ?? ()
65 LWP 29000 "rpc worker-2900" 0x00007ff6bb7b9ad3 in ?? ()
66 LWP 29001 "rpc worker-2900" 0x00007ff6bb7b9ad3 in ?? ()
67 LWP 29002 "rpc worker-2900" 0x00007ff6bb7b9ad3 in ?? ()
68 LWP 29003 "rpc worker-2900" 0x00007ff6bb7b9ad3 in ?? ()
69 LWP 29004 "rpc worker-2900" 0x00007ff6bb7b9ad3 in ?? ()
70 LWP 29005 "rpc worker-2900" 0x00007ff6bb7b9ad3 in ?? ()
71 LWP 29006 "rpc worker-2900" 0x00007ff6bb7b9ad3 in ?? ()
72 LWP 29007 "rpc worker-2900" 0x00007ff6bb7b9ad3 in ?? ()
73 LWP 29008 "rpc worker-2900" 0x00007ff6bb7b9ad3 in ?? ()
74 LWP 29009 "rpc worker-2900" 0x00007ff6bb7b9ad3 in ?? ()
75 LWP 29010 "rpc worker-2901" 0x00007ff6bb7b9ad3 in ?? ()
76 LWP 29011 "rpc worker-2901" 0x00007ff6bb7b9ad3 in ?? ()
77 LWP 29012 "rpc worker-2901" 0x00007ff6bb7b9ad3 in ?? ()
78 LWP 29013 "rpc worker-2901" 0x00007ff6bb7b9ad3 in ?? ()
79 LWP 29014 "rpc worker-2901" 0x00007ff6bb7b9ad3 in ?? ()
80 LWP 29015 "rpc worker-2901" 0x00007ff6bb7b9ad3 in ?? ()
81 LWP 29016 "rpc worker-2901" 0x00007ff6bb7b9ad3 in ?? ()
82 LWP 29017 "rpc worker-2901" 0x00007ff6bb7b9ad3 in ?? ()
83 LWP 29018 "rpc worker-2901" 0x00007ff6bb7b9ad3 in ?? ()
84 LWP 29019 "rpc worker-2901" 0x00007ff6bb7b9ad3 in ?? ()
85 LWP 29020 "rpc worker-2902" 0x00007ff6bb7b9ad3 in ?? ()
86 LWP 29021 "rpc worker-2902" 0x00007ff6bb7b9ad3 in ?? ()
87 LWP 29022 "rpc worker-2902" 0x00007ff6bb7b9ad3 in ?? ()
88 LWP 29023 "rpc worker-2902" 0x00007ff6bb7b9ad3 in ?? ()
89 LWP 29024 "rpc worker-2902" 0x00007ff6bb7b9ad3 in ?? ()
90 LWP 29025 "rpc worker-2902" 0x00007ff6bb7b9ad3 in ?? ()
91 LWP 29026 "rpc worker-2902" 0x00007ff6bb7b9ad3 in ?? ()
92 LWP 29027 "rpc worker-2902" 0x00007ff6bb7b9ad3 in ?? ()
93 LWP 29028 "rpc worker-2902" 0x00007ff6bb7b9ad3 in ?? ()
94 LWP 29029 "rpc worker-2902" 0x00007ff6bb7b9ad3 in ?? ()
95 LWP 29030 "rpc worker-2903" 0x00007ff6bb7b9ad3 in ?? ()
96 LWP 29031 "rpc worker-2903" 0x00007ff6bb7b9ad3 in ?? ()
97 LWP 29032 "rpc worker-2903" 0x00007ff6bb7b9ad3 in ?? ()
98 LWP 29033 "rpc worker-2903" 0x00007ff6bb7b9ad3 in ?? ()
99 LWP 29034 "rpc worker-2903" 0x00007ff6bb7b9ad3 in ?? ()
100 LWP 29035 "rpc worker-2903" 0x00007ff6bb7b9ad3 in ?? ()
101 LWP 29036 "rpc worker-2903" 0x00007ff6bb7b9ad3 in ?? ()
102 LWP 29037 "rpc worker-2903" 0x00007ff6bb7b9ad3 in ?? ()
103 LWP 29038 "rpc worker-2903" 0x00007ff6bb7b9ad3 in ?? ()
104 LWP 29039 "rpc worker-2903" 0x00007ff6bb7b9ad3 in ?? ()
105 LWP 29040 "rpc worker-2904" 0x00007ff6bb7b9ad3 in ?? ()
106 LWP 29041 "rpc worker-2904" 0x00007ff6bb7b9ad3 in ?? ()
107 LWP 29042 "rpc worker-2904" 0x00007ff6bb7b9ad3 in ?? ()
108 LWP 29043 "rpc worker-2904" 0x00007ff6bb7b9ad3 in ?? ()
109 LWP 29044 "rpc worker-2904" 0x00007ff6bb7b9ad3 in ?? ()
110 LWP 29045 "rpc worker-2904" 0x00007ff6bb7b9ad3 in ?? ()
111 LWP 29046 "rpc worker-2904" 0x00007ff6bb7b9ad3 in ?? ()
112 LWP 29047 "rpc worker-2904" 0x00007ff6bb7b9ad3 in ?? ()
113 LWP 29048 "rpc worker-2904" 0x00007ff6bb7b9ad3 in ?? ()
114 LWP 29049 "rpc worker-2904" 0x00007ff6bb7b9ad3 in ?? ()
115 LWP 29050 "rpc worker-2905" 0x00007ff6bb7b9ad3 in ?? ()
116 LWP 29051 "rpc worker-2905" 0x00007ff6bb7b9ad3 in ?? ()
117 LWP 29052 "diag-logger-290" 0x00007ff6bb7b9fb9 in ?? ()
118 LWP 29053 "result-tracker-" 0x00007ff6bb7b9fb9 in ?? ()
119 LWP 29054 "excess-log-dele" 0x00007ff6bb7b9fb9 in ?? ()
120 LWP 29055 "acceptor-29055" 0x00007ff6b45250c7 in ?? ()
121 LWP 29056 "heartbeat-29056" 0x00007ff6bb7b9fb9 in ?? ()
122 LWP 29057 "maintenance_sch" 0x00007ff6bb7b9fb9 in ?? ()
123 LWP 29370 "wal-append [wor" 0x00007ff6bb7b9fb9 in ?? ()
124 LWP 29393 "prepare [worker" 0x00007ff6bb7b9fb9 in ?? ()
125 LWP 29394 "wal-append [wor" 0x00007ff6bb7b9fb9 in ?? ()
126 LWP 29399 "wal-append [wor" 0x00007ff6bb7b9fb9 in ?? ()
127 LWP 29420 "raft [worker]-2" 0x00007ff6bb7b9fb9 in ?? ()
128 LWP 29447 "wal-append [wor" 0x00007ff6bb7b9fb9 in ?? ()
Thread 128 (LWP 29447):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x000060200017bd98 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x000061a00013ffb4 in ?? ()
#5 0x00007ff66ae8b090 in ?? ()
#6 0x0000000000000003 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007ff66ae8b0b0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 127 (LWP 29420):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000000fb in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00007ff665574b24 in ?? ()
#5 0x00007ff6655747b0 in ?? ()
#6 0x00000000000001f7 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007ff6655747d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 126 (LWP 29399):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x0000603000468a18 in ?? ()
#2 0x0000000000000182 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061a0000259b0 in ?? ()
#5 0x00007ff668605090 in ?? ()
#6 0x0000000000000304 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 125 (LWP 29394):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x0000602000185638 in ?? ()
#2 0x0000000000000148 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061a0000223b0 in ?? ()
#5 0x00007ff668e1d090 in ?? ()
#6 0x0000000000000290 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 124 (LWP 29393):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000000ea in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007ff669635b20 in ?? ()
#5 0x00007ff6696357b0 in ?? ()
#6 0x00000000000001d4 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 123 (LWP 29370):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x0000602000189e58 in ?? ()
#2 0x000000000000019a in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061a0000265b0 in ?? ()
#5 0x00007ff66b6a5090 in ?? ()
#6 0x0000000000000334 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 29057):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000015 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007ff670875550 in ?? ()
#6 0x000000000000002a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 29056):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0xd35e57c419f6ada3 in ?? ()
#2 0x000000000000000b in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x0000613000019cc4 in ?? ()
#5 0x00007ff67108d5b0 in ?? ()
#6 0x0000000000000017 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007ff67108d5d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007ff67108d650 in ?? ()
#12 0x00000ffece211acc in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 120 (LWP 29055):
#0 0x00007ff6b45250c7 in ?? ()
#1 0x00006140000220a8 in ?? ()
#2 0x00007ff6718b4d50 in ?? ()
#3 0x00007ff6718b4d80 in ?? ()
#4 0x00007ff6718b4e80 in ?? ()
#5 0x00007ff6718b4d70 in ?? ()
#6 0x00007ff6718b4de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x0000000500000014 in ?? ()
#10 0x00007ff6718b4f00 in ?? ()
#11 0x00007ff6718b463c in ?? ()
#12 0x00000032718b45b0 in ?? ()
#13 0x00007ff6710b8000 in ?? ()
#14 0x0000000000000000 in ?? ()
Thread 119 (LWP 29054):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x00007ff6720cef00 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 29053):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x00007ff6728e70c0 in ?? ()
#2 0x0000000000000005 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000879b0 in ?? ()
#5 0x00007ff6728e70b0 in ?? ()
#6 0x000000000000000a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 29052):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 29051):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 29050):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 29049):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 29048):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 29047):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 29046):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 29045):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 29044):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 29043):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 29042):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 29041):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 29040):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 29039):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 29038):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 29037):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 29036):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 29035):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 29034):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 29033):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 29032):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 29031):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 29030):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 29029):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 29028):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 29027):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 29026):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 29025):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 29024):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 29023):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 29022):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 29021):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 29020):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 29019):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 29018):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 29017):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 29016):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 29015):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 29014):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 29013):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 29012):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 29011):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x00000000000000b1 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007ff687cd7e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff687cd7e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007ff6bb7b9770 in ?? ()
#10 0x00007ff687cd7e70 in ?? ()
#11 0x00000ffed0f9afd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 29010):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x00000000000000b9 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a688c in ?? ()
#4 0x00007ff6884efe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff6884efe70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a6840 in ?? ()
#9 0x00007ff6bb7b9770 in ?? ()
#10 0x00007ff6884efe70 in ?? ()
#11 0x00000ffed109dfd0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 74 (LWP 29009):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x000000000000005e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a0088 in ?? ()
#4 0x00007ff688d07e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff688d07e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 73 (LWP 29008):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x00000000000000d6 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001967c8 in ?? ()
#4 0x00007ff68951fe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff68951fe70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 72 (LWP 29007):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000028 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00018ffc8 in ?? ()
#4 0x00007ff689d37e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff689d37e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 71 (LWP 29006):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 29005):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 29004):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 29003):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 29002):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 29001):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 29000):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 28999):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 28998):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 28997):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 28996):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 28995):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 28994):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 28993):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 28992):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 28991):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012683c in ?? ()
#4 0x00007ff691eb8e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff691eb8e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001267f0 in ?? ()
#9 0x00007ff6bb7b9770 in ?? ()
#10 0x00007ff691eb8e70 in ?? ()
#11 0x00000ffed23d71d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 55 (LWP 28990):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000120038 in ?? ()
#4 0x00007ff6926d0e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff6926d0e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 54 (LWP 28989):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00011d04c in ?? ()
#4 0x00007ff692ee8e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff692ee8e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00011d000 in ?? ()
#9 0x00007ff6bb7b9770 in ?? ()
#10 0x00007ff692ee8e70 in ?? ()
#11 0x00000ffed25dd1d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 53 (LWP 28988):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000116848 in ?? ()
#4 0x00007ff693700e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff693700e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 52 (LWP 28987):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 28986):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 28985):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 28984):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 28983):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 28982):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 28981):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 28980):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 28979):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 28978):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 28977):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 28976):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 28975):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 28974):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 28973):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 28972):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 28971):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x000000000000000b in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007ff69c099e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff69c099e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007ff6bb7b9770 in ?? ()
#10 0x00007ff69c099e70 in ?? ()
#11 0x00000ffed38133d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 28970):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x000000000000007a in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00009ffe8 in ?? ()
#4 0x00007ff69c8b1e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff69c8b1e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 34 (LWP 28969):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000015 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000967fc in ?? ()
#4 0x00007ff69d0c9e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff69d0c9e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000967b0 in ?? ()
#9 0x00007ff6bb7b9770 in ?? ()
#10 0x00007ff69d0c9e70 in ?? ()
#11 0x00000ffed3a193d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 33 (LWP 28968):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000049 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00008fffc in ?? ()
#4 0x00007ff69d8e1e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007ff69d8e1e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00008ffb0 in ?? ()
#9 0x00007ff6bb7b9770 in ?? ()
#10 0x00007ff69d8e1e70 in ?? ()
#11 0x00000ffed3b1c3d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 32 (LWP 28967):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 28966):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 28965):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 28964):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 28963):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 28962):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 28961):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 28960):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 28959):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 28958):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 28957):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 28956):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 28955):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 28954):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 28953):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 28952):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 28951):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x00007ff6a62adc80 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 15 (LWP 28950):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 28949):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x00007ff6a72af1c0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 13 (LWP 28948):
#0 0x00007ff6bb7b9ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 28947):
#0 0x00007ff6b4523a47 in ?? ()
#1 0x00007ff6a82b22e0 in ?? ()
#2 0x000061a00000f680 in ?? ()
#3 0x00007ff6a82b22d0 in ?? ()
#4 0x00007ff6a82b24e0 in ?? ()
#5 0x00007ff6a82b2320 in ?? ()
#6 0x0000614000022498 in ?? ()
#7 0x00007ff6a82b23a0 in ?? ()
#8 0x00007ff6b6b8325d in ?? ()
#9 0x3fb66d309a7df000 in ?? ()
#10 0x000061a00000f680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f680 in ?? ()
#13 0x0000000018000000 in ?? ()
#14 0x00007ff600000000 in ?? ()
#15 0x41d9e149f1e29a38 in ?? ()
#16 0x00000fff5504e474 in ?? ()
#17 0x00007ff6a82b2380 in ?? ()
#18 0x00007ff6b6b87ba3 in ?? ()
#19 0x00007ff6a82b2350 in ?? ()
#20 0x3fb66d309a7df000 in ?? ()
#21 0x00000000a82b23a0 in ?? ()
#22 0x000061a00000f680 in ?? ()
#23 0x0000614000022498 in ?? ()
#24 0x3fb66d309a7df000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 28946):
#0 0x00007ff6b4523a47 in ?? ()
#1 0x00007ff6a8ab32e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007ff6a8ab32d0 in ?? ()
#4 0x00007ff6a8ab34e0 in ?? ()
#5 0x00007ff6a8ab3320 in ?? ()
#6 0x0000614000022298 in ?? ()
#7 0x00007ff6a8ab33a0 in ?? ()
#8 0x00007ff6b6b8325d in ?? ()
#9 0x3fb96c3d3bb25000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x000000002e000000 in ?? ()
#14 0x00007ff600000000 in ?? ()
#15 0x41d9e149f1e29a38 in ?? ()
#16 0x00000fff5514e674 in ?? ()
#17 0x00007ff6a8ab3380 in ?? ()
#18 0x00007ff6b6b87ba3 in ?? ()
#19 0x00007ff6a8ab3350 in ?? ()
#20 0x3fb96c3d3bb25000 in ?? ()
#21 0x00000000a8ab33a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000022298 in ?? ()
#24 0x3fb96c3d3bb25000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 28945):
#0 0x00007ff6b4523a47 in ?? ()
#1 0x00007ff6a92b42e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007ff6a92b42d0 in ?? ()
#4 0x00007ff6a92b44e0 in ?? ()
#5 0x00007ff6a92b4320 in ?? ()
#6 0x0000614000022098 in ?? ()
#7 0x00007ff6a92b43a0 in ?? ()
#8 0x00007ff6b6b8325d in ?? ()
#9 0x3fb604e3699db000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x0000000038400000 in ?? ()
#14 0x00007ff600000000 in ?? ()
#15 0x41d9e149f1e29a3b in ?? ()
#16 0x00000fff5524e874 in ?? ()
#17 0x00007ff6a92b4380 in ?? ()
#18 0x00007ff6b6b87ba3 in ?? ()
#19 0x00007ff6a92b4350 in ?? ()
#20 0x3fb604e3699db000 in ?? ()
#21 0x00000000a92b43a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000022098 in ?? ()
#24 0x3fb604e3699db000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 28944):
#0 0x00007ff6b4523a47 in ?? ()
#1 0x00007ff6ab2ad2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007ff6ab2ad2d0 in ?? ()
#4 0x00007ff6ab2ad4e0 in ?? ()
#5 0x00007ff6ab2ad320 in ?? ()
#6 0x0000614000021e98 in ?? ()
#7 0x00007ff6ab2ad3a0 in ?? ()
#8 0x00007ff6b6b8325d in ?? ()
#9 0x3fb97d985c2e4000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000042c00000 in ?? ()
#14 0x00007ff600000000 in ?? ()
#15 0x41d9e149f1e29a3c in ?? ()
#16 0x00000fff5564da74 in ?? ()
#17 0x00007ff6ab2ad380 in ?? ()
#18 0x00007ff6b6b87ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 28941):
#0 0x00007ff6b4516cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007ff6accb7788 in ?? ()
#3 0x0000602000016790 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 28940):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x00007ff6ac4b6080 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 28939):
#0 0x00007ff6bb7bd9e2 in ?? ()
#1 0x00007ff6abcb3b60 in ?? ()
#2 0x00007ff6abcb3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007ff6abcb3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007ff6abcb3c10 in ?? ()
#8 0x0000611000088950 in ?? ()
#9 0x00007ff6abcb3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007ff6c10e1f0c in ?? ()
#12 0x00007ff6c10d17b9 in ?? ()
#13 0x00007ff6c10d5da6 in ?? ()
#14 0x00007ff6c10da7e0 in ?? ()
#15 0x00007ff6c10da609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007ff6b83d00be in ?? ()
#18 0x00007ff6bb7b36db in ?? ()
#19 0x00000fff5578e67c in ?? ()
#20 0x00007ff6abcb3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 28933):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x00007ff6adcb8c40 in ?? ()
#2 0x000000000000001b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007ff6adcb8c30 in ?? ()
#6 0x0000000000000036 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 28932):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 28931):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 28930):
#0 0x00007ff6bb7b9fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007ff6af4bc410 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 28929):
#0 0x00007ff6bb7bdd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250113 14:56:17.834911 28831 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID f7ec7575f7d64455a4ecaa314f439592 and pid 29060
************************ BEGIN STACKS **************************
[New LWP 29061]
[New LWP 29062]
[New LWP 29063]
[New LWP 29064]
[New LWP 29070]
[New LWP 29071]
[New LWP 29072]
[New LWP 29075]
[New LWP 29076]
[New LWP 29077]
[New LWP 29078]
[New LWP 29079]
[New LWP 29080]
[New LWP 29081]
[New LWP 29082]
[New LWP 29083]
[New LWP 29084]
[New LWP 29085]
[New LWP 29086]
[New LWP 29087]
[New LWP 29088]
[New LWP 29089]
[New LWP 29090]
[New LWP 29091]
[New LWP 29092]
[New LWP 29093]
[New LWP 29094]
[New LWP 29095]
[New LWP 29096]
[New LWP 29097]
[New LWP 29098]
[New LWP 29099]
[New LWP 29100]
[New LWP 29101]
[New LWP 29102]
[New LWP 29103]
[New LWP 29104]
[New LWP 29105]
[New LWP 29106]
[New LWP 29107]
[New LWP 29108]
[New LWP 29109]
[New LWP 29110]
[New LWP 29111]
[New LWP 29112]
[New LWP 29113]
[New LWP 29114]
[New LWP 29115]
[New LWP 29116]
[New LWP 29117]
[New LWP 29118]
[New LWP 29119]
[New LWP 29120]
[New LWP 29121]
[New LWP 29122]
[New LWP 29123]
[New LWP 29124]
[New LWP 29125]
[New LWP 29126]
[New LWP 29127]
[New LWP 29128]
[New LWP 29129]
[New LWP 29130]
[New LWP 29131]
[New LWP 29132]
[New LWP 29133]
[New LWP 29134]
[New LWP 29135]
[New LWP 29136]
[New LWP 29137]
[New LWP 29138]
[New LWP 29139]
[New LWP 29140]
[New LWP 29141]
[New LWP 29142]
[New LWP 29143]
[New LWP 29144]
[New LWP 29145]
[New LWP 29146]
[New LWP 29147]
[New LWP 29148]
[New LWP 29149]
[New LWP 29150]
[New LWP 29151]
[New LWP 29152]
[New LWP 29153]
[New LWP 29154]
[New LWP 29155]
[New LWP 29156]
[New LWP 29157]
[New LWP 29158]
[New LWP 29159]
[New LWP 29160]
[New LWP 29161]
[New LWP 29162]
[New LWP 29163]
[New LWP 29164]
[New LWP 29165]
[New LWP 29166]
[New LWP 29167]
[New LWP 29168]
[New LWP 29169]
[New LWP 29170]
[New LWP 29171]
[New LWP 29172]
[New LWP 29173]
[New LWP 29174]
[New LWP 29175]
[New LWP 29176]
[New LWP 29177]
[New LWP 29178]
[New LWP 29179]
[New LWP 29180]
[New LWP 29181]
[New LWP 29182]
[New LWP 29183]
[New LWP 29184]
[New LWP 29185]
[New LWP 29186]
[New LWP 29187]
[New LWP 29188]
[New LWP 29419]
[New LWP 29448]
Cannot access memory at address 0x5f35535f35534b51
Cannot access memory at address 0x5f35535f35534b49
Cannot access memory at address 0x5f35535f35534b51
Cannot access memory at address 0x5f35535f35534b51
Cannot access memory at address 0x5f35535f35534b49
0x00007f492e028d50 in ?? ()
Id Target Id Frame
* 1 LWP 29060 "kudu" 0x00007f492e028d50 in ?? ()
2 LWP 29061 "kudu" 0x00007f492e024fb9 in ?? ()
3 LWP 29062 "kudu" 0x00007f492e024fb9 in ?? ()
4 LWP 29063 "kudu" 0x00007f492e024fb9 in ?? ()
5 LWP 29064 "kernel-watcher-" 0x00007f492e024fb9 in ?? ()
6 LWP 29070 "ntp client-2907" 0x00007f492e0289e2 in ?? ()
7 LWP 29071 "file cache-evic" 0x00007f492e024fb9 in ?? ()
8 LWP 29072 "sq_acceptor" 0x00007f4926d81cb9 in ?? ()
9 LWP 29075 "rpc reactor-290" 0x00007f4926d8ea47 in ?? ()
10 LWP 29076 "rpc reactor-290" 0x00007f4926d8ea47 in ?? ()
11 LWP 29077 "rpc reactor-290" 0x00007f4926d8ea47 in ?? ()
12 LWP 29078 "rpc reactor-290" 0x00007f4926d8ea47 in ?? ()
13 LWP 29079 "MaintenanceMgr " 0x00007f492e024ad3 in ?? ()
14 LWP 29080 "txn-status-mana" 0x00007f492e024fb9 in ?? ()
15 LWP 29081 "collect_and_rem" 0x00007f492e024fb9 in ?? ()
16 LWP 29082 "tc-session-exp-" 0x00007f492e024fb9 in ?? ()
17 LWP 29083 "rpc worker-2908" 0x00007f492e024ad3 in ?? ()
18 LWP 29084 "rpc worker-2908" 0x00007f492e024ad3 in ?? ()
19 LWP 29085 "rpc worker-2908" 0x00007f492e024ad3 in ?? ()
20 LWP 29086 "rpc worker-2908" 0x00007f492e024ad3 in ?? ()
21 LWP 29087 "rpc worker-2908" 0x00007f492e024ad3 in ?? ()
22 LWP 29088 "rpc worker-2908" 0x00007f492e024ad3 in ?? ()
23 LWP 29089 "rpc worker-2908" 0x00007f492e024ad3 in ?? ()
24 LWP 29090 "rpc worker-2909" 0x00007f492e024ad3 in ?? ()
25 LWP 29091 "rpc worker-2909" 0x00007f492e024ad3 in ?? ()
26 LWP 29092 "rpc worker-2909" 0x00007f492e024ad3 in ?? ()
27 LWP 29093 "rpc worker-2909" 0x00007f492e024ad3 in ?? ()
28 LWP 29094 "rpc worker-2909" 0x00007f492e024ad3 in ?? ()
29 LWP 29095 "rpc worker-2909" 0x00007f492e024ad3 in ?? ()
30 LWP 29096 "rpc worker-2909" 0x00007f492e024ad3 in ?? ()
31 LWP 29097 "rpc worker-2909" 0x00007f492e024ad3 in ?? ()
32 LWP 29098 "rpc worker-2909" 0x00007f492e024ad3 in ?? ()
33 LWP 29099 "rpc worker-2909" 0x00007f492e024ad3 in ?? ()
34 LWP 29100 "rpc worker-2910" 0x00007f492e024ad3 in ?? ()
35 LWP 29101 "rpc worker-2910" 0x00007f492e024ad3 in ?? ()
36 LWP 29102 "rpc worker-2910" 0x00007f492e024ad3 in ?? ()
37 LWP 29103 "rpc worker-2910" 0x00007f492e024ad3 in ?? ()
38 LWP 29104 "rpc worker-2910" 0x00007f492e024ad3 in ?? ()
39 LWP 29105 "rpc worker-2910" 0x00007f492e024ad3 in ?? ()
40 LWP 29106 "rpc worker-2910" 0x00007f492e024ad3 in ?? ()
41 LWP 29107 "rpc worker-2910" 0x00007f492e024ad3 in ?? ()
42 LWP 29108 "rpc worker-2910" 0x00007f492e024ad3 in ?? ()
43 LWP 29109 "rpc worker-2910" 0x00007f492e024ad3 in ?? ()
44 LWP 29110 "rpc worker-2911" 0x00007f492e024ad3 in ?? ()
45 LWP 29111 "rpc worker-2911" 0x00007f492e024ad3 in ?? ()
46 LWP 29112 "rpc worker-2911" 0x00007f492e024ad3 in ?? ()
47 LWP 29113 "rpc worker-2911" 0x00007f492e024ad3 in ?? ()
48 LWP 29114 "rpc worker-2911" 0x00007f492e024ad3 in ?? ()
49 LWP 29115 "rpc worker-2911" 0x00007f492e024ad3 in ?? ()
50 LWP 29116 "rpc worker-2911" 0x00007f492e024ad3 in ?? ()
51 LWP 29117 "rpc worker-2911" 0x00007f492e024ad3 in ?? ()
52 LWP 29118 "rpc worker-2911" 0x00007f492e024ad3 in ?? ()
53 LWP 29119 "rpc worker-2911" 0x00007f492e024ad3 in ?? ()
54 LWP 29120 "rpc worker-2912" 0x00007f492e024ad3 in ?? ()
55 LWP 29121 "rpc worker-2912" 0x00007f492e024ad3 in ?? ()
56 LWP 29122 "rpc worker-2912" 0x00007f492e024ad3 in ?? ()
57 LWP 29123 "rpc worker-2912" 0x00007f492e024ad3 in ?? ()
58 LWP 29124 "rpc worker-2912" 0x00007f492e024ad3 in ?? ()
59 LWP 29125 "rpc worker-2912" 0x00007f492e024ad3 in ?? ()
60 LWP 29126 "rpc worker-2912" 0x00007f492e024ad3 in ?? ()
61 LWP 29127 "rpc worker-2912" 0x00007f492e024ad3 in ?? ()
62 LWP 29128 "rpc worker-2912" 0x00007f492e024ad3 in ?? ()
63 LWP 29129 "rpc worker-2912" 0x00007f492e024ad3 in ?? ()
64 LWP 29130 "rpc worker-2913" 0x00007f492e024ad3 in ?? ()
65 LWP 29131 "rpc worker-2913" 0x00007f492e024ad3 in ?? ()
66 LWP 29132 "rpc worker-2913" 0x00007f492e024ad3 in ?? ()
67 LWP 29133 "rpc worker-2913" 0x00007f492e024ad3 in ?? ()
68 LWP 29134 "rpc worker-2913" 0x00007f492e024ad3 in ?? ()
69 LWP 29135 "rpc worker-2913" 0x00007f492e024ad3 in ?? ()
70 LWP 29136 "rpc worker-2913" 0x00007f492e024ad3 in ?? ()
71 LWP 29137 "rpc worker-2913" 0x00007f492e024ad3 in ?? ()
72 LWP 29138 "rpc worker-2913" 0x00007f492e024ad3 in ?? ()
73 LWP 29139 "rpc worker-2913" 0x00007f492e024ad3 in ?? ()
74 LWP 29140 "rpc worker-2914" 0x00007f492e024ad3 in ?? ()
75 LWP 29141 "rpc worker-2914" 0x00007f492e024ad3 in ?? ()
76 LWP 29142 "rpc worker-2914" 0x00007f492e024ad3 in ?? ()
77 LWP 29143 "rpc worker-2914" 0x00007f492e024ad3 in ?? ()
78 LWP 29144 "rpc worker-2914" 0x00007f492e024ad3 in ?? ()
79 LWP 29145 "rpc worker-2914" 0x00007f492e024ad3 in ?? ()
80 LWP 29146 "rpc worker-2914" 0x00007f492e024ad3 in ?? ()
81 LWP 29147 "rpc worker-2914" 0x00007f492e024ad3 in ?? ()
82 LWP 29148 "rpc worker-2914" 0x00007f492e024ad3 in ?? ()
83 LWP 29149 "rpc worker-2914" 0x00007f492e024ad3 in ?? ()
84 LWP 29150 "rpc worker-2915" 0x00007f492e024ad3 in ?? ()
85 LWP 29151 "rpc worker-2915" 0x00007f492e024ad3 in ?? ()
86 LWP 29152 "rpc worker-2915" 0x00007f492e024ad3 in ?? ()
87 LWP 29153 "rpc worker-2915" 0x00007f492e024ad3 in ?? ()
88 LWP 29154 "rpc worker-2915" 0x00007f492e024ad3 in ?? ()
89 LWP 29155 "rpc worker-2915" 0x00007f492e024ad3 in ?? ()
90 LWP 29156 "rpc worker-2915" 0x00007f492e024ad3 in ?? ()
91 LWP 29157 "rpc worker-2915" 0x00007f492e024ad3 in ?? ()
92 LWP 29158 "rpc worker-2915" 0x00007f492e024ad3 in ?? ()
93 LWP 29159 "rpc worker-2915" 0x00007f492e024ad3 in ?? ()
94 LWP 29160 "rpc worker-2916" 0x00007f492e024ad3 in ?? ()
95 LWP 29161 "rpc worker-2916" 0x00007f492e024ad3 in ?? ()
96 LWP 29162 "rpc worker-2916" 0x00007f492e024ad3 in ?? ()
97 LWP 29163 "rpc worker-2916" 0x00007f492e024ad3 in ?? ()
98 LWP 29164 "rpc worker-2916" 0x00007f492e024ad3 in ?? ()
99 LWP 29165 "rpc worker-2916" 0x00007f492e024ad3 in ?? ()
100 LWP 29166 "rpc worker-2916" 0x00007f492e024ad3 in ?? ()
101 LWP 29167 "rpc worker-2916" 0x00007f492e024ad3 in ?? ()
102 LWP 29168 "rpc worker-2916" 0x00007f492e024ad3 in ?? ()
103 LWP 29169 "rpc worker-2916" 0x00007f492e024ad3 in ?? ()
104 LWP 29170 "rpc worker-2917" 0x00007f492e024ad3 in ?? ()
105 LWP 29171 "rpc worker-2917" 0x00007f492e024ad3 in ?? ()
106 LWP 29172 "rpc worker-2917" 0x00007f492e024ad3 in ?? ()
107 LWP 29173 "rpc worker-2917" 0x00007f492e024ad3 in ?? ()
108 LWP 29174 "rpc worker-2917" 0x00007f492e024ad3 in ?? ()
109 LWP 29175 "rpc worker-2917" 0x00007f492e024ad3 in ?? ()
110 LWP 29176 "rpc worker-2917" 0x00007f492e024ad3 in ?? ()
111 LWP 29177 "rpc worker-2917" 0x00007f492e024ad3 in ?? ()
112 LWP 29178 "rpc worker-2917" 0x00007f492e024ad3 in ?? ()
113 LWP 29179 "rpc worker-2917" 0x00007f492e024ad3 in ?? ()
114 LWP 29180 "rpc worker-2918" 0x00007f492e024ad3 in ?? ()
115 LWP 29181 "rpc worker-2918" 0x00007f492e024ad3 in ?? ()
116 LWP 29182 "rpc worker-2918" 0x00007f492e024ad3 in ?? ()
117 LWP 29183 "diag-logger-291" 0x00007f492e024fb9 in ?? ()
118 LWP 29184 "result-tracker-" 0x00007f492e024fb9 in ?? ()
119 LWP 29185 "excess-log-dele" 0x00007f492e024fb9 in ?? ()
120 LWP 29186 "acceptor-29186" 0x00007f4926d900c7 in ?? ()
121 LWP 29187 "heartbeat-29187" 0x00007f492e024fb9 in ?? ()
122 LWP 29188 "maintenance_sch" 0x00007f492e024fb9 in ?? ()
123 LWP 29419 "raft [worker]-2" 0x00007f492e024fb9 in ?? ()
124 LWP 29448 "wal-append [wor" 0x00007f492e024fb9 in ?? ()
Thread 124 (LWP 29448):
#0 0x00007f492e024fb9 in ?? ()
#1 0x000060200019dc78 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 123 (LWP 29419):
#0 0x00007f492e024fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000000f4 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007f48e28c8b20 in ?? ()
#5 0x00007f48e28c87b0 in ?? ()
#6 0x00000000000001e8 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 122 (LWP 29188):
#0 0x00007f492e024fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000016 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007f48e30e0550 in ?? ()
#6 0x000000000000002c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 29187):
#0 0x00007f492e024fb9 in ?? ()
#1 0xd35e57c419f6ada3 in ?? ()
#2 0x000000000000000b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019cc0 in ?? ()
#5 0x00007f48e38f85b0 in ?? ()
#6 0x0000000000000016 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 29186):
#0 0x00007f4926d900c7 in ?? ()
#1 0x00006140000220a8 in ?? ()
#2 0x00007f48e411fd50 in ?? ()
#3 0x00007f48e411fd80 in ?? ()
#4 0x00007f48e411fe80 in ?? ()
#5 0x00007f48e411fd70 in ?? ()
#6 0x00007f48e411fde0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x0000000500000014 in ?? ()
#10 0x00007f48e411ff00 in ?? ()
#11 0x00007f48e411f63c in ?? ()
#12 0x00000032e411f5b0 in ?? ()
#13 0x00007f48e3923000 in ?? ()
#14 0x0000000000000000 in ?? ()
Thread 119 (LWP 29185):
#0 0x00007f492e024fb9 in ?? ()
#1 0x00007f48e4939f00 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 29184):
#0 0x00007f492e024fb9 in ?? ()
#1 0x00007f48e51520c0 in ?? ()
#2 0x0000000000000005 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000879b0 in ?? ()
#5 0x00007f48e51520b0 in ?? ()
#6 0x000000000000000a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 29183):
#0 0x00007f492e024fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 29182):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 29181):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 29180):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 29179):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 29178):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 29177):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 29176):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 29175):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 29174):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 29173):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 29172):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 29171):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 29170):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 29169):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 29168):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 29167):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 29166):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 29165):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 29164):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 29163):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 29162):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 29161):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 29160):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 29159):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 29158):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 29157):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 29156):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 29155):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 29154):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 29153):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 29152):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 29151):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 29150):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 29149):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 29148):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 29147):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 29146):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 29145):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 29144):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 29143):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 29142):
#0 0x00007f492e024ad3 in ?? ()
#1 0x00000000000000d4 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001ad088 in ?? ()
#4 0x00007f48fa542e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f48fa542e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 75 (LWP 29141):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000070 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a6888 in ?? ()
#4 0x00007f48fad5ae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f48fad5ae70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 29140):
#0 0x00007f492e024ad3 in ?? ()
#1 0x00000000000000e9 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001a008c in ?? ()
#4 0x00007f48fb572e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f48fb572e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001a0040 in ?? ()
#9 0x00007f492e024770 in ?? ()
#10 0x00007f48fb572e70 in ?? ()
#11 0x00000fe91f6ae5d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 73 (LWP 29139):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000089 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001967cc in ?? ()
#4 0x00007f48fbd8ae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f48fbd8ae70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d000196780 in ?? ()
#9 0x00007f492e024770 in ?? ()
#10 0x00007f48fbd8ae70 in ?? ()
#11 0x00000fe91f7b15d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 72 (LWP 29138):
#0 0x00007f492e024ad3 in ?? ()
#1 0x00000000000000c7 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00018ffcc in ?? ()
#4 0x00007f48fc5a2e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f48fc5a2e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00018ff80 in ?? ()
#9 0x00007f492e024770 in ?? ()
#10 0x00007f48fc5a2e70 in ?? ()
#11 0x00000fe91f8b45d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 71 (LWP 29137):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 29136):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 29135):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 29134):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 29133):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 29132):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 29131):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 29130):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 29129):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 29128):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 29127):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 29126):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 29125):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 29124):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 29123):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 29122):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007f4904723e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4904723e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 29121):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012003c in ?? ()
#4 0x00007f4904f3be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4904f3be70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00011fff0 in ?? ()
#9 0x00007f492e024770 in ?? ()
#10 0x00007f4904f3be70 in ?? ()
#11 0x00000fe9209e77d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 54 (LWP 29120):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00011d04c in ?? ()
#4 0x00007f4905753e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4905753e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00011d000 in ?? ()
#9 0x00007f492e024770 in ?? ()
#10 0x00007f4905753e70 in ?? ()
#11 0x00000fe920aea7d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 53 (LWP 29119):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000116848 in ?? ()
#4 0x00007f4905f6be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f4905f6be70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 52 (LWP 29118):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 29117):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 29116):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 29115):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 29114):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 29113):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 29112):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 29111):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 29110):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 29109):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 29108):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 29107):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 29106):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 29105):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 29104):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 29103):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 29102):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000055 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000a67ec in ?? ()
#4 0x00007f490e904e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f490e904e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000a67a0 in ?? ()
#9 0x00007f492e024770 in ?? ()
#10 0x00007f490e904e70 in ?? ()
#11 0x00000fe921d209d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 35 (LWP 29101):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000004 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00009ffe8 in ?? ()
#4 0x00007f490f11ce50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f490f11ce70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 34 (LWP 29100):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000022 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000967f8 in ?? ()
#4 0x00007f490f934e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f490f934e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 33 (LWP 29099):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000068 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d00008fff8 in ?? ()
#4 0x00007f491014ce50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f491014ce70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 32 (LWP 29098):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 29097):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 29096):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 29095):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 29094):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 29093):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 29092):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 29091):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 29090):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 29089):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 29088):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 29087):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 29086):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 29085):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 29084):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 29083):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 29082):
#0 0x00007f492e024fb9 in ?? ()
#1 0x00007f4918b17c80 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 15 (LWP 29081):
#0 0x00007f492e024fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 29080):
#0 0x00007f492e024fb9 in ?? ()
#1 0x00007f4919b551c0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 13 (LWP 29079):
#0 0x00007f492e024ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 29078):
#0 0x00007f4926d8ea47 in ?? ()
#1 0x00007f491ab942e0 in ?? ()
#2 0x000061a00000f680 in ?? ()
#3 0x00007f491ab942d0 in ?? ()
#4 0x00007f491ab944e0 in ?? ()
#5 0x00007f491ab94320 in ?? ()
#6 0x0000614000022498 in ?? ()
#7 0x00007f491ab943a0 in ?? ()
#8 0x00007f49293ee25d in ?? ()
#9 0x3fb976b3c41c3000 in ?? ()
#10 0x000061a00000f680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f680 in ?? ()
#13 0x000000003f000000 in ?? ()
#14 0x00007f4900000000 in ?? ()
#15 0x41d9e149f1e29a39 in ?? ()
#16 0x00000fe9a356a874 in ?? ()
#17 0x00007f491ab94380 in ?? ()
#18 0x00007f49293f2ba3 in ?? ()
#19 0x00007f491ab94350 in ?? ()
#20 0x3fb976b3c41c3000 in ?? ()
#21 0x000000001ab943a0 in ?? ()
#22 0x000061a00000f680 in ?? ()
#23 0x0000614000022498 in ?? ()
#24 0x3fb976b3c41c3000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 29077):
#0 0x00007f4926d8ea47 in ?? ()
#1 0x00007f491b3ab2e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007f491b3ab2d0 in ?? ()
#4 0x00007f491b3ab4e0 in ?? ()
#5 0x00007f491b3ab320 in ?? ()
#6 0x0000614000022298 in ?? ()
#7 0x00007f491b3ab3a0 in ?? ()
#8 0x00007f49293ee25d in ?? ()
#9 0x3fb9875f8de42000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000070c00000 in ?? ()
#14 0x00007f4900000000 in ?? ()
#15 0x41d9e149f1e29a3f in ?? ()
#16 0x00000fe9a366d674 in ?? ()
#17 0x00007f491b3ab380 in ?? ()
#18 0x00007f49293f2ba3 in ?? ()
#19 0x00007f491b3ab350 in ?? ()
#20 0x3fb9875f8de42000 in ?? ()
#21 0x000000001b3ab3a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000022298 in ?? ()
#24 0x3fb9875f8de42000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 29076):
#0 0x00007f4926d8ea47 in ?? ()
#1 0x00007f491bbb42e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007f491bbb42d0 in ?? ()
#4 0x00007f491bbb44e0 in ?? ()
#5 0x00007f491bbb4320 in ?? ()
#6 0x0000614000022098 in ?? ()
#7 0x00007f491bbb43a0 in ?? ()
#8 0x00007f49293ee25d in ?? ()
#9 0x3fb96818ee0ef000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x00000000f3000000 in ?? ()
#14 0x00007f4900000000 in ?? ()
#15 0x41d9e149f1e29a39 in ?? ()
#16 0x00000fe9a376e874 in ?? ()
#17 0x00007f491bbb4380 in ?? ()
#18 0x00007f49293f2ba3 in ?? ()
#19 0x00007f491bbb4350 in ?? ()
#20 0x3fb96818ee0ef000 in ?? ()
#21 0x000000001bbb43a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000022098 in ?? ()
#24 0x3fb96818ee0ef000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 29075):
#0 0x00007f4926d8ea47 in ?? ()
#1 0x00007f491dbad2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007f491dbad2d0 in ?? ()
#4 0x00007f491dbad4e0 in ?? ()
#5 0x00007f491dbad320 in ?? ()
#6 0x0000614000021e98 in ?? ()
#7 0x00007f491dbad3a0 in ?? ()
#8 0x00007f49293ee25d in ?? ()
#9 0x3fb9512a2109a000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000043000000 in ?? ()
#14 0x00007f4900000000 in ?? ()
#15 0x41d9e149f1e29a3b in ?? ()
#16 0x00000fe9a3b6da74 in ?? ()
#17 0x00007f491dbad380 in ?? ()
#18 0x00007f49293f2ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 29072):
#0 0x00007f4926d81cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f491f5b7788 in ?? ()
#3 0x0000602000015cd0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 29071):
#0 0x00007f492e024fb9 in ?? ()
#1 0x00007f491edb6080 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 29070):
#0 0x00007f492e0289e2 in ?? ()
#1 0x00007f491e5b3b60 in ?? ()
#2 0x00007f491e5b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007f491e5b3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007f491e5b3c10 in ?? ()
#8 0x0000611000088950 in ?? ()
#9 0x00007f491e5b3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007f493394cf0c in ?? ()
#12 0x00007f493393c7b9 in ?? ()
#13 0x00007f4933940da6 in ?? ()
#14 0x00007f49339457e0 in ?? ()
#15 0x00007f4933945609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007f492ac3b0be in ?? ()
#18 0x00007f492e01e6db in ?? ()
#19 0x00000fe9a3cae67c in ?? ()
#20 0x00007f491e5b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 29064):
#0 0x00007f492e024fb9 in ?? ()
#1 0x00007f49205b8c40 in ?? ()
#2 0x000000000000001c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007f49205b8c30 in ?? ()
#6 0x0000000000000038 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 29063):
#0 0x00007f492e024fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 29062):
#0 0x00007f492e024fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 29061):
#0 0x00007f492e024fb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007f4921dbc410 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 29060):
#0 0x00007f492e028d50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250113 14:56:18.427583 28831 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID 12d6ff88cee74893a7e808a1c5b8bba2 and pid 29190
************************ BEGIN STACKS **************************
[New LWP 29191]
[New LWP 29192]
[New LWP 29193]
[New LWP 29194]
[New LWP 29200]
[New LWP 29201]
[New LWP 29202]
[New LWP 29205]
[New LWP 29206]
[New LWP 29207]
[New LWP 29208]
[New LWP 29209]
[New LWP 29210]
[New LWP 29211]
[New LWP 29212]
[New LWP 29213]
[New LWP 29214]
[New LWP 29215]
[New LWP 29216]
[New LWP 29217]
[New LWP 29218]
[New LWP 29219]
[New LWP 29220]
[New LWP 29221]
[New LWP 29222]
[New LWP 29223]
[New LWP 29224]
[New LWP 29225]
[New LWP 29226]
[New LWP 29227]
[New LWP 29228]
[New LWP 29229]
[New LWP 29230]
[New LWP 29231]
[New LWP 29232]
[New LWP 29233]
[New LWP 29234]
[New LWP 29235]
[New LWP 29236]
[New LWP 29237]
[New LWP 29238]
[New LWP 29239]
[New LWP 29240]
[New LWP 29241]
[New LWP 29242]
[New LWP 29243]
[New LWP 29244]
[New LWP 29245]
[New LWP 29246]
[New LWP 29247]
[New LWP 29248]
[New LWP 29249]
[New LWP 29250]
[New LWP 29251]
[New LWP 29252]
[New LWP 29253]
[New LWP 29254]
[New LWP 29255]
[New LWP 29256]
[New LWP 29257]
[New LWP 29258]
[New LWP 29259]
[New LWP 29260]
[New LWP 29261]
[New LWP 29262]
[New LWP 29263]
[New LWP 29264]
[New LWP 29265]
[New LWP 29266]
[New LWP 29267]
[New LWP 29268]
[New LWP 29269]
[New LWP 29270]
[New LWP 29271]
[New LWP 29272]
[New LWP 29273]
[New LWP 29274]
[New LWP 29275]
[New LWP 29276]
[New LWP 29277]
[New LWP 29278]
[New LWP 29279]
[New LWP 29280]
[New LWP 29281]
[New LWP 29282]
[New LWP 29283]
[New LWP 29284]
[New LWP 29285]
[New LWP 29286]
[New LWP 29287]
[New LWP 29288]
[New LWP 29289]
[New LWP 29290]
[New LWP 29291]
[New LWP 29292]
[New LWP 29293]
[New LWP 29294]
[New LWP 29295]
[New LWP 29296]
[New LWP 29297]
[New LWP 29298]
[New LWP 29299]
[New LWP 29300]
[New LWP 29301]
[New LWP 29302]
[New LWP 29303]
[New LWP 29304]
[New LWP 29305]
[New LWP 29306]
[New LWP 29307]
[New LWP 29308]
[New LWP 29309]
[New LWP 29310]
[New LWP 29311]
[New LWP 29312]
[New LWP 29313]
[New LWP 29314]
[New LWP 29315]
[New LWP 29316]
[New LWP 29317]
[New LWP 29318]
[New LWP 29438]
Cannot access memory at address 0x5f35535f35534b51
Cannot access memory at address 0x5f35535f35534b49
Cannot access memory at address 0x5f35535f35534b51
Cannot access memory at address 0x5f35535f35534b51
Cannot access memory at address 0x5f35535f35534b49
0x00007fd69ddbfd50 in ?? ()
Id Target Id Frame
* 1 LWP 29190 "kudu" 0x00007fd69ddbfd50 in ?? ()
2 LWP 29191 "kudu" 0x00007fd69ddbbfb9 in ?? ()
3 LWP 29192 "kudu" 0x00007fd69ddbbfb9 in ?? ()
4 LWP 29193 "kudu" 0x00007fd69ddbbfb9 in ?? ()
5 LWP 29194 "kernel-watcher-" 0x00007fd69ddbbfb9 in ?? ()
6 LWP 29200 "ntp client-2920" 0x00007fd69ddbf9e2 in ?? ()
7 LWP 29201 "file cache-evic" 0x00007fd69ddbbfb9 in ?? ()
8 LWP 29202 "sq_acceptor" 0x00007fd696b18cb9 in ?? ()
9 LWP 29205 "rpc reactor-292" 0x00007fd696b25a47 in ?? ()
10 LWP 29206 "rpc reactor-292" 0x00007fd696b25a47 in ?? ()
11 LWP 29207 "rpc reactor-292" 0x00007fd696b25a47 in ?? ()
12 LWP 29208 "rpc reactor-292" 0x00007fd696b25a47 in ?? ()
13 LWP 29209 "MaintenanceMgr " 0x00007fd69ddbbad3 in ?? ()
14 LWP 29210 "txn-status-mana" 0x00007fd69ddbbfb9 in ?? ()
15 LWP 29211 "collect_and_rem" 0x00007fd69ddbbfb9 in ?? ()
16 LWP 29212 "tc-session-exp-" 0x00007fd69ddbbfb9 in ?? ()
17 LWP 29213 "rpc worker-2921" 0x00007fd69ddbbad3 in ?? ()
18 LWP 29214 "rpc worker-2921" 0x00007fd69ddbbad3 in ?? ()
19 LWP 29215 "rpc worker-2921" 0x00007fd69ddbbad3 in ?? ()
20 LWP 29216 "rpc worker-2921" 0x00007fd69ddbbad3 in ?? ()
21 LWP 29217 "rpc worker-2921" 0x00007fd69ddbbad3 in ?? ()
22 LWP 29218 "rpc worker-2921" 0x00007fd69ddbbad3 in ?? ()
23 LWP 29219 "rpc worker-2921" 0x00007fd69ddbbad3 in ?? ()
24 LWP 29220 "rpc worker-2922" 0x00007fd69ddbbad3 in ?? ()
25 LWP 29221 "rpc worker-2922" 0x00007fd69ddbbad3 in ?? ()
26 LWP 29222 "rpc worker-2922" 0x00007fd69ddbbad3 in ?? ()
27 LWP 29223 "rpc worker-2922" 0x00007fd69ddbbad3 in ?? ()
28 LWP 29224 "rpc worker-2922" 0x00007fd69ddbbad3 in ?? ()
29 LWP 29225 "rpc worker-2922" 0x00007fd69ddbbad3 in ?? ()
30 LWP 29226 "rpc worker-2922" 0x00007fd69ddbbad3 in ?? ()
31 LWP 29227 "rpc worker-2922" 0x00007fd69ddbbad3 in ?? ()
32 LWP 29228 "rpc worker-2922" 0x00007fd69ddbbad3 in ?? ()
33 LWP 29229 "rpc worker-2922" 0x00007fd69ddbbad3 in ?? ()
34 LWP 29230 "rpc worker-2923" 0x00007fd69ddbbad3 in ?? ()
35 LWP 29231 "rpc worker-2923" 0x00007fd69ddbbad3 in ?? ()
36 LWP 29232 "rpc worker-2923" 0x00007fd69ddbbad3 in ?? ()
37 LWP 29233 "rpc worker-2923" 0x00007fd69ddbbad3 in ?? ()
38 LWP 29234 "rpc worker-2923" 0x00007fd69ddbbad3 in ?? ()
39 LWP 29235 "rpc worker-2923" 0x00007fd69ddbbad3 in ?? ()
40 LWP 29236 "rpc worker-2923" 0x00007fd69ddbbad3 in ?? ()
41 LWP 29237 "rpc worker-2923" 0x00007fd69ddbbad3 in ?? ()
42 LWP 29238 "rpc worker-2923" 0x00007fd69ddbbad3 in ?? ()
43 LWP 29239 "rpc worker-2923" 0x00007fd69ddbbad3 in ?? ()
44 LWP 29240 "rpc worker-2924" 0x00007fd69ddbbad3 in ?? ()
45 LWP 29241 "rpc worker-2924" 0x00007fd69ddbbad3 in ?? ()
46 LWP 29242 "rpc worker-2924" 0x00007fd69ddbbad3 in ?? ()
47 LWP 29243 "rpc worker-2924" 0x00007fd69ddbbad3 in ?? ()
48 LWP 29244 "rpc worker-2924" 0x00007fd69ddbbad3 in ?? ()
49 LWP 29245 "rpc worker-2924" 0x00007fd69ddbbad3 in ?? ()
50 LWP 29246 "rpc worker-2924" 0x00007fd69ddbbad3 in ?? ()
51 LWP 29247 "rpc worker-2924" 0x00007fd69ddbbad3 in ?? ()
52 LWP 29248 "rpc worker-2924" 0x00007fd69ddbbad3 in ?? ()
53 LWP 29249 "rpc worker-2924" 0x00007fd69ddbbad3 in ?? ()
54 LWP 29250 "rpc worker-2925" 0x00007fd69ddbbad3 in ?? ()
55 LWP 29251 "rpc worker-2925" 0x00007fd69ddbbad3 in ?? ()
56 LWP 29252 "rpc worker-2925" 0x00007fd69ddbbad3 in ?? ()
57 LWP 29253 "rpc worker-2925" 0x00007fd69ddbbad3 in ?? ()
58 LWP 29254 "rpc worker-2925" 0x00007fd69ddbbad3 in ?? ()
59 LWP 29255 "rpc worker-2925" 0x00007fd69ddbbad3 in ?? ()
60 LWP 29256 "rpc worker-2925" 0x00007fd69ddbbad3 in ?? ()
61 LWP 29257 "rpc worker-2925" 0x00007fd69ddbbad3 in ?? ()
62 LWP 29258 "rpc worker-2925" 0x00007fd69ddbbad3 in ?? ()
63 LWP 29259 "rpc worker-2925" 0x00007fd69ddbbad3 in ?? ()
64 LWP 29260 "rpc worker-2926" 0x00007fd69ddbbad3 in ?? ()
65 LWP 29261 "rpc worker-2926" 0x00007fd69ddbbad3 in ?? ()
66 LWP 29262 "rpc worker-2926" 0x00007fd69ddbbad3 in ?? ()
67 LWP 29263 "rpc worker-2926" 0x00007fd69ddbbad3 in ?? ()
68 LWP 29264 "rpc worker-2926" 0x00007fd69ddbbad3 in ?? ()
69 LWP 29265 "rpc worker-2926" 0x00007fd69ddbbad3 in ?? ()
70 LWP 29266 "rpc worker-2926" 0x00007fd69ddbbad3 in ?? ()
71 LWP 29267 "rpc worker-2926" 0x00007fd69ddbbad3 in ?? ()
72 LWP 29268 "rpc worker-2926" 0x00007fd69ddbbad3 in ?? ()
73 LWP 29269 "rpc worker-2926" 0x00007fd69ddbbad3 in ?? ()
74 LWP 29270 "rpc worker-2927" 0x00007fd69ddbbad3 in ?? ()
75 LWP 29271 "rpc worker-2927" 0x00007fd69ddbbad3 in ?? ()
76 LWP 29272 "rpc worker-2927" 0x00007fd69ddbbad3 in ?? ()
77 LWP 29273 "rpc worker-2927" 0x00007fd69ddbbad3 in ?? ()
78 LWP 29274 "rpc worker-2927" 0x00007fd69ddbbad3 in ?? ()
79 LWP 29275 "rpc worker-2927" 0x00007fd69ddbbad3 in ?? ()
80 LWP 29276 "rpc worker-2927" 0x00007fd69ddbbad3 in ?? ()
81 LWP 29277 "rpc worker-2927" 0x00007fd69ddbbad3 in ?? ()
82 LWP 29278 "rpc worker-2927" 0x00007fd69ddbbad3 in ?? ()
83 LWP 29279 "rpc worker-2927" 0x00007fd69ddbbad3 in ?? ()
84 LWP 29280 "rpc worker-2928" 0x00007fd69ddbbad3 in ?? ()
85 LWP 29281 "rpc worker-2928" 0x00007fd69ddbbad3 in ?? ()
86 LWP 29282 "rpc worker-2928" 0x00007fd69ddbbad3 in ?? ()
87 LWP 29283 "rpc worker-2928" 0x00007fd69ddbbad3 in ?? ()
88 LWP 29284 "rpc worker-2928" 0x00007fd69ddbbad3 in ?? ()
89 LWP 29285 "rpc worker-2928" 0x00007fd69ddbbad3 in ?? ()
90 LWP 29286 "rpc worker-2928" 0x00007fd69ddbbad3 in ?? ()
91 LWP 29287 "rpc worker-2928" 0x00007fd69ddbbad3 in ?? ()
92 LWP 29288 "rpc worker-2928" 0x00007fd69ddbbad3 in ?? ()
93 LWP 29289 "rpc worker-2928" 0x00007fd69ddbbad3 in ?? ()
94 LWP 29290 "rpc worker-2929" 0x00007fd69ddbbad3 in ?? ()
95 LWP 29291 "rpc worker-2929" 0x00007fd69ddbbad3 in ?? ()
96 LWP 29292 "rpc worker-2929" 0x00007fd69ddbbad3 in ?? ()
97 LWP 29293 "rpc worker-2929" 0x00007fd69ddbbad3 in ?? ()
98 LWP 29294 "rpc worker-2929" 0x00007fd69ddbbad3 in ?? ()
99 LWP 29295 "rpc worker-2929" 0x00007fd69ddbbad3 in ?? ()
100 LWP 29296 "rpc worker-2929" 0x00007fd69ddbbad3 in ?? ()
101 LWP 29297 "rpc worker-2929" 0x00007fd69ddbbad3 in ?? ()
102 LWP 29298 "rpc worker-2929" 0x00007fd69ddbbad3 in ?? ()
103 LWP 29299 "rpc worker-2929" 0x00007fd69ddbbad3 in ?? ()
104 LWP 29300 "rpc worker-2930" 0x00007fd69ddbbad3 in ?? ()
105 LWP 29301 "rpc worker-2930" 0x00007fd69ddbbad3 in ?? ()
106 LWP 29302 "rpc worker-2930" 0x00007fd69ddbbad3 in ?? ()
107 LWP 29303 "rpc worker-2930" 0x00007fd69ddbbad3 in ?? ()
108 LWP 29304 "rpc worker-2930" 0x00007fd69ddbbad3 in ?? ()
109 LWP 29305 "rpc worker-2930" 0x00007fd69ddbbad3 in ?? ()
110 LWP 29306 "rpc worker-2930" 0x00007fd69ddbbad3 in ?? ()
111 LWP 29307 "rpc worker-2930" 0x00007fd69ddbbad3 in ?? ()
112 LWP 29308 "rpc worker-2930" 0x00007fd69ddbbad3 in ?? ()
113 LWP 29309 "rpc worker-2930" 0x00007fd69ddbbad3 in ?? ()
114 LWP 29310 "rpc worker-2931" 0x00007fd69ddbbad3 in ?? ()
115 LWP 29311 "rpc worker-2931" 0x00007fd69ddbbad3 in ?? ()
116 LWP 29312 "rpc worker-2931" 0x00007fd69ddbbad3 in ?? ()
117 LWP 29313 "diag-logger-293" 0x00007fd69ddbbfb9 in ?? ()
118 LWP 29314 "result-tracker-" 0x00007fd69ddbbfb9 in ?? ()
119 LWP 29315 "excess-log-dele" 0x00007fd69ddbbfb9 in ?? ()
120 LWP 29316 "acceptor-29316" 0x00007fd696b270c7 in ?? ()
121 LWP 29317 "heartbeat-29317" 0x00007fd69ddbbfb9 in ?? ()
122 LWP 29318 "maintenance_sch" 0x00007fd69ddbbfb9 in ?? ()
123 LWP 29438 "raft [worker]-2" 0x00007fd69ddbbfb9 in ?? ()
Thread 123 (LWP 29438):
#0 0x00007fd69ddbbfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000023 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x00007fd6515dfb24 in ?? ()
#5 0x00007fd6515df7b0 in ?? ()
#6 0x0000000000000047 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007fd6515df7d0 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 122 (LWP 29318):
#0 0x00007fd69ddbbfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000016 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000616000017580 in ?? ()
#5 0x00007fd652e77550 in ?? ()
#6 0x000000000000002c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 29317):
#0 0x00007fd69ddbbfb9 in ?? ()
#1 0xd35e57c419f6ada3 in ?? ()
#2 0x000000000000000d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000019cc0 in ?? ()
#5 0x00007fd65368f5b0 in ?? ()
#6 0x000000000000001a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 120 (LWP 29316):
#0 0x00007fd696b270c7 in ?? ()
#1 0x00006140000220a8 in ?? ()
#2 0x00007fd653eb6d50 in ?? ()
#3 0x00007fd653eb6d80 in ?? ()
#4 0x00007fd653eb6e80 in ?? ()
#5 0x00007fd653eb6d70 in ?? ()
#6 0x00007fd653eb6de0 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x00000000008acceb in __sanitizer::theDepot ()
#9 0x0000000500000014 in ?? ()
#10 0x00007fd653eb6f00 in ?? ()
#11 0x00007fd653eb663c in ?? ()
#12 0x0000003253eb65b0 in ?? ()
#13 0x00007fd6536ba000 in ?? ()
#14 0x0000000000000000 in ?? ()
Thread 119 (LWP 29315):
#0 0x00007fd69ddbbfb9 in ?? ()
#1 0x00007fd6546d0f00 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 118 (LWP 29314):
#0 0x00007fd69ddbbfb9 in ?? ()
#1 0x00007fd654ee90c0 in ?? ()
#2 0x0000000000000005 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00006110000879b0 in ?? ()
#5 0x00007fd654ee90b0 in ?? ()
#6 0x000000000000000a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 117 (LWP 29313):
#0 0x00007fd69ddbbfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 29312):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 29311):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 29310):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 29309):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 29308):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 29307):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 29306):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 29305):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 29304):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 29303):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 29302):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 29301):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 29300):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 29299):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 29298):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 29297):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 29296):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 29295):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 29294):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 29293):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 29292):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 29291):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 29290):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 29289):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 29288):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 29287):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 29286):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 29285):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 29284):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 29283):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 29282):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 29281):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 29280):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 29279):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 29278):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 29277):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 29276):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 79 (LWP 29275):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 29274):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 29273):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 29272):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x00000000000000f3 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001ad08c in ?? ()
#4 0x00007fd66a2d9e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd66a2d9e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001ad040 in ?? ()
#9 0x00007fd69ddbb770 in ?? ()
#10 0x00007fd66a2d9e70 in ?? ()
#11 0x00000ffacd45b3d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 75 (LWP 29271):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x00000000000000c0 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a6888 in ?? ()
#4 0x00007fd66aaf1e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd66aaf1e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 74 (LWP 29270):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x000000000000003e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001a0088 in ?? ()
#4 0x00007fd66b309e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd66b309e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 73 (LWP 29269):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 29268):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 29267):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 29266):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 29265):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 29264):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 29263):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 29262):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 29261):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 29260):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 29259):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 29258):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 29257):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 29256):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 59 (LWP 29255):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 29254):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 29253):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 29252):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000126838 in ?? ()
#4 0x00007fd6744bae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd6744bae70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 55 (LWP 29251):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00012003c in ?? ()
#4 0x00007fd674cd2e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd674cd2e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00011fff0 in ?? ()
#9 0x00007fd69ddbb770 in ?? ()
#10 0x00007fd674cd2e70 in ?? ()
#11 0x00000fface99a5d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 54 (LWP 29250):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00011d04c in ?? ()
#4 0x00007fd6754eae50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd6754eae70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00011d000 in ?? ()
#9 0x00007fd69ddbb770 in ?? ()
#10 0x00007fd6754eae70 in ?? ()
#11 0x00000ffacea9d5d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 53 (LWP 29249):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000116848 in ?? ()
#4 0x00007fd675d02e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd675d02e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 52 (LWP 29248):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 29247):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 29246):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 29245):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 29244):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 29243):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 29242):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 29241):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 29240):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 29239):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 29238):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 29237):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 29236):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 29235):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 29234):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 29233):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 29232):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x00000000000000bc in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0000a67e8 in ?? ()
#4 0x00007fd67e69be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd67e69be70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 35 (LWP 29231):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00009ffec in ?? ()
#4 0x00007fd67eeb3e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd67eeb3e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00009ffa0 in ?? ()
#9 0x00007fd69ddbb770 in ?? ()
#10 0x00007fd67eeb3e70 in ?? ()
#11 0x00000ffacfdd67d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 34 (LWP 29230):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0000967fc in ?? ()
#4 0x00007fd67f6cbe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd67f6cbe70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000967b0 in ?? ()
#9 0x00007fd69ddbb770 in ?? ()
#10 0x00007fd67f6cbe70 in ?? ()
#11 0x00000ffacfed97d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 33 (LWP 29229):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00008fffc in ?? ()
#4 0x00007fd67fee3e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd67fee3e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00008ffb0 in ?? ()
#9 0x00007fd69ddbb770 in ?? ()
#10 0x00007fd67fee3e70 in ?? ()
#11 0x00000ffacffdc7d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 32 (LWP 29228):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x000000000000003f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00008d00c in ?? ()
#4 0x00007fd6806fbe50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd6806fbe70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00008cfc0 in ?? ()
#9 0x00007fd69ddbb770 in ?? ()
#10 0x00007fd6806fbe70 in ?? ()
#11 0x00000ffad00df7d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 31 (LWP 29227):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00008680c in ?? ()
#4 0x00007fd680f13e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd680f13e70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0000867c0 in ?? ()
#9 0x00007fd69ddbb770 in ?? ()
#10 0x00007fd680f13e70 in ?? ()
#11 0x00000ffad01e27d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 30 (LWP 29226):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x000000000000009f in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00008000c in ?? ()
#4 0x00007fd68172be50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd68172be70 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00007ffc0 in ?? ()
#9 0x00007fd69ddbb770 in ?? ()
#10 0x00007fd68172be70 in ?? ()
#11 0x00000ffad02e57d0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 29 (LWP 29225):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x000000000000001e in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000076818 in ?? ()
#4 0x00007fd681f43e50 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007fd681f43e70 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 28 (LWP 29224):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 29223):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 29222):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 29221):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 29220):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 29219):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 29218):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 29217):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 29216):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 19 (LWP 29215):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 29214):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 29213):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 16 (LWP 29212):
#0 0x00007fd69ddbbfb9 in ?? ()
#1 0x00007fd6888adc80 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 15 (LWP 29211):
#0 0x00007fd69ddbbfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 14 (LWP 29210):
#0 0x00007fd69ddbbfb9 in ?? ()
#1 0x00007fd6898af1c0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 13 (LWP 29209):
#0 0x00007fd69ddbbad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 29208):
#0 0x00007fd696b25a47 in ?? ()
#1 0x00007fd68a8b22e0 in ?? ()
#2 0x000061a00000f680 in ?? ()
#3 0x00007fd68a8b22d0 in ?? ()
#4 0x00007fd68a8b24e0 in ?? ()
#5 0x00007fd68a8b2320 in ?? ()
#6 0x0000614000022498 in ?? ()
#7 0x00007fd68a8b23a0 in ?? ()
#8 0x00007fd69918525d in ?? ()
#9 0x3fb88cc6b7e14000 in ?? ()
#10 0x000061a00000f680 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f680 in ?? ()
#13 0x00000000eaf00000 in ?? ()
#14 0x00007fd600000000 in ?? ()
#15 0x41d9e149f1e29a3b in ?? ()
#16 0x00000ffb5150e474 in ?? ()
#17 0x00007fd68a8b2380 in ?? ()
#18 0x00007fd699189ba3 in ?? ()
#19 0x00007fd68a8b2350 in ?? ()
#20 0x3fb88cc6b7e14000 in ?? ()
#21 0x000000008a8b23a0 in ?? ()
#22 0x000061a00000f680 in ?? ()
#23 0x0000614000022498 in ?? ()
#24 0x3fb88cc6b7e14000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 29207):
#0 0x00007fd696b25a47 in ?? ()
#1 0x00007fd68b0b32e0 in ?? ()
#2 0x000061a00000f080 in ?? ()
#3 0x00007fd68b0b32d0 in ?? ()
#4 0x00007fd68b0b34e0 in ?? ()
#5 0x00007fd68b0b3320 in ?? ()
#6 0x0000614000022298 in ?? ()
#7 0x00007fd68b0b33a0 in ?? ()
#8 0x00007fd69918525d in ?? ()
#9 0x3fb8efbceb1da000 in ?? ()
#10 0x000061a00000f080 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000f080 in ?? ()
#13 0x0000000054000000 in ?? ()
#14 0x00007fd600000000 in ?? ()
#15 0x41d9e149f1e29a38 in ?? ()
#16 0x00000ffb5160e674 in ?? ()
#17 0x00007fd68b0b3380 in ?? ()
#18 0x00007fd699189ba3 in ?? ()
#19 0x00007fd68b0b3350 in ?? ()
#20 0x3fb8efbceb1da000 in ?? ()
#21 0x000000008b0b33a0 in ?? ()
#22 0x000061a00000f080 in ?? ()
#23 0x0000614000022298 in ?? ()
#24 0x3fb8efbceb1da000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 29206):
#0 0x00007fd696b25a47 in ?? ()
#1 0x00007fd68b8b42e0 in ?? ()
#2 0x000061a00000ea80 in ?? ()
#3 0x00007fd68b8b42d0 in ?? ()
#4 0x00007fd68b8b44e0 in ?? ()
#5 0x00007fd68b8b4320 in ?? ()
#6 0x0000614000022098 in ?? ()
#7 0x00007fd68b8b43a0 in ?? ()
#8 0x00007fd69918525d in ?? ()
#9 0x3fb97fc310af4000 in ?? ()
#10 0x000061a00000ea80 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000ea80 in ?? ()
#13 0x000000001d000000 in ?? ()
#14 0x00007fd600000000 in ?? ()
#15 0x41d9e149f1e29a38 in ?? ()
#16 0x00000ffb5170e874 in ?? ()
#17 0x00007fd68b8b4380 in ?? ()
#18 0x00007fd699189ba3 in ?? ()
#19 0x00007fd68b8b4350 in ?? ()
#20 0x3fb97fc310af4000 in ?? ()
#21 0x000000008b8b43a0 in ?? ()
#22 0x000061a00000ea80 in ?? ()
#23 0x0000614000022098 in ?? ()
#24 0x3fb97fc310af4000 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 29205):
#0 0x00007fd696b25a47 in ?? ()
#1 0x00007fd68d8ad2e0 in ?? ()
#2 0x000061a00000e480 in ?? ()
#3 0x00007fd68d8ad2d0 in ?? ()
#4 0x00007fd68d8ad4e0 in ?? ()
#5 0x00007fd68d8ad320 in ?? ()
#6 0x0000614000021e98 in ?? ()
#7 0x00007fd68d8ad3a0 in ?? ()
#8 0x00007fd69918525d in ?? ()
#9 0x3fb955733dd58000 in ?? ()
#10 0x000061a00000e480 in ?? ()
#11 0x42a2309ce5400000 in ?? ()
#12 0x000061a00000e480 in ?? ()
#13 0x0000000068000000 in ?? ()
#14 0x00007fd600000000 in ?? ()
#15 0x41d9e149f1e29a39 in ?? ()
#16 0x00000ffb51b0da74 in ?? ()
#17 0x00007fd68d8ad380 in ?? ()
#18 0x00007fd699189ba3 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 29202):
#0 0x00007fd696b18cb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007fd68f2b7788 in ?? ()
#3 0x00006020000159d0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x00000000008a4931 in __sanitizer::theDepot ()
#7 0x0000000000000000 in ?? ()
Thread 7 (LWP 29201):
#0 0x00007fd69ddbbfb9 in ?? ()
#1 0x00007fd68eab6080 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 29200):
#0 0x00007fd69ddbf9e2 in ?? ()
#1 0x00007fd68e2b3b60 in ?? ()
#2 0x00007fd68e2b3c00 in ?? ()
#3 0x000000000000001e in ?? ()
#4 0x0000000000000030 in ?? ()
#5 0x00007fd68e2b3bb0 in ?? ()
#6 0x000000000177bae0 in ?? ()
#7 0x00007fd68e2b3c10 in ?? ()
#8 0x0000611000088950 in ?? ()
#9 0x00007fd68e2b3c00 in ?? ()
#10 0x000000000089ee27 in __sanitizer::theDepot ()
#11 0x00007fd6a36e3f0c in ?? ()
#12 0x00007fd6a36d37b9 in ?? ()
#13 0x00007fd6a36d7da6 in ?? ()
#14 0x00007fd6a36dc7e0 in ?? ()
#15 0x00007fd6a36dc609 in ?? ()
#16 0x0000000000a71ecd in __sanitizer::theDepot ()
#17 0x00007fd69a9d20be in ?? ()
#18 0x00007fd69ddb56db in ?? ()
#19 0x00000ffb51c4e67c in ?? ()
#20 0x00007fd68e2b3400 in ?? ()
#21 0x0000000000000000 in ?? ()
Thread 5 (LWP 29194):
#0 0x00007fd69ddbbfb9 in ?? ()
#1 0x00007fd6902b8c40 in ?? ()
#2 0x000000000000001c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200000f728 in ?? ()
#5 0x00007fd6902b8c30 in ?? ()
#6 0x0000000000000038 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 29193):
#0 0x00007fd69ddbbfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 29192):
#0 0x00007fd69ddbbfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 29191):
#0 0x00007fd69ddbbfb9 in ?? ()
#1 0x5f5347414c46000a in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000016eb8 in ?? ()
#5 0x00007fd691abc410 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 29190):
#0 0x00007fd69ddbfd50 in ?? ()
#1 0x0000000000000000 in ?? ()
************************* END STACKS ***************************
I20250113 14:56:19.005744 28831 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task19rJWQ/build/asan/bin/kudu pid 28929
I20250113 14:56:19.014276 29051 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 14:56:19.150063 28831 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task19rJWQ/build/asan/bin/kudu with pid 28929
W20250113 14:56:19.172873 29207 connection.cc:537] server connection from 127.28.39.193:34111 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 14:56:19.173532 28831 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task19rJWQ/build/asan/bin/kudu pid 29060
I20250113 14:56:19.174649 29182 generic_service.cc:196] Checking for leaks (request via RPC)
W20250113 14:56:19.254887 29207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.39.193:38947: connect: Connection refused (error 111)
W20250113 14:56:19.257704 29207 consensus_peers.cc:487] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 -> Peer 43263e8341a8434891e23c68568253de (127.28.39.193:38947): Couldn't send request to peer 43263e8341a8434891e23c68568253de. Status: Network error: Client connection negotiation failed: client connection to 127.28.39.193:38947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250113 14:56:19.263108 29207 consensus_peers.cc:487] T 596da1f3892f4daf9a4f308ac6f21949 P 12d6ff88cee74893a7e808a1c5b8bba2 -> Peer 43263e8341a8434891e23c68568253de (127.28.39.193:38947): Couldn't send request to peer 43263e8341a8434891e23c68568253de. Status: Network error: Client connection negotiation failed: client connection to 127.28.39.193:38947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 14:56:19.307453 28831 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task19rJWQ/build/asan/bin/kudu with pid 29060
W20250113 14:56:19.330174 29208 connection.cc:537] client connection to 127.28.39.194:44385 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250113 14:56:19.330175 29205 connection.cc:537] server connection from 127.28.39.194:56269 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250113 14:56:19.330782 28831 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task19rJWQ/build/asan/bin/kudu pid 29190
W20250113 14:56:19.331357 29208 consensus_peers.cc:487] T b46a2c13dda4486892efa9475b7b3032 P 12d6ff88cee74893a7e808a1c5b8bba2 -> Peer f7ec7575f7d64455a4ecaa314f439592 (127.28.39.194:44385): Couldn't send request to peer f7ec7575f7d64455a4ecaa314f439592. Status: Network error: Client connection negotiation failed: client connection to 127.28.39.194:44385: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250113 14:56:19.337435 29312 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 14:56:19.470693 28831 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task19rJWQ/build/asan/bin/kudu with pid 29190
I20250113 14:56:19.494202 28831 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task19rJWQ/build/asan/bin/kudu pid 28839
I20250113 14:56:19.495357 28900 generic_service.cc:196] Checking for leaks (request via RPC)
I20250113 14:56:19.603819 28831 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task19rJWQ/build/asan/bin/kudu with pid 28839
2025-01-13T14:56:19Z chronyd exiting
I20250113 14:56:19.638320 28831 test_util.cc:181] -----------------------------------------------
I20250113 14:56:19.638406 28831 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-task19rJWQ/test-tmp/replace_tablet-itest.0.ReplaceTabletITest.ReplaceTabletsWhileWriting.1736780171237853-28831-0
[ FAILED ] ReplaceTabletITest.ReplaceTabletsWhileWriting (8328 ms)
[----------] 1 test from ReplaceTabletITest (8328 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (8329 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] ReplaceTabletITest.ReplaceTabletsWhileWriting
1 FAILED TEST
I20250113 14:56:19.639806 28831 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/batcher.cc:441: suppressed but not reported on 3 messages since previous log ~2 seconds ago