[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from TxnStatusManagerITest
[ RUN ] TxnStatusManagerITest.StaleTransactionsCleanup
2026-08-07T22:42:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:42:34Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260807 22:42:34.319078 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.103.62:39909
--webserver_interface=127.4.103.62
--webserver_port=0
--builtin_ntp_servers=127.4.103.20:32827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.103.62:39909
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false with env {}
W20260807 22:42:34.774755 4525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:42:34.775318 4525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:42:34.775509 4525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:42:34.784649 4525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:42:34.784847 4525 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 22:42:34.784893 4525 flags.cc:425] Enabled experimental flag: --txn_manager_lazily_initialized=false
W20260807 22:42:34.785033 4525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:42:34.785116 4525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:42:34.785238 4525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:42:34.796926 4525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:32827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.103.62:39909
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.103.62:39909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.103.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:42:34.799544 4525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:42:34.802675 4525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:42:34.829077 4535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:34.829078 4532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:34.831421 4531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:42:34.832396 4525 server_base.cc:1034] running on GCE node
I20260807 22:42:34.833853 4525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:42:34.836850 4525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:42:34.838382 4525 hybrid_clock.cc:648] HybridClock initialized: now 1786142554838253 us; error 82 us; skew 500 ppm
I20260807 22:42:34.839054 4525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:42:34.847096 4525 webserver.cc:460] Webserver started at http://127.4.103.62:39209/ using document root <none> and password file <none>
I20260807 22:42:34.848757 4525 fs_manager.cc:359] Metadata directory not provided
I20260807 22:42:34.848953 4525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:42:34.849586 4525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:42:34.854308 4525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/data/instance:
uuid: "ac824328c84e4a26b8aabb987b1965c1"
format_stamp: "Formatted at 2026-08-07 22:42:34 on dist-test-slave-kvfs"
I20260807 22:42:34.855360 4525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/wal/instance:
uuid: "ac824328c84e4a26b8aabb987b1965c1"
format_stamp: "Formatted at 2026-08-07 22:42:34 on dist-test-slave-kvfs"
I20260807 22:42:34.866123 4525 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.001s
I20260807 22:42:34.873190 4543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:34.875828 4525 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260807 22:42:34.876220 4525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/wal
uuid: "ac824328c84e4a26b8aabb987b1965c1"
format_stamp: "Formatted at 2026-08-07 22:42:34 on dist-test-slave-kvfs"
I20260807 22:42:34.876492 4525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:42:34.899350 4525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:42:34.901469 4525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:42:34.902004 4525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:42:34.938107 4525 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.62:39909
I20260807 22:42:34.938129 4640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.62:39909 every 8 connection(s)
I20260807 22:42:34.941097 4525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/data/info.pb
I20260807 22:42:34.942906 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 4525
I20260807 22:42:34.943608 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/master-0/wal/instance
I20260807 22:42:34.954589 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:42:34.981719 4642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1: Bootstrap starting.
I20260807 22:42:34.988890 4642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1: Neither blocks nor log segments found. Creating new log.
I20260807 22:42:34.990869 4642 log.cc:824] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1: Log is configured to *not* fsync() on all Append() calls
I20260807 22:42:34.996948 4642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1: No bootstrap required, opened a new log
I20260807 22:42:35.008765 4642 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac824328c84e4a26b8aabb987b1965c1" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 39909 } }
I20260807 22:42:35.009595 4642 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:42:35.009774 4642 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac824328c84e4a26b8aabb987b1965c1, State: Initialized, Role: FOLLOWER
I20260807 22:42:35.011142 4642 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac824328c84e4a26b8aabb987b1965c1" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 39909 } }
I20260807 22:42:35.011535 4642 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:42:35.011806 4642 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:42:35.012091 4642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:42:35.015935 4642 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac824328c84e4a26b8aabb987b1965c1" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 39909 } }
I20260807 22:42:35.017156 4642 leader_election.cc:302] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac824328c84e4a26b8aabb987b1965c1; no voters:
I20260807 22:42:35.018137 4642 leader_election.cc:288] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:42:35.018451 4651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:42:35.020038 4651 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [term 1 LEADER]: Becoming Leader. State: Replica: ac824328c84e4a26b8aabb987b1965c1, State: Running, Role: LEADER
I20260807 22:42:35.021108 4651 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac824328c84e4a26b8aabb987b1965c1" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 39909 } }
I20260807 22:42:35.022128 4642 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:42:35.029285 4653 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac824328c84e4a26b8aabb987b1965c1. Latest consensus state: current_term: 1 leader_uuid: "ac824328c84e4a26b8aabb987b1965c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac824328c84e4a26b8aabb987b1965c1" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 39909 } } }
I20260807 22:42:35.032680 4653 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [sys.catalog]: This master's current role is: LEADER
I20260807 22:42:35.029053 4652 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac824328c84e4a26b8aabb987b1965c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac824328c84e4a26b8aabb987b1965c1" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 39909 } } }
I20260807 22:42:35.034231 4652 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1 [sys.catalog]: This master's current role is: LEADER
I20260807 22:42:35.037776 4662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:42:35.050009 4662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:42:35.074898 4662 catalog_manager.cc:1347] Generated new cluster ID: 1a12a7f736634d02bc185cf48bfb1a5c
I20260807 22:42:35.075163 4662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:42:35.094630 4662 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:42:35.097885 4662 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:42:35.119356 4662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac824328c84e4a26b8aabb987b1965c1: Generated new TSK 0
I20260807 22:42:35.121532 4662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:42:35.135529 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.1:0
--local_ip_for_outbound_sockets=127.4.103.1
--webserver_interface=127.4.103.1
--webserver_port=0
--tserver_master_addrs=127.4.103.62:39909
--builtin_ntp_servers=127.4.103.20:32827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
I20260807 22:42:35.144909 4572 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45946:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:42:35.167474 4642 master.cc:440] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 3; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20260807 22:42:35.679870 4685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:42:35.680371 4685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:42:35.680555 4685 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:42:35.680687 4685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:42:35.690431 4685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:42:35.690610 4685 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:42:35.690759 4685 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:42:35.690841 4685 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:42:35.691009 4685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.1
I20260807 22:42:35.702533 4685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:32827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.103.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:39909
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.1
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:42:35.704591 4685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:42:35.707086 4685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:42:35.725942 4692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:35.725821 4693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:35.725883 4695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:42:35.727545 4685 server_base.cc:1034] running on GCE node
I20260807 22:42:35.728546 4685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:42:35.729897 4685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:42:35.731171 4685 hybrid_clock.cc:648] HybridClock initialized: now 1786142555731037 us; error 73 us; skew 500 ppm
I20260807 22:42:35.731684 4685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:42:35.735337 4685 webserver.cc:460] Webserver started at http://127.4.103.1:35133/ using document root <none> and password file <none>
I20260807 22:42:35.736492 4685 fs_manager.cc:359] Metadata directory not provided
I20260807 22:42:35.736631 4685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:42:35.737094 4685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:42:35.740952 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/data/instance:
uuid: "911c259088884ab88c1c7decdc18af7b"
format_stamp: "Formatted at 2026-08-07 22:42:35 on dist-test-slave-kvfs"
I20260807 22:42:35.742177 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/wal/instance:
uuid: "911c259088884ab88c1c7decdc18af7b"
format_stamp: "Formatted at 2026-08-07 22:42:35 on dist-test-slave-kvfs"
I20260807 22:42:35.750033 4685 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260807 22:42:35.755267 4706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:35.757575 4685 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260807 22:42:35.757838 4685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/wal
uuid: "911c259088884ab88c1c7decdc18af7b"
format_stamp: "Formatted at 2026-08-07 22:42:35 on dist-test-slave-kvfs"
I20260807 22:42:35.758113 4685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:42:35.776188 4685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:42:35.777694 4685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:42:35.778052 4685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:42:35.783171 4685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:42:35.783335 4685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:35.783560 4685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:42:35.783705 4685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:35.862432 4685 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.1:43617
I20260807 22:42:35.862447 4867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.1:43617 every 8 connection(s)
I20260807 22:42:35.864570 4685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
I20260807 22:42:35.869001 4868 heartbeater.cc:347] Connected to a master server at 127.4.103.62:39909
I20260807 22:42:35.869549 4868 heartbeater.cc:464] Registering TS with master...
I20260807 22:42:35.870874 4868 heartbeater.cc:511] Master 127.4.103.62:39909 requested a full tablet report, sending...
I20260807 22:42:35.871904 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 4685
I20260807 22:42:35.872341 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-0/wal/instance
I20260807 22:42:35.876338 4572 ts_manager.cc:194] Registered new tserver with Master: 911c259088884ab88c1c7decdc18af7b (127.4.103.1:43617)
I20260807 22:42:35.877743 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.2:0
--local_ip_for_outbound_sockets=127.4.103.2
--webserver_interface=127.4.103.2
--webserver_port=0
--tserver_master_addrs=127.4.103.62:39909
--builtin_ntp_servers=127.4.103.20:32827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
I20260807 22:42:35.878868 4572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.1:56703
I20260807 22:42:36.210791 4572 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45948:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:42:36.413136 4870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:42:36.413569 4870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:42:36.413743 4870 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:42:36.413857 4870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:42:36.422844 4870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:42:36.423087 4870 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:42:36.423218 4870 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:42:36.423316 4870 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:42:36.423544 4870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.2
I20260807 22:42:36.435210 4870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:32827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.103.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:39909
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.2
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:42:36.437165 4870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:42:36.439705 4870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:42:36.459285 4883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:36.459378 4886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:36.459307 4884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:42:36.461087 4870 server_base.cc:1034] running on GCE node
I20260807 22:42:36.462085 4870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:42:36.463693 4870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:42:36.465005 4870 hybrid_clock.cc:648] HybridClock initialized: now 1786142556464901 us; error 70 us; skew 500 ppm
I20260807 22:42:36.465572 4870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:42:36.475224 4870 webserver.cc:460] Webserver started at http://127.4.103.2:36583/ using document root <none> and password file <none>
I20260807 22:42:36.476472 4870 fs_manager.cc:359] Metadata directory not provided
I20260807 22:42:36.476656 4870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:42:36.477109 4870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:42:36.480873 4870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/data/instance:
uuid: "b0a6a86696504fb2a9fa64dbb60f2dad"
format_stamp: "Formatted at 2026-08-07 22:42:36 on dist-test-slave-kvfs"
I20260807 22:42:36.481961 4870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/wal/instance:
uuid: "b0a6a86696504fb2a9fa64dbb60f2dad"
format_stamp: "Formatted at 2026-08-07 22:42:36 on dist-test-slave-kvfs"
I20260807 22:42:36.489449 4870 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260807 22:42:36.494424 4893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:36.496297 4870 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260807 22:42:36.496603 4870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/wal
uuid: "b0a6a86696504fb2a9fa64dbb60f2dad"
format_stamp: "Formatted at 2026-08-07 22:42:36 on dist-test-slave-kvfs"
I20260807 22:42:36.496871 4870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:42:36.520128 4870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:42:36.521355 4870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:42:36.521742 4870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:42:36.525939 4870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:42:36.526165 4870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260807 22:42:36.526381 4870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:42:36.526557 4870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:36.601110 4870 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.2:46285
I20260807 22:42:36.601280 5070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.2:46285 every 8 connection(s)
I20260807 22:42:36.603024 4870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
I20260807 22:42:36.604480 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 4870
I20260807 22:42:36.604746 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-1/wal/instance
I20260807 22:42:36.606523 5072 heartbeater.cc:347] Connected to a master server at 127.4.103.62:39909
I20260807 22:42:36.607722 5072 heartbeater.cc:464] Registering TS with master...
I20260807 22:42:36.608938 5072 heartbeater.cc:511] Master 127.4.103.62:39909 requested a full tablet report, sending...
I20260807 22:42:36.609119 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.3:0
--local_ip_for_outbound_sockets=127.4.103.3
--webserver_interface=127.4.103.3
--webserver_port=0
--tserver_master_addrs=127.4.103.62:39909
--builtin_ntp_servers=127.4.103.20:32827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
I20260807 22:42:36.610754 4572 ts_manager.cc:194] Registered new tserver with Master: b0a6a86696504fb2a9fa64dbb60f2dad (127.4.103.2:46285)
I20260807 22:42:36.611981 4572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.2:33025
I20260807 22:42:36.884781 4868 heartbeater.cc:503] Master 127.4.103.62:39909 was elected leader, sending a full tablet report...
W20260807 22:42:37.146533 5075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:42:37.146988 5075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:42:37.147176 5075 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:42:37.147287 5075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:42:37.156827 5075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:42:37.157025 5075 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:42:37.157176 5075 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:42:37.157272 5075 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:42:37.157474 5075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.3
I20260807 22:42:37.169970 5075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:32827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.103.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:39909
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.3
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:42:37.172309 5075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:42:37.174937 5075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:42:37.193168 5083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:37.193125 5092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:37.193327 5087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:42:37.194607 5075 server_base.cc:1034] running on GCE node
I20260807 22:42:37.195492 5075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:42:37.196877 5075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:42:37.198233 5075 hybrid_clock.cc:648] HybridClock initialized: now 1786142557198115 us; error 73 us; skew 500 ppm
I20260807 22:42:37.198758 5075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:42:37.202297 5075 webserver.cc:460] Webserver started at http://127.4.103.3:33531/ using document root <none> and password file <none>
I20260807 22:42:37.203366 5075 fs_manager.cc:359] Metadata directory not provided
I20260807 22:42:37.203562 5075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:42:37.204006 5075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:42:37.208120 5075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/data/instance:
uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e"
format_stamp: "Formatted at 2026-08-07 22:42:37 on dist-test-slave-kvfs"
I20260807 22:42:37.209267 5075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/wal/instance:
uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e"
format_stamp: "Formatted at 2026-08-07 22:42:37 on dist-test-slave-kvfs"
I20260807 22:42:37.217198 5075 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260807 22:42:37.224119 5110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:37.226629 5075 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260807 22:42:37.226969 5075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/wal
uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e"
format_stamp: "Formatted at 2026-08-07 22:42:37 on dist-test-slave-kvfs"
I20260807 22:42:37.227366 5075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:42:37.231962 4572 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45956:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20260807 22:42:37.274482 5075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:42:37.275820 5075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:42:37.276292 5075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:42:37.280395 5075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:42:37.280573 5075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:37.280741 5075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:42:37.280865 5075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:37.359153 5075 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.3:39155
I20260807 22:42:37.359210 5286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.3:39155 every 8 connection(s)
I20260807 22:42:37.361011 5075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
I20260807 22:42:37.364411 5287 heartbeater.cc:347] Connected to a master server at 127.4.103.62:39909
I20260807 22:42:37.364801 5287 heartbeater.cc:464] Registering TS with master...
I20260807 22:42:37.365931 5287 heartbeater.cc:511] Master 127.4.103.62:39909 requested a full tablet report, sending...
I20260807 22:42:37.367676 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 5075
I20260807 22:42:37.367929 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.StaleTransactionsCleanup.1786142554140270-4508-0/minicluster-data/ts-2/wal/instance
I20260807 22:42:37.368158 4572 ts_manager.cc:194] Registered new tserver with Master: 5b67b3b6ffef47ce9ed32ffb9d91e69e (127.4.103.3:39155)
I20260807 22:42:37.369328 4572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.3:51981
I20260807 22:42:37.375077 4508 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:42:37.615749 5072 heartbeater.cc:503] Master 127.4.103.62:39909 was elected leader, sending a full tablet report...
I20260807 22:42:38.254446 4572 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45972:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:42:38.255779 4572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:42:38.302630 4970 tablet_service.cc:1467] Processing CreateTablet for tablet 4dcd5dc74da24a6ebc2c25dc4b936d75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=862f41cce80e49158ae0d8db232bb9b3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:42:38.306128 4970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dcd5dc74da24a6ebc2c25dc4b936d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:42:38.320648 5190 tablet_service.cc:1467] Processing CreateTablet for tablet 4dcd5dc74da24a6ebc2c25dc4b936d75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=862f41cce80e49158ae0d8db232bb9b3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:42:38.323137 5190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dcd5dc74da24a6ebc2c25dc4b936d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:42:38.322283 4782 tablet_service.cc:1467] Processing CreateTablet for tablet 4dcd5dc74da24a6ebc2c25dc4b936d75 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=862f41cce80e49158ae0d8db232bb9b3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:42:38.324477 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dcd5dc74da24a6ebc2c25dc4b936d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:42:38.327644 5322 tablet_bootstrap.cc:492] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad: Bootstrap starting.
I20260807 22:42:38.333189 5322 tablet_bootstrap.cc:654] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad: Neither blocks nor log segments found. Creating new log.
I20260807 22:42:38.335808 5322 log.cc:824] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad: Log is configured to *not* fsync() on all Append() calls
I20260807 22:42:38.339735 5322 tablet_bootstrap.cc:492] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad: No bootstrap required, opened a new log
I20260807 22:42:38.340200 5322 ts_tablet_manager.cc:1397] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260807 22:42:38.344183 5328 tablet_bootstrap.cc:492] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b: Bootstrap starting.
I20260807 22:42:38.348323 5327 tablet_bootstrap.cc:492] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e: Bootstrap starting.
I20260807 22:42:38.349560 5328 tablet_bootstrap.cc:654] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b: Neither blocks nor log segments found. Creating new log.
I20260807 22:42:38.351307 5328 log.cc:824] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b: Log is configured to *not* fsync() on all Append() calls
I20260807 22:42:38.353355 5327 tablet_bootstrap.cc:654] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e: Neither blocks nor log segments found. Creating new log.
I20260807 22:42:38.354684 5327 log.cc:824] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e: Log is configured to *not* fsync() on all Append() calls
I20260807 22:42:38.354292 5322 raft_consensus.cc:348] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:38.354918 5322 raft_consensus.cc:374] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:42:38.355105 5322 raft_consensus.cc:729] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a6a86696504fb2a9fa64dbb60f2dad, State: Initialized, Role: FOLLOWER
I20260807 22:42:38.355434 5328 tablet_bootstrap.cc:492] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b: No bootstrap required, opened a new log
I20260807 22:42:38.355885 5328 ts_tablet_manager.cc:1397] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260807 22:42:38.355937 5322 consensus_queue.cc:260] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:38.366298 5327 tablet_bootstrap.cc:492] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e: No bootstrap required, opened a new log
I20260807 22:42:38.366761 5327 ts_tablet_manager.cc:1397] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.004s
I20260807 22:42:38.357828 5331 tablet_replica.cc:439] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } } }
I20260807 22:42:38.369918 5331 tablet_replica.cc:442] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:38.370679 5331 raft_consensus.cc:482] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:42:38.370308 5328 raft_consensus.cc:348] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:38.371030 5328 raft_consensus.cc:374] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:42:38.371167 5328 raft_consensus.cc:729] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 911c259088884ab88c1c7decdc18af7b, State: Initialized, Role: FOLLOWER
I20260807 22:42:38.371371 5331 raft_consensus.cc:504] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:38.372175 5328 consensus_queue.cc:260] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:38.374528 5287 heartbeater.cc:503] Master 127.4.103.62:39909 was elected leader, sending a full tablet report...
I20260807 22:42:38.380312 5331 leader_election.cc:288] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b67b3b6ffef47ce9ed32ffb9d91e69e (127.4.103.3:39155), 911c259088884ab88c1c7decdc18af7b (127.4.103.1:43617)
I20260807 22:42:38.382499 5322 ts_tablet_manager.cc:1428] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad: Time spent starting tablet: real 0.033s user 0.006s sys 0.009s
I20260807 22:42:38.393770 5328 ts_tablet_manager.cc:1428] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b: Time spent starting tablet: real 0.038s user 0.020s sys 0.004s
I20260807 22:42:38.397291 5221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcd5dc74da24a6ebc2c25dc4b936d75" candidate_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" is_pre_election: true
I20260807 22:42:38.398495 5327 raft_consensus.cc:348] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:38.399376 5327 raft_consensus.cc:374] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:42:38.399636 5327 raft_consensus.cc:729] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b67b3b6ffef47ce9ed32ffb9d91e69e, State: Initialized, Role: FOLLOWER
W20260807 22:42:38.399957 4894 leader_election.cc:341] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b67b3b6ffef47ce9ed32ffb9d91e69e (127.4.103.3:39155): Illegal state: must be running to vote when last-logged opid is not known
I20260807 22:42:38.400007 5334 tablet_replica.cc:439] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } } }
I20260807 22:42:38.400575 5334 tablet_replica.cc:442] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:38.400846 5334 raft_consensus.cc:482] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:42:38.400818 5327 consensus_queue.cc:260] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:38.401173 5334 raft_consensus.cc:504] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:38.403539 5327 ts_tablet_manager.cc:1428] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e: Time spent starting tablet: real 0.037s user 0.018s sys 0.003s
I20260807 22:42:38.410090 5339 tablet_replica.cc:439] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } } }
I20260807 22:42:38.410837 5339 tablet_replica.cc:442] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:38.411257 5339 raft_consensus.cc:482] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:42:38.411782 5339 raft_consensus.cc:504] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:38.414215 5339 leader_election.cc:288] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0a6a86696504fb2a9fa64dbb60f2dad (127.4.103.2:46285), 911c259088884ab88c1c7decdc18af7b (127.4.103.1:43617)
I20260807 22:42:38.433270 5334 leader_election.cc:288] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b67b3b6ffef47ce9ed32ffb9d91e69e (127.4.103.3:39155), b0a6a86696504fb2a9fa64dbb60f2dad (127.4.103.2:46285)
I20260807 22:42:38.445827 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcd5dc74da24a6ebc2c25dc4b936d75" candidate_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "911c259088884ab88c1c7decdc18af7b" is_pre_election: true
I20260807 22:42:38.446816 4809 raft_consensus.cc:2454] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0a6a86696504fb2a9fa64dbb60f2dad in term 0.
I20260807 22:42:38.448134 4895 leader_election.cc:302] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 911c259088884ab88c1c7decdc18af7b, b0a6a86696504fb2a9fa64dbb60f2dad; no voters: 5b67b3b6ffef47ce9ed32ffb9d91e69e
I20260807 22:42:38.449803 5332 raft_consensus.cc:2781] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:42:38.450068 5332 raft_consensus.cc:482] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:42:38.450253 5332 raft_consensus.cc:3037] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:42:38.451367 5221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcd5dc74da24a6ebc2c25dc4b936d75" candidate_uuid: "911c259088884ab88c1c7decdc18af7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" is_pre_election: true
I20260807 22:42:38.451807 5221 raft_consensus.cc:2454] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 911c259088884ab88c1c7decdc18af7b in term 0.
I20260807 22:42:38.455142 5332 raft_consensus.cc:504] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:38.456975 5221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcd5dc74da24a6ebc2c25dc4b936d75" candidate_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e"
I20260807 22:42:38.457329 5221 raft_consensus.cc:3037] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:42:38.457823 4809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcd5dc74da24a6ebc2c25dc4b936d75" candidate_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "911c259088884ab88c1c7decdc18af7b"
I20260807 22:42:38.458134 4809 raft_consensus.cc:3037] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:42:38.458103 4708 leader_election.cc:302] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b67b3b6ffef47ce9ed32ffb9d91e69e, 911c259088884ab88c1c7decdc18af7b; no voters:
I20260807 22:42:38.459789 5332 leader_election.cc:288] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [CANDIDATE]: Term 1 election: Requested vote from peers 5b67b3b6ffef47ce9ed32ffb9d91e69e (127.4.103.3:39155), 911c259088884ab88c1c7decdc18af7b (127.4.103.1:43617)
I20260807 22:42:38.461860 5003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcd5dc74da24a6ebc2c25dc4b936d75" candidate_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" is_pre_election: true
I20260807 22:42:38.462211 5221 raft_consensus.cc:2454] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a6a86696504fb2a9fa64dbb60f2dad in term 1.
I20260807 22:42:38.462706 5003 raft_consensus.cc:2379] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b67b3b6ffef47ce9ed32ffb9d91e69e in current term 1: Already voted for candidate b0a6a86696504fb2a9fa64dbb60f2dad in this term.
I20260807 22:42:38.462741 4894 leader_election.cc:302] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b67b3b6ffef47ce9ed32ffb9d91e69e, b0a6a86696504fb2a9fa64dbb60f2dad; no voters:
I20260807 22:42:38.465039 5332 raft_consensus.cc:2781] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:42:38.465701 5332 raft_consensus.cc:686] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [term 1 LEADER]: Becoming Leader. State: Replica: b0a6a86696504fb2a9fa64dbb60f2dad, State: Running, Role: LEADER
I20260807 22:42:38.466737 5332 consensus_queue.cc:237] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:38.470619 4809 raft_consensus.cc:2454] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a6a86696504fb2a9fa64dbb60f2dad in term 1.
I20260807 22:42:38.470132 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcd5dc74da24a6ebc2c25dc4b936d75" candidate_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "911c259088884ab88c1c7decdc18af7b" is_pre_election: true
I20260807 22:42:38.473352 5337 raft_consensus.cc:2741] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260807 22:42:38.474340 5331 tablet_replica.cc:439] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad: TxnStatusTablet state changed. Reason: New leader b0a6a86696504fb2a9fa64dbb60f2dad. Latest consensus state: current_term: 1 leader_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } } }
I20260807 22:42:38.474779 5331 tablet_replica.cc:442] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:38.475795 5351 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:42:38.476274 5351 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:42:38.476872 4808 raft_consensus.cc:2379] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b67b3b6ffef47ce9ed32ffb9d91e69e in current term 1: Already voted for candidate b0a6a86696504fb2a9fa64dbb60f2dad in this term.
I20260807 22:42:38.476862 4572 catalog_manager.cc:5495] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad reported cstate change: term changed from 0 to 1, leader changed from <none> to b0a6a86696504fb2a9fa64dbb60f2dad (127.4.103.2). New cstate: current_term: 1 leader_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } health_report { overall_health: UNKNOWN } } }
I20260807 22:42:38.480312 5112 leader_election.cc:302] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b67b3b6ffef47ce9ed32ffb9d91e69e; no voters: 911c259088884ab88c1c7decdc18af7b, b0a6a86696504fb2a9fa64dbb60f2dad
I20260807 22:42:38.481487 5341 raft_consensus.cc:2726] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 22:42:38.484954 5003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcd5dc74da24a6ebc2c25dc4b936d75" candidate_uuid: "911c259088884ab88c1c7decdc18af7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" is_pre_election: true
I20260807 22:42:38.522919 5331 consensus_queue.cc:1035] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [LEADER]: Connected to new peer: Peer: permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:42:38.524516 5331 consensus_queue.cc:1035] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:42:38.530452 5337 tablet_replica.cc:439] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b: TxnStatusTablet state changed. Reason: New leader b0a6a86696504fb2a9fa64dbb60f2dad. Latest consensus state: current_term: 1 leader_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } } }
I20260807 22:42:38.531062 5337 tablet_replica.cc:442] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:38.536747 5341 tablet_replica.cc:439] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e: TxnStatusTablet state changed. Reason: New leader b0a6a86696504fb2a9fa64dbb60f2dad. Latest consensus state: current_term: 1 leader_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } } }
I20260807 22:42:38.537467 5341 tablet_replica.cc:442] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:38.551110 5331 tablet_replica.cc:439] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } } }
I20260807 22:42:38.551638 5331 tablet_replica.cc:442] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:38.555269 5331 tablet_replica.cc:439] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } } }
I20260807 22:42:38.555645 5331 tablet_replica.cc:442] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:38.555787 5341 tablet_replica.cc:439] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } } }
I20260807 22:42:38.556250 5341 tablet_replica.cc:442] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:38.558917 5337 tablet_replica.cc:439] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } } }
I20260807 22:42:38.559353 5337 tablet_replica.cc:442] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:39.144439 4903 txn_status_manager.cc:1390] automatically aborted stale txn (ID 0) past 0.539s from last keepalive heartbeat (effective timeout is 0.500s)
I20260807 22:42:40.187879 5393 raft_consensus.cc:482] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0a6a86696504fb2a9fa64dbb60f2dad)
I20260807 22:42:40.188238 5393 raft_consensus.cc:504] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:40.189286 5393 leader_election.cc:288] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b67b3b6ffef47ce9ed32ffb9d91e69e (127.4.103.3:39155), b0a6a86696504fb2a9fa64dbb60f2dad (127.4.103.2:46285)
I20260807 22:42:40.189957 5003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcd5dc74da24a6ebc2c25dc4b936d75" candidate_uuid: "911c259088884ab88c1c7decdc18af7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" is_pre_election: true
I20260807 22:42:40.189985 5221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcd5dc74da24a6ebc2c25dc4b936d75" candidate_uuid: "911c259088884ab88c1c7decdc18af7b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" is_pre_election: true
I20260807 22:42:40.191002 4712 leader_election.cc:302] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 911c259088884ab88c1c7decdc18af7b; no voters: 5b67b3b6ffef47ce9ed32ffb9d91e69e, b0a6a86696504fb2a9fa64dbb60f2dad
I20260807 22:42:40.191433 5393 raft_consensus.cc:2726] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 22:42:41.634253 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 4685
I20260807 22:42:41.636427 4862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:42:41.748636 5398 raft_consensus.cc:482] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0a6a86696504fb2a9fa64dbb60f2dad)
I20260807 22:42:41.748917 5398 raft_consensus.cc:504] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 911c259088884ab88c1c7decdc18af7b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:41.749013 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 4685
I20260807 22:42:41.783001 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 4870
W20260807 22:42:41.782861 4895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.103.1:43617 (error 108)
I20260807 22:42:41.784750 5064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:42:41.844871 5402 raft_consensus.cc:482] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0a6a86696504fb2a9fa64dbb60f2dad)
I20260807 22:42:41.845244 5402 raft_consensus.cc:504] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:41.846169 5402 leader_election.cc:288] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0a6a86696504fb2a9fa64dbb60f2dad (127.4.103.2:46285), 911c259088884ab88c1c7decdc18af7b (127.4.103.1:43617)
W20260807 22:42:41.847630 5112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.103.1:43617: connect: Connection refused (error 111)
W20260807 22:42:41.849469 5112 leader_election.cc:334] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 911c259088884ab88c1c7decdc18af7b (127.4.103.1:43617): Network error: Client connection negotiation failed: client connection to 127.4.103.1:43617: connect: Connection refused (error 111)
I20260807 22:42:41.925652 5003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dcd5dc74da24a6ebc2c25dc4b936d75" candidate_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" is_pre_election: true
W20260807 22:42:41.926017 4895 consensus_peers.cc:487] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P b0a6a86696504fb2a9fa64dbb60f2dad -> Peer 911c259088884ab88c1c7decdc18af7b (127.4.103.1:43617): Couldn't send request to peer 911c259088884ab88c1c7decdc18af7b. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.1:43617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 22:42:41.926380 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 4870
I20260807 22:42:41.926641 5114 leader_election.cc:302] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b67b3b6ffef47ce9ed32ffb9d91e69e; no voters: 911c259088884ab88c1c7decdc18af7b, b0a6a86696504fb2a9fa64dbb60f2dad
I20260807 22:42:41.927260 5402 raft_consensus.cc:2726] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 22:42:41.965603 5111 connection.cc:535] server connection from 127.4.103.2:55415 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 22:42:41.966512 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 5075
I20260807 22:42:41.968104 5282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:42:42.072926 5402 raft_consensus.cc:482] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b0a6a86696504fb2a9fa64dbb60f2dad)
I20260807 22:42:42.073226 5402 raft_consensus.cc:504] T 4dcd5dc74da24a6ebc2c25dc4b936d75 P 5b67b3b6ffef47ce9ed32ffb9d91e69e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b67b3b6ffef47ce9ed32ffb9d91e69e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 39155 } } peers { permanent_uuid: "b0a6a86696504fb2a9fa64dbb60f2dad" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 46285 } } peers { permanent_uuid: "911c259088884ab88c1c7decdc18af7b" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43617 } }
I20260807 22:42:42.073863 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 5075
I20260807 22:42:42.103044 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 4525
I20260807 22:42:42.104835 4635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:42:42.239534 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 4525
2026-08-07T22:42:42Z chronyd exiting
[ OK ] TxnStatusManagerITest.StaleTransactionsCleanup (8036 ms)
[ RUN ] TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime
2026-08-07T22:42:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:42:42Z Disabled control of system clock
I20260807 22:42:42.335769 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.103.62:43463
--webserver_interface=127.4.103.62
--webserver_port=0
--builtin_ntp_servers=127.4.103.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.103.62:43463
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false with env {}
W20260807 22:42:42.755309 5414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:42:42.755712 5414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:42:42.755797 5414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:42:42.763960 5414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:42:42.764174 5414 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 22:42:42.764221 5414 flags.cc:425] Enabled experimental flag: --txn_manager_lazily_initialized=false
W20260807 22:42:42.764276 5414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:42:42.764326 5414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:42:42.764397 5414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:42:42.774698 5414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.103.62:43463
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.103.62:43463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.103.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:42:42.776597 5414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:42:42.779649 5414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:42:42.796646 5426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:42.796702 5427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:42.796653 5430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:42:42.798179 5414 server_base.cc:1034] running on GCE node
I20260807 22:42:42.799226 5414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:42:42.801250 5414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:42:42.802534 5414 hybrid_clock.cc:648] HybridClock initialized: now 1786142562802458 us; error 52 us; skew 500 ppm
I20260807 22:42:42.802971 5414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:42:42.809993 5414 webserver.cc:460] Webserver started at http://127.4.103.62:37473/ using document root <none> and password file <none>
I20260807 22:42:42.810971 5414 fs_manager.cc:359] Metadata directory not provided
I20260807 22:42:42.811177 5414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:42:42.811590 5414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:42:42.815050 5414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/data/instance:
uuid: "781f3131d24c4430b12647f053761cac"
format_stamp: "Formatted at 2026-08-07 22:42:42 on dist-test-slave-kvfs"
I20260807 22:42:42.816000 5414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/wal/instance:
uuid: "781f3131d24c4430b12647f053761cac"
format_stamp: "Formatted at 2026-08-07 22:42:42 on dist-test-slave-kvfs"
I20260807 22:42:42.822878 5414 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260807 22:42:42.827116 5437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:42.828997 5414 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260807 22:42:42.829265 5414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/wal
uuid: "781f3131d24c4430b12647f053761cac"
format_stamp: "Formatted at 2026-08-07 22:42:42 on dist-test-slave-kvfs"
I20260807 22:42:42.829559 5414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:42:42.858229 5414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:42:42.859411 5414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:42:42.859799 5414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:42:42.886181 5414 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.62:43463
I20260807 22:42:42.886204 5512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.62:43463 every 8 connection(s)
I20260807 22:42:42.887995 5414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/data/info.pb
I20260807 22:42:42.893214 5515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:42:42.894578 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 5414
I20260807 22:42:42.894847 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/master-0/wal/instance
I20260807 22:42:42.910158 5515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac: Bootstrap starting.
I20260807 22:42:42.916756 5515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac: Neither blocks nor log segments found. Creating new log.
I20260807 22:42:42.918442 5515 log.cc:824] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac: Log is configured to *not* fsync() on all Append() calls
I20260807 22:42:42.922293 5515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac: No bootstrap required, opened a new log
I20260807 22:42:42.931111 5515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781f3131d24c4430b12647f053761cac" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 43463 } }
I20260807 22:42:42.931676 5515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:42:42.931777 5515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781f3131d24c4430b12647f053761cac, State: Initialized, Role: FOLLOWER
I20260807 22:42:42.932518 5515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781f3131d24c4430b12647f053761cac" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 43463 } }
I20260807 22:42:42.932804 5515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:42:42.932986 5515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:42:42.933207 5515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:42:42.935940 5515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781f3131d24c4430b12647f053761cac" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 43463 } }
I20260807 22:42:42.936664 5515 leader_election.cc:302] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 781f3131d24c4430b12647f053761cac; no voters:
I20260807 22:42:42.937366 5515 leader_election.cc:288] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:42:42.937574 5521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:42:42.938359 5521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [term 1 LEADER]: Becoming Leader. State: Replica: 781f3131d24c4430b12647f053761cac, State: Running, Role: LEADER
I20260807 22:42:42.939005 5521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781f3131d24c4430b12647f053761cac" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 43463 } }
I20260807 22:42:42.940160 5515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:42:42.943192 5522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "781f3131d24c4430b12647f053761cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781f3131d24c4430b12647f053761cac" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 43463 } } }
I20260807 22:42:42.942932 5525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 781f3131d24c4430b12647f053761cac. Latest consensus state: current_term: 1 leader_uuid: "781f3131d24c4430b12647f053761cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781f3131d24c4430b12647f053761cac" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 43463 } } }
I20260807 22:42:42.943637 5522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [sys.catalog]: This master's current role is: LEADER
I20260807 22:42:42.943655 5525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac [sys.catalog]: This master's current role is: LEADER
I20260807 22:42:42.945570 5535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:42:42.951821 5535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:42:42.962939 5535 catalog_manager.cc:1347] Generated new cluster ID: a5e07f5d69df4efbbc6f1c4efb01e2c9
I20260807 22:42:42.963128 5535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:42:42.983397 5535 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:42:42.985388 5535 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:42:42.994544 5535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 781f3131d24c4430b12647f053761cac: Generated new TSK 0
I20260807 22:42:42.995774 5535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:42:43.007539 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.1:0
--local_ip_for_outbound_sockets=127.4.103.1
--webserver_interface=127.4.103.1
--webserver_port=0
--tserver_master_addrs=127.4.103.62:43463
--builtin_ntp_servers=127.4.103.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
I20260807 22:42:43.012009 5458 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53342:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:42:43.029490 5515 master.cc:440] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 3; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20260807 22:42:43.454706 5556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:42:43.455219 5556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:42:43.455389 5556 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:42:43.455518 5556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:42:43.464370 5556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:42:43.464613 5556 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:42:43.464731 5556 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:42:43.464813 5556 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:42:43.464977 5556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.1
I20260807 22:42:43.475749 5556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.103.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:43463
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.1
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:42:43.477820 5556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:42:43.480639 5556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:42:43.498109 5567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:43.498073 5568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:43.498005 5570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:42:43.499184 5556 server_base.cc:1034] running on GCE node
I20260807 22:42:43.500129 5556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:42:43.501497 5556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:42:43.502751 5556 hybrid_clock.cc:648] HybridClock initialized: now 1786142563502665 us; error 45 us; skew 500 ppm
I20260807 22:42:43.503197 5556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:42:43.506665 5556 webserver.cc:460] Webserver started at http://127.4.103.1:38127/ using document root <none> and password file <none>
I20260807 22:42:43.507711 5556 fs_manager.cc:359] Metadata directory not provided
I20260807 22:42:43.507877 5556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:42:43.508355 5556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:42:43.511862 5556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/data/instance:
uuid: "25ec82ddc9ae44d58cdb78786e71c3c1"
format_stamp: "Formatted at 2026-08-07 22:42:43 on dist-test-slave-kvfs"
I20260807 22:42:43.512970 5556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/wal/instance:
uuid: "25ec82ddc9ae44d58cdb78786e71c3c1"
format_stamp: "Formatted at 2026-08-07 22:42:43 on dist-test-slave-kvfs"
I20260807 22:42:43.519758 5556 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260807 22:42:43.524327 5579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:43.526014 5556 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:42:43.526232 5556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/wal
uuid: "25ec82ddc9ae44d58cdb78786e71c3c1"
format_stamp: "Formatted at 2026-08-07 22:42:43 on dist-test-slave-kvfs"
I20260807 22:42:43.526479 5556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:42:43.548869 5556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:42:43.550115 5556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:42:43.550506 5556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:42:43.554688 5556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:42:43.554867 5556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:43.555161 5556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:42:43.555378 5556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:43.617689 5556 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.1:37385
I20260807 22:42:43.617740 5744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.1:37385 every 8 connection(s)
I20260807 22:42:43.619412 5556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
I20260807 22:42:43.622306 5745 heartbeater.cc:347] Connected to a master server at 127.4.103.62:43463
I20260807 22:42:43.622574 5745 heartbeater.cc:464] Registering TS with master...
I20260807 22:42:43.623459 5745 heartbeater.cc:511] Master 127.4.103.62:43463 requested a full tablet report, sending...
I20260807 22:42:43.624600 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 5556
I20260807 22:42:43.624811 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-0/wal/instance
I20260807 22:42:43.626362 5458 ts_manager.cc:194] Registered new tserver with Master: 25ec82ddc9ae44d58cdb78786e71c3c1 (127.4.103.1:37385)
I20260807 22:42:43.628517 5458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.1:45927
I20260807 22:42:43.629607 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.2:0
--local_ip_for_outbound_sockets=127.4.103.2
--webserver_interface=127.4.103.2
--webserver_port=0
--tserver_master_addrs=127.4.103.62:43463
--builtin_ntp_servers=127.4.103.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
W20260807 22:42:44.046226 5747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:42:44.046665 5747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:42:44.046801 5747 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:42:44.046890 5747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:42:44.056876 5747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:42:44.057116 5747 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:42:44.057220 5747 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:42:44.057301 5747 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:42:44.057469 5747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.2
I20260807 22:42:44.068109 5458 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53348:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20260807 22:42:44.071503 5747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.103.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:43463
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.2
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:42:44.073344 5747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:42:44.075680 5747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:42:44.092615 5759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:44.092615 5761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:44.092887 5758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:42:44.094163 5747 server_base.cc:1034] running on GCE node
I20260807 22:42:44.095095 5747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:42:44.096582 5747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:42:44.097922 5747 hybrid_clock.cc:648] HybridClock initialized: now 1786142564097817 us; error 66 us; skew 500 ppm
I20260807 22:42:44.098381 5747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:42:44.101816 5747 webserver.cc:460] Webserver started at http://127.4.103.2:36591/ using document root <none> and password file <none>
I20260807 22:42:44.102846 5747 fs_manager.cc:359] Metadata directory not provided
I20260807 22:42:44.103030 5747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:42:44.103468 5747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:42:44.107308 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/data/instance:
uuid: "74d399adb1f04e74a30f9f0e04af7b63"
format_stamp: "Formatted at 2026-08-07 22:42:44 on dist-test-slave-kvfs"
I20260807 22:42:44.108575 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/wal/instance:
uuid: "74d399adb1f04e74a30f9f0e04af7b63"
format_stamp: "Formatted at 2026-08-07 22:42:44 on dist-test-slave-kvfs"
I20260807 22:42:44.116336 5747 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260807 22:42:44.121958 5772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:44.123953 5747 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260807 22:42:44.124269 5747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/wal
uuid: "74d399adb1f04e74a30f9f0e04af7b63"
format_stamp: "Formatted at 2026-08-07 22:42:44 on dist-test-slave-kvfs"
I20260807 22:42:44.124584 5747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:42:44.143433 5747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:42:44.144631 5747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:42:44.145009 5747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:42:44.148558 5747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:42:44.148706 5747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:44.148887 5747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:42:44.148994 5747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:44.204442 5747 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.2:34067
I20260807 22:42:44.204464 5940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.2:34067 every 8 connection(s)
I20260807 22:42:44.206179 5747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
I20260807 22:42:44.209110 5942 heartbeater.cc:347] Connected to a master server at 127.4.103.62:43463
I20260807 22:42:44.209401 5942 heartbeater.cc:464] Registering TS with master...
I20260807 22:42:44.210327 5942 heartbeater.cc:511] Master 127.4.103.62:43463 requested a full tablet report, sending...
I20260807 22:42:44.211453 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 5747
I20260807 22:42:44.211731 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-1/wal/instance
I20260807 22:42:44.212358 5458 ts_manager.cc:194] Registered new tserver with Master: 74d399adb1f04e74a30f9f0e04af7b63 (127.4.103.2:34067)
I20260807 22:42:44.213593 5458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.2:39901
I20260807 22:42:44.216459 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.3:0
--local_ip_for_outbound_sockets=127.4.103.3
--webserver_interface=127.4.103.3
--webserver_port=0
--tserver_master_addrs=127.4.103.62:43463
--builtin_ntp_servers=127.4.103.20:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
I20260807 22:42:44.633531 5745 heartbeater.cc:503] Master 127.4.103.62:43463 was elected leader, sending a full tablet report...
W20260807 22:42:44.639573 5946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:42:44.639978 5946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:42:44.640173 5946 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:42:44.640259 5946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:42:44.648878 5946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:42:44.649151 5946 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:42:44.649266 5946 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:42:44.649348 5946 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:42:44.649519 5946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.3
I20260807 22:42:44.660483 5946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.103.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:43463
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.3
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:42:44.662490 5946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:42:44.665066 5946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:42:44.682425 5954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:44.682662 5951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:44.682487 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:42:44.683681 5946 server_base.cc:1034] running on GCE node
I20260807 22:42:44.684609 5946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:42:44.685956 5946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:42:44.687273 5946 hybrid_clock.cc:648] HybridClock initialized: now 1786142564687143 us; error 72 us; skew 500 ppm
I20260807 22:42:44.687747 5946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:42:44.690804 5946 webserver.cc:460] Webserver started at http://127.4.103.3:42353/ using document root <none> and password file <none>
I20260807 22:42:44.691720 5946 fs_manager.cc:359] Metadata directory not provided
I20260807 22:42:44.691846 5946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:42:44.692266 5946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:42:44.695817 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/data/instance:
uuid: "3f1146d061404977aafac330bcf1827e"
format_stamp: "Formatted at 2026-08-07 22:42:44 on dist-test-slave-kvfs"
I20260807 22:42:44.696841 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/wal/instance:
uuid: "3f1146d061404977aafac330bcf1827e"
format_stamp: "Formatted at 2026-08-07 22:42:44 on dist-test-slave-kvfs"
I20260807 22:42:44.703529 5946 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260807 22:42:44.707985 5961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:44.709918 5946 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:42:44.710150 5946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/wal
uuid: "3f1146d061404977aafac330bcf1827e"
format_stamp: "Formatted at 2026-08-07 22:42:44 on dist-test-slave-kvfs"
I20260807 22:42:44.710390 5946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:42:44.729707 5946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:42:44.731109 5946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:42:44.731490 5946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:42:44.735193 5946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:42:44.735332 5946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:44.735483 5946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:42:44.735587 5946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:44.788777 5946 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.3:43199
I20260807 22:42:44.788790 6145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.3:43199 every 8 connection(s)
I20260807 22:42:44.790505 5946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
I20260807 22:42:44.793358 6147 heartbeater.cc:347] Connected to a master server at 127.4.103.62:43463
I20260807 22:42:44.793646 6147 heartbeater.cc:464] Registering TS with master...
I20260807 22:42:44.794494 6147 heartbeater.cc:511] Master 127.4.103.62:43463 requested a full tablet report, sending...
I20260807 22:42:44.796211 5457 ts_manager.cc:194] Registered new tserver with Master: 3f1146d061404977aafac330bcf1827e (127.4.103.3:43199)
I20260807 22:42:44.797292 5457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.3:36673
I20260807 22:42:44.797328 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 5946
I20260807 22:42:44.797555 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime.1786142554140270-4508-0/minicluster-data/ts-2/wal/instance
I20260807 22:42:44.801358 4508 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:42:45.090035 5457 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53372:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:42:45.091250 5457 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:42:45.126492 5656 tablet_service.cc:1467] Processing CreateTablet for tablet 4bdb144053e74012aae2545f2b1eeefb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=31691b0bd4c64de1ab7b8c5743fb9591]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:42:45.128149 5845 tablet_service.cc:1467] Processing CreateTablet for tablet 4bdb144053e74012aae2545f2b1eeefb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=31691b0bd4c64de1ab7b8c5743fb9591]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:42:45.129262 5656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bdb144053e74012aae2545f2b1eeefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:42:45.128540 6044 tablet_service.cc:1467] Processing CreateTablet for tablet 4bdb144053e74012aae2545f2b1eeefb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=31691b0bd4c64de1ab7b8c5743fb9591]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:42:45.130582 5845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bdb144053e74012aae2545f2b1eeefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:42:45.132004 6044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bdb144053e74012aae2545f2b1eeefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:42:45.150710 6178 tablet_bootstrap.cc:492] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: Bootstrap starting.
I20260807 22:42:45.151127 6180 tablet_bootstrap.cc:492] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1: Bootstrap starting.
I20260807 22:42:45.151587 6181 tablet_bootstrap.cc:492] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: Bootstrap starting.
I20260807 22:42:45.155933 6180 tablet_bootstrap.cc:654] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1: Neither blocks nor log segments found. Creating new log.
I20260807 22:42:45.156126 6181 tablet_bootstrap.cc:654] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: Neither blocks nor log segments found. Creating new log.
I20260807 22:42:45.157667 6180 log.cc:824] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1: Log is configured to *not* fsync() on all Append() calls
I20260807 22:42:45.157673 6181 log.cc:824] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: Log is configured to *not* fsync() on all Append() calls
I20260807 22:42:45.158200 6178 tablet_bootstrap.cc:654] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: Neither blocks nor log segments found. Creating new log.
I20260807 22:42:45.160215 6178 log.cc:824] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: Log is configured to *not* fsync() on all Append() calls
I20260807 22:42:45.163311 6180 tablet_bootstrap.cc:492] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1: No bootstrap required, opened a new log
I20260807 22:42:45.163486 6178 tablet_bootstrap.cc:492] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: No bootstrap required, opened a new log
I20260807 22:42:45.163494 6181 tablet_bootstrap.cc:492] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: No bootstrap required, opened a new log
I20260807 22:42:45.163722 6180 ts_tablet_manager.cc:1397] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.005s
I20260807 22:42:45.163825 6181 ts_tablet_manager.cc:1397] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20260807 22:42:45.163832 6178 ts_tablet_manager.cc:1397] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260807 22:42:45.173892 6180 raft_consensus.cc:348] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:45.174502 6180 raft_consensus.cc:374] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:42:45.174626 6180 raft_consensus.cc:729] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ec82ddc9ae44d58cdb78786e71c3c1, State: Initialized, Role: FOLLOWER
I20260807 22:42:45.175498 6180 consensus_queue.cc:260] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:45.178253 6180 ts_tablet_manager.cc:1428] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260807 22:42:45.177687 6178 raft_consensus.cc:348] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:45.178009 6181 raft_consensus.cc:348] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:45.178490 6178 raft_consensus.cc:374] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:42:45.178603 6181 raft_consensus.cc:374] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:42:45.178633 6178 raft_consensus.cc:729] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d399adb1f04e74a30f9f0e04af7b63, State: Initialized, Role: FOLLOWER
I20260807 22:42:45.178720 6181 raft_consensus.cc:729] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1146d061404977aafac330bcf1827e, State: Initialized, Role: FOLLOWER
I20260807 22:42:45.179680 6178 consensus_queue.cc:260] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:45.179842 6181 consensus_queue.cc:260] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:45.185269 6188 tablet_replica.cc:439] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } } }
I20260807 22:42:45.186388 6181 ts_tablet_manager.cc:1428] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: Time spent starting tablet: real 0.022s user 0.019s sys 0.002s
I20260807 22:42:45.186772 6188 tablet_replica.cc:442] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:45.187005 6147 heartbeater.cc:503] Master 127.4.103.62:43463 was elected leader, sending a full tablet report...
I20260807 22:42:45.187427 6188 raft_consensus.cc:482] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:42:45.188271 6188 raft_consensus.cc:504] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:45.189946 6192 tablet_replica.cc:439] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } } }
I20260807 22:42:45.190662 6192 tablet_replica.cc:442] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:45.192857 6178 ts_tablet_manager.cc:1428] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: Time spent starting tablet: real 0.029s user 0.018s sys 0.013s
I20260807 22:42:45.193568 5942 heartbeater.cc:503] Master 127.4.103.62:43463 was elected leader, sending a full tablet report...
I20260807 22:42:45.194136 6194 raft_consensus.cc:482] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:42:45.194525 6194 raft_consensus.cc:504] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:45.196007 6188 leader_election.cc:288] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74d399adb1f04e74a30f9f0e04af7b63 (127.4.103.2:34067), 3f1146d061404977aafac330bcf1827e (127.4.103.3:43199)
I20260807 22:42:45.198050 6191 tablet_replica.cc:439] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } } }
I20260807 22:42:45.198563 6191 tablet_replica.cc:442] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:45.201154 6194 leader_election.cc:288] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25ec82ddc9ae44d58cdb78786e71c3c1 (127.4.103.1:37385), 3f1146d061404977aafac330bcf1827e (127.4.103.3:43199)
W20260807 22:42:45.209136 5945 tablet.cc:2369] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:42:45.224193 5878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d399adb1f04e74a30f9f0e04af7b63" is_pre_election: true
I20260807 22:42:45.224355 6192 raft_consensus.cc:482] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:42:45.224835 5878 raft_consensus.cc:2454] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25ec82ddc9ae44d58cdb78786e71c3c1 in term 0.
I20260807 22:42:45.224777 6192 raft_consensus.cc:504] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:45.226454 5581 leader_election.cc:302] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25ec82ddc9ae44d58cdb78786e71c3c1, 74d399adb1f04e74a30f9f0e04af7b63; no voters:
I20260807 22:42:45.227025 6188 raft_consensus.cc:2781] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:42:45.227209 6188 raft_consensus.cc:482] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:42:45.227322 6188 raft_consensus.cc:3037] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:42:45.231168 6192 leader_election.cc:288] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74d399adb1f04e74a30f9f0e04af7b63 (127.4.103.2:34067), 25ec82ddc9ae44d58cdb78786e71c3c1 (127.4.103.1:37385)
I20260807 22:42:45.231189 6188 raft_consensus.cc:504] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:45.232612 6188 leader_election.cc:288] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [CANDIDATE]: Term 1 election: Requested vote from peers 74d399adb1f04e74a30f9f0e04af7b63 (127.4.103.2:34067), 3f1146d061404977aafac330bcf1827e (127.4.103.3:43199)
I20260807 22:42:45.233099 5878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d399adb1f04e74a30f9f0e04af7b63"
I20260807 22:42:45.233379 5878 raft_consensus.cc:3037] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:42:45.241039 6076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1146d061404977aafac330bcf1827e" is_pre_election: true
I20260807 22:42:45.241811 6076 raft_consensus.cc:2454] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25ec82ddc9ae44d58cdb78786e71c3c1 in term 0.
I20260807 22:42:45.242750 5878 raft_consensus.cc:2454] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ec82ddc9ae44d58cdb78786e71c3c1 in term 1.
I20260807 22:42:45.243527 6076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1146d061404977aafac330bcf1827e"
I20260807 22:42:45.243780 6076 raft_consensus.cc:3037] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:42:45.247083 5581 leader_election.cc:302] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25ec82ddc9ae44d58cdb78786e71c3c1, 74d399adb1f04e74a30f9f0e04af7b63; no voters:
I20260807 22:42:45.247978 6188 raft_consensus.cc:2781] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:42:45.248363 6076 raft_consensus.cc:2454] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ec82ddc9ae44d58cdb78786e71c3c1 in term 1.
I20260807 22:42:45.248602 6188 raft_consensus.cc:686] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [term 1 LEADER]: Becoming Leader. State: Replica: 25ec82ddc9ae44d58cdb78786e71c3c1, State: Running, Role: LEADER
I20260807 22:42:45.249790 6188 consensus_queue.cc:237] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:45.252453 5684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "3f1146d061404977aafac330bcf1827e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" is_pre_election: true
I20260807 22:42:45.261206 5684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "74d399adb1f04e74a30f9f0e04af7b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" is_pre_election: true
I20260807 22:42:45.265101 6190 tablet_replica.cc:439] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1: TxnStatusTablet state changed. Reason: New leader 25ec82ddc9ae44d58cdb78786e71c3c1. Latest consensus state: current_term: 1 leader_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } } }
I20260807 22:42:45.265645 6190 tablet_replica.cc:442] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:45.268560 6076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "74d399adb1f04e74a30f9f0e04af7b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1146d061404977aafac330bcf1827e" is_pre_election: true
I20260807 22:42:45.267954 5457 catalog_manager.cc:5495] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25ec82ddc9ae44d58cdb78786e71c3c1 (127.4.103.1). New cstate: current_term: 1 leader_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } health_report { overall_health: UNKNOWN } } }
I20260807 22:42:45.269233 6076 raft_consensus.cc:2379] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74d399adb1f04e74a30f9f0e04af7b63 in current term 1: Already voted for candidate 25ec82ddc9ae44d58cdb78786e71c3c1 in this term.
I20260807 22:42:45.269217 6207 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:42:45.269577 6207 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:42:45.270340 5774 leader_election.cc:302] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74d399adb1f04e74a30f9f0e04af7b63; no voters: 25ec82ddc9ae44d58cdb78786e71c3c1, 3f1146d061404977aafac330bcf1827e
I20260807 22:42:45.270829 5878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "3f1146d061404977aafac330bcf1827e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d399adb1f04e74a30f9f0e04af7b63" is_pre_election: true
I20260807 22:42:45.271087 6194 raft_consensus.cc:2726] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 22:42:45.271745 5878 raft_consensus.cc:2379] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f1146d061404977aafac330bcf1827e in current term 1: Already voted for candidate 25ec82ddc9ae44d58cdb78786e71c3c1 in this term.
I20260807 22:42:45.272866 5963 leader_election.cc:302] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f1146d061404977aafac330bcf1827e; no voters: 25ec82ddc9ae44d58cdb78786e71c3c1, 74d399adb1f04e74a30f9f0e04af7b63
I20260807 22:42:45.274606 6192 raft_consensus.cc:2726] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 22:42:45.281404 5739 generic_service.cc:175] {username='slave'} at 127.0.0.1:59708 changed flags via RPC: txn_staleness_tracker_interval_ms from '100' to '0'
I20260807 22:42:45.284593 5935 generic_service.cc:175] {username='slave'} at 127.0.0.1:46782 changed flags via RPC: txn_staleness_tracker_interval_ms from '100' to '0'
I20260807 22:42:45.287165 6141 generic_service.cc:175] {username='slave'} at 127.0.0.1:52412 changed flags via RPC: txn_staleness_tracker_interval_ms from '100' to '0'
I20260807 22:42:45.292560 6190 consensus_queue.cc:1035] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 22:42:45.293701 6148 tablet.cc:2369] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:42:45.296178 6190 consensus_queue.cc:1035] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:42:45.314149 6192 tablet_replica.cc:439] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: TxnStatusTablet state changed. Reason: New leader 25ec82ddc9ae44d58cdb78786e71c3c1. Latest consensus state: current_term: 1 leader_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } } }
I20260807 22:42:45.314683 6192 tablet_replica.cc:442] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:45.326838 6192 tablet_replica.cc:439] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } } }
I20260807 22:42:45.327229 6192 tablet_replica.cc:442] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:45.327615 6194 tablet_replica.cc:439] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: TxnStatusTablet state changed. Reason: New leader 25ec82ddc9ae44d58cdb78786e71c3c1. Latest consensus state: current_term: 1 leader_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } } }
I20260807 22:42:45.328083 6194 tablet_replica.cc:442] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:45.328804 6188 tablet_replica.cc:439] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } } }
I20260807 22:42:45.334879 6188 tablet_replica.cc:442] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:45.341355 6190 tablet_replica.cc:439] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } } }
I20260807 22:42:45.341777 6190 tablet_replica.cc:442] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:45.354724 6194 tablet_replica.cc:439] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } } }
I20260807 22:42:45.355110 6194 tablet_replica.cc:442] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:46.883910 5739 generic_service.cc:175] {username='slave'} at 127.0.0.1:59708 changed flags via RPC: txn_staleness_tracker_interval_ms from '0' to '100'
I20260807 22:42:46.885615 5935 generic_service.cc:175] {username='slave'} at 127.0.0.1:46782 changed flags via RPC: txn_staleness_tracker_interval_ms from '0' to '100'
I20260807 22:42:46.887209 6141 generic_service.cc:175] {username='slave'} at 127.0.0.1:52412 changed flags via RPC: txn_staleness_tracker_interval_ms from '0' to '100'
I20260807 22:42:47.085130 5588 txn_status_manager.cc:1390] automatically aborted stale txn (ID 0) past 1.700s from last keepalive heartbeat (effective timeout is 0.500s)
I20260807 22:42:48.392216 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 5556
I20260807 22:42:48.393748 5739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:42:48.432663 6251 raft_consensus.cc:482] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25ec82ddc9ae44d58cdb78786e71c3c1)
I20260807 22:42:48.432999 6251 raft_consensus.cc:504] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:48.434089 6251 leader_election.cc:288] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 74d399adb1f04e74a30f9f0e04af7b63 (127.4.103.2:34067), 25ec82ddc9ae44d58cdb78786e71c3c1 (127.4.103.1:37385)
I20260807 22:42:48.434962 6252 raft_consensus.cc:482] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25ec82ddc9ae44d58cdb78786e71c3c1)
I20260807 22:42:48.435331 5878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "3f1146d061404977aafac330bcf1827e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "74d399adb1f04e74a30f9f0e04af7b63" is_pre_election: true
I20260807 22:42:48.435238 6252 raft_consensus.cc:504] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:48.435796 5878 raft_consensus.cc:2454] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f1146d061404977aafac330bcf1827e in term 1.
I20260807 22:42:48.436507 6252 leader_election.cc:288] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25ec82ddc9ae44d58cdb78786e71c3c1 (127.4.103.1:37385), 3f1146d061404977aafac330bcf1827e (127.4.103.3:43199)
I20260807 22:42:48.436584 5963 leader_election.cc:302] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1146d061404977aafac330bcf1827e, 74d399adb1f04e74a30f9f0e04af7b63; no voters:
I20260807 22:42:48.436911 6251 raft_consensus.cc:2781] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 22:42:48.437058 6251 raft_consensus.cc:482] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 25ec82ddc9ae44d58cdb78786e71c3c1)
I20260807 22:42:48.437183 6251 raft_consensus.cc:3037] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:42:48.437052 6076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "74d399adb1f04e74a30f9f0e04af7b63" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3f1146d061404977aafac330bcf1827e" is_pre_election: true
I20260807 22:42:48.440883 6251 raft_consensus.cc:504] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:48.441522 6076 raft_consensus.cc:2379] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 74d399adb1f04e74a30f9f0e04af7b63 in current term 2: Already voted for candidate 3f1146d061404977aafac330bcf1827e in this term.
I20260807 22:42:48.441816 6251 leader_election.cc:288] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [CANDIDATE]: Term 2 election: Requested vote from peers 74d399adb1f04e74a30f9f0e04af7b63 (127.4.103.2:34067), 25ec82ddc9ae44d58cdb78786e71c3c1 (127.4.103.1:37385)
I20260807 22:42:48.442644 5878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "3f1146d061404977aafac330bcf1827e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "74d399adb1f04e74a30f9f0e04af7b63"
I20260807 22:42:48.442970 5878 raft_consensus.cc:3037] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:42:48.446283 5878 raft_consensus.cc:2454] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f1146d061404977aafac330bcf1827e in term 2.
I20260807 22:42:48.446887 5963 leader_election.cc:302] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f1146d061404977aafac330bcf1827e, 74d399adb1f04e74a30f9f0e04af7b63; no voters:
I20260807 22:42:48.447324 6251 raft_consensus.cc:2781] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 2 FOLLOWER]: Leader election won for term 2
I20260807 22:42:48.448309 6251 raft_consensus.cc:686] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 2 LEADER]: Becoming Leader. State: Replica: 3f1146d061404977aafac330bcf1827e, State: Running, Role: LEADER
I20260807 22:42:48.449018 6251 consensus_queue.cc:237] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:48.451784 6253 tablet_replica.cc:439] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: TxnStatusTablet state changed. Reason: New leader 3f1146d061404977aafac330bcf1827e. Latest consensus state: current_term: 2 leader_uuid: "3f1146d061404977aafac330bcf1827e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } } }
I20260807 22:42:48.452523 6253 tablet_replica.cc:442] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:48.453820 6256 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:42:48.454110 6256 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:42:48.454227 5457 catalog_manager.cc:5495] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e reported cstate change: term changed from 1 to 2, leader changed from 25ec82ddc9ae44d58cdb78786e71c3c1 (127.4.103.1) to 3f1146d061404977aafac330bcf1827e (127.4.103.3). New cstate: current_term: 2 leader_uuid: "3f1146d061404977aafac330bcf1827e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } health_report { overall_health: HEALTHY } } }
I20260807 22:42:48.468233 6256 txn_status_manager.cc:727] Starting 1 aborts task
I20260807 22:42:48.490880 5878 raft_consensus.cc:1261] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 2 FOLLOWER]: Refusing update from remote peer 3f1146d061404977aafac330bcf1827e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260807 22:42:48.491991 6253 consensus_queue.cc:1035] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 22:42:48.494719 6252 tablet_replica.cc:439] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: TxnStatusTablet state changed. Reason: New leader 3f1146d061404977aafac330bcf1827e. Latest consensus state: current_term: 2 leader_uuid: "3f1146d061404977aafac330bcf1827e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } } }
I20260807 22:42:48.495111 6252 tablet_replica.cc:442] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:48.499046 6252 tablet_replica.cc:439] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3f1146d061404977aafac330bcf1827e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } } }
I20260807 22:42:48.499439 6252 tablet_replica.cc:442] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:48.504132 6253 tablet_replica.cc:439] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3f1146d061404977aafac330bcf1827e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } } }
I20260807 22:42:48.504581 6253 tablet_replica.cc:442] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e: This TxnStatusTablet replica's current role is: LEADER
W20260807 22:42:48.514938 5965 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.103.1:37385 timed out after 0.080s (SENT)
W20260807 22:42:48.516973 5777 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.103.1:37385 timed out after 0.080s (SENT)
I20260807 22:42:48.590582 5682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "74d399adb1f04e74a30f9f0e04af7b63" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" is_pre_election: true
I20260807 22:42:48.591295 6076 raft_consensus.cc:1226] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [term 2 LEADER]: Rejecting Update request from peer 25ec82ddc9ae44d58cdb78786e71c3c1 for earlier term 1. Current term is 2. Ops: []
I20260807 22:42:48.591320 5878 raft_consensus.cc:1226] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 2 FOLLOWER]: Rejecting Update request from peer 25ec82ddc9ae44d58cdb78786e71c3c1 for earlier term 1. Current term is 2. Ops: []
I20260807 22:42:48.591935 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 5556
I20260807 22:42:48.592180 5682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "3f1146d061404977aafac330bcf1827e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" is_pre_election: true
I20260807 22:42:48.592888 5682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "3f1146d061404977aafac330bcf1827e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1"
I20260807 22:42:48.593089 5679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "3f1146d061404977aafac330bcf1827e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" is_pre_election: true
I20260807 22:42:48.593304 5680 raft_consensus.cc:3032] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [term 1 LEADER]: Stepping down as leader of term 1
I20260807 22:42:48.593545 5680 raft_consensus.cc:729] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 25ec82ddc9ae44d58cdb78786e71c3c1, State: Running, Role: LEADER
I20260807 22:42:48.593701 5681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "3f1146d061404977aafac330bcf1827e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1"
I20260807 22:42:48.594070 5680 consensus_queue.cc:260] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:48.594426 5680 raft_consensus.cc:3037] T 4bdb144053e74012aae2545f2b1eeefb P 25ec82ddc9ae44d58cdb78786e71c3c1 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:42:48.594384 5684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdb144053e74012aae2545f2b1eeefb" candidate_uuid: "74d399adb1f04e74a30f9f0e04af7b63" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" is_pre_election: true
I20260807 22:42:48.595031 5777 leader_election.cc:302] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74d399adb1f04e74a30f9f0e04af7b63; no voters: 25ec82ddc9ae44d58cdb78786e71c3c1, 3f1146d061404977aafac330bcf1827e
I20260807 22:42:48.595721 6252 raft_consensus.cc:2726] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 22:42:48.603655 5965 leader_election.cc:334] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25ec82ddc9ae44d58cdb78786e71c3c1 (127.4.103.1:37385): Timed out: RequestConsensusVote RPC to 127.4.103.1:37385 timed out after 0.080s (SENT)
W20260807 22:42:48.623934 5965 leader_election.cc:334] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 25ec82ddc9ae44d58cdb78786e71c3c1 (127.4.103.1:37385): Timed out: RequestConsensusVote RPC to 127.4.103.1:37385 timed out after 0.090s (SENT)
W20260807 22:42:48.633152 5773 connection.cc:535] server connection from 127.4.103.1:51791 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 22:42:48.633492 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 5747
I20260807 22:42:48.634987 5935 generic_service.cc:196] Checking for leaks (request via RPC)
W20260807 22:42:48.636291 5965 consensus_peers.cc:487] T 4bdb144053e74012aae2545f2b1eeefb P 3f1146d061404977aafac330bcf1827e -> Peer 25ec82ddc9ae44d58cdb78786e71c3c1 (127.4.103.1:37385): Couldn't send request to peer 25ec82ddc9ae44d58cdb78786e71c3c1. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.1:37385: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 22:42:48.761276 6252 raft_consensus.cc:482] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3f1146d061404977aafac330bcf1827e)
I20260807 22:42:48.761646 6252 raft_consensus.cc:504] T 4bdb144053e74012aae2545f2b1eeefb P 74d399adb1f04e74a30f9f0e04af7b63 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74d399adb1f04e74a30f9f0e04af7b63" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 34067 } } peers { permanent_uuid: "25ec82ddc9ae44d58cdb78786e71c3c1" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 37385 } } peers { permanent_uuid: "3f1146d061404977aafac330bcf1827e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 43199 } }
I20260807 22:42:48.762056 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 5747
I20260807 22:42:48.794166 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 5946
I20260807 22:42:48.795709 6141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:42:48.936643 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 5946
I20260807 22:42:48.973541 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 5414
I20260807 22:42:48.975023 5507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:42:49.107625 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 5414
2026-08-07T22:42:49Z chronyd exiting
[ OK ] TxnStatusManagerITest.ToggleStaleTxnTrackerInRuntime (6870 ms)
[ RUN ] TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances
2026-08-07T22:42:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:42:49Z Disabled control of system clock
I20260807 22:42:49.212543 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.103.62:36831
--webserver_interface=127.4.103.62
--webserver_port=0
--builtin_ntp_servers=127.4.103.20:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.103.62:36831
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false with env {}
W20260807 22:42:49.633667 6294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:42:49.634106 6294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:42:49.634195 6294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:42:49.642443 6294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:42:49.642594 6294 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 22:42:49.642634 6294 flags.cc:425] Enabled experimental flag: --txn_manager_lazily_initialized=false
W20260807 22:42:49.642685 6294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:42:49.642735 6294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:42:49.642812 6294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:42:49.653559 6294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.103.62:36831
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.103.62:36831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.103.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:42:49.655414 6294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:42:49.657712 6294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:42:49.672551 6299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:49.672528 6300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:49.672587 6303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:42:49.674249 6294 server_base.cc:1034] running on GCE node
I20260807 22:42:49.675351 6294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:42:49.677423 6294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:42:49.678735 6294 hybrid_clock.cc:648] HybridClock initialized: now 1786142569678655 us; error 51 us; skew 500 ppm
I20260807 22:42:49.679212 6294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:42:49.688757 6294 webserver.cc:460] Webserver started at http://127.4.103.62:42653/ using document root <none> and password file <none>
I20260807 22:42:49.689891 6294 fs_manager.cc:359] Metadata directory not provided
I20260807 22:42:49.690090 6294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:42:49.690505 6294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:42:49.694207 6294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/data/instance:
uuid: "990d28323d364c14b95e93d5f1333323"
format_stamp: "Formatted at 2026-08-07 22:42:49 on dist-test-slave-kvfs"
I20260807 22:42:49.695183 6294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/wal/instance:
uuid: "990d28323d364c14b95e93d5f1333323"
format_stamp: "Formatted at 2026-08-07 22:42:49 on dist-test-slave-kvfs"
I20260807 22:42:49.702154 6294 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260807 22:42:49.706666 6310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:49.708724 6294 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:42:49.709007 6294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/wal
uuid: "990d28323d364c14b95e93d5f1333323"
format_stamp: "Formatted at 2026-08-07 22:42:49 on dist-test-slave-kvfs"
I20260807 22:42:49.709280 6294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:42:49.732697 6294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:42:49.733946 6294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:42:49.734335 6294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:42:49.760421 6294 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.62:36831
I20260807 22:42:49.760425 6400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.62:36831 every 8 connection(s)
I20260807 22:42:49.763094 6294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/data/info.pb
I20260807 22:42:49.764367 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 6294
I20260807 22:42:49.764593 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/master-0/wal/instance
I20260807 22:42:49.769927 6401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:42:49.782765 6401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323: Bootstrap starting.
I20260807 22:42:49.786829 6401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323: Neither blocks nor log segments found. Creating new log.
I20260807 22:42:49.788199 6401 log.cc:824] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323: Log is configured to *not* fsync() on all Append() calls
I20260807 22:42:49.791622 6401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323: No bootstrap required, opened a new log
I20260807 22:42:49.799742 6401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990d28323d364c14b95e93d5f1333323" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 36831 } }
I20260807 22:42:49.800294 6401 raft_consensus.cc:374] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:42:49.800432 6401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 990d28323d364c14b95e93d5f1333323, State: Initialized, Role: FOLLOWER
I20260807 22:42:49.801103 6401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990d28323d364c14b95e93d5f1333323" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 36831 } }
I20260807 22:42:49.801368 6401 raft_consensus.cc:388] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:42:49.801535 6401 raft_consensus.cc:482] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:42:49.801731 6401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:42:49.804347 6401 raft_consensus.cc:504] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990d28323d364c14b95e93d5f1333323" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 36831 } }
I20260807 22:42:49.804992 6401 leader_election.cc:302] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 990d28323d364c14b95e93d5f1333323; no voters:
I20260807 22:42:49.805691 6401 leader_election.cc:288] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:42:49.805855 6407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:42:49.806663 6407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [term 1 LEADER]: Becoming Leader. State: Replica: 990d28323d364c14b95e93d5f1333323, State: Running, Role: LEADER
I20260807 22:42:49.807243 6407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990d28323d364c14b95e93d5f1333323" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 36831 } }
I20260807 22:42:49.808434 6401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:42:49.811055 6408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "990d28323d364c14b95e93d5f1333323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990d28323d364c14b95e93d5f1333323" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 36831 } } }
I20260807 22:42:49.811028 6409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 990d28323d364c14b95e93d5f1333323. Latest consensus state: current_term: 1 leader_uuid: "990d28323d364c14b95e93d5f1333323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "990d28323d364c14b95e93d5f1333323" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 36831 } } }
I20260807 22:42:49.811575 6408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [sys.catalog]: This master's current role is: LEADER
I20260807 22:42:49.811583 6409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323 [sys.catalog]: This master's current role is: LEADER
I20260807 22:42:49.814479 6421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:42:49.821918 6421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:42:49.836385 6421 catalog_manager.cc:1347] Generated new cluster ID: 937b8d73ce38422ab40497b39a918121
I20260807 22:42:49.836581 6421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:42:49.861825 6421 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:42:49.863773 6421 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:42:49.875053 6421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 990d28323d364c14b95e93d5f1333323: Generated new TSK 0
I20260807 22:42:49.876263 6421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:42:49.881938 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.1:0
--local_ip_for_outbound_sockets=127.4.103.1
--webserver_interface=127.4.103.1
--webserver_port=0
--tserver_master_addrs=127.4.103.62:36831
--builtin_ntp_servers=127.4.103.20:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
I20260807 22:42:49.887209 6341 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41792:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:42:49.905213 6401 master.cc:440] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 3; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20260807 22:42:50.322542 6444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:42:50.322968 6444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:42:50.323120 6444 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:42:50.323218 6444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:42:50.331477 6444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:42:50.331652 6444 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:42:50.331768 6444 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:42:50.331840 6444 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:42:50.331993 6444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.1
I20260807 22:42:50.342644 6444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.103.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:36831
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.1
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:42:50.344622 6444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:42:50.347028 6444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:42:50.363997 6453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:50.364063 6452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:50.363945 6455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:42:50.365345 6444 server_base.cc:1034] running on GCE node
I20260807 22:42:50.366299 6444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:42:50.367637 6444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:42:50.368922 6444 hybrid_clock.cc:648] HybridClock initialized: now 1786142570368813 us; error 64 us; skew 500 ppm
I20260807 22:42:50.369387 6444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:42:50.373014 6444 webserver.cc:460] Webserver started at http://127.4.103.1:42603/ using document root <none> and password file <none>
I20260807 22:42:50.374073 6444 fs_manager.cc:359] Metadata directory not provided
I20260807 22:42:50.374276 6444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:42:50.374745 6444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:42:50.381026 6444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data/instance:
uuid: "753b63233367448f8f28e3a1719be1af"
format_stamp: "Formatted at 2026-08-07 22:42:50 on dist-test-slave-kvfs"
I20260807 22:42:50.382010 6444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal/instance:
uuid: "753b63233367448f8f28e3a1719be1af"
format_stamp: "Formatted at 2026-08-07 22:42:50 on dist-test-slave-kvfs"
I20260807 22:42:50.388957 6444 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260807 22:42:50.393262 6463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:50.395113 6444 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260807 22:42:50.395346 6444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
uuid: "753b63233367448f8f28e3a1719be1af"
format_stamp: "Formatted at 2026-08-07 22:42:50 on dist-test-slave-kvfs"
I20260807 22:42:50.395617 6444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:42:50.416827 6444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:42:50.417986 6444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:42:50.418349 6444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:42:50.421984 6444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:42:50.422163 6444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:50.422307 6444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:42:50.422433 6444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:50.478330 6444 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.1:43211
I20260807 22:42:50.478381 6633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.1:43211 every 8 connection(s)
I20260807 22:42:50.479974 6444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
I20260807 22:42:50.481065 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 6444
I20260807 22:42:50.481328 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal/instance
I20260807 22:42:50.482982 6634 heartbeater.cc:347] Connected to a master server at 127.4.103.62:36831
I20260807 22:42:50.483278 6634 heartbeater.cc:464] Registering TS with master...
I20260807 22:42:50.484277 6634 heartbeater.cc:511] Master 127.4.103.62:36831 requested a full tablet report, sending...
I20260807 22:42:50.485531 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.2:0
--local_ip_for_outbound_sockets=127.4.103.2
--webserver_interface=127.4.103.2
--webserver_port=0
--tserver_master_addrs=127.4.103.62:36831
--builtin_ntp_servers=127.4.103.20:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
I20260807 22:42:50.486770 6341 ts_manager.cc:194] Registered new tserver with Master: 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:42:50.488606 6341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.1:40999
W20260807 22:42:50.899780 6637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:42:50.900296 6637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:42:50.900427 6637 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:42:50.900549 6637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:42:50.908941 6637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:42:50.909112 6637 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:42:50.909196 6637 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:42:50.909261 6637 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:42:50.909431 6637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.2
I20260807 22:42:50.920637 6637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.103.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:36831
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.2
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:42:50.922808 6637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:42:50.925669 6637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 22:42:50.941402 6341 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41806:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:42:50.946296 6650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:50.946226 6649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:50.946254 6652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:42:50.947646 6637 server_base.cc:1034] running on GCE node
I20260807 22:42:50.948637 6637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:42:50.950032 6637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:42:50.951349 6637 hybrid_clock.cc:648] HybridClock initialized: now 1786142570951281 us; error 48 us; skew 500 ppm
I20260807 22:42:50.951763 6637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:42:50.955389 6637 webserver.cc:460] Webserver started at http://127.4.103.2:33969/ using document root <none> and password file <none>
I20260807 22:42:50.956511 6637 fs_manager.cc:359] Metadata directory not provided
I20260807 22:42:50.956689 6637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:42:50.957093 6637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:42:50.960808 6637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data/instance:
uuid: "672a15bd3cdb41c29eb4863f63479a24"
format_stamp: "Formatted at 2026-08-07 22:42:50 on dist-test-slave-kvfs"
I20260807 22:42:50.961953 6637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal/instance:
uuid: "672a15bd3cdb41c29eb4863f63479a24"
format_stamp: "Formatted at 2026-08-07 22:42:50 on dist-test-slave-kvfs"
I20260807 22:42:50.969049 6637 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260807 22:42:50.973444 6659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:50.975317 6637 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260807 22:42:50.975538 6637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
uuid: "672a15bd3cdb41c29eb4863f63479a24"
format_stamp: "Formatted at 2026-08-07 22:42:50 on dist-test-slave-kvfs"
I20260807 22:42:50.975811 6637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:42:51.006327 6637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:42:51.007788 6637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:42:51.008239 6637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:42:51.012132 6637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:42:51.012246 6637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:51.012395 6637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:42:51.012496 6637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:51.065761 6637 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.2:38723
I20260807 22:42:51.065897 6817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.2:38723 every 8 connection(s)
I20260807 22:42:51.067507 6637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
I20260807 22:42:51.070938 6818 heartbeater.cc:347] Connected to a master server at 127.4.103.62:36831
I20260807 22:42:51.071286 6818 heartbeater.cc:464] Registering TS with master...
I20260807 22:42:51.072350 6818 heartbeater.cc:511] Master 127.4.103.62:36831 requested a full tablet report, sending...
I20260807 22:42:51.074199 6341 ts_manager.cc:194] Registered new tserver with Master: 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723)
I20260807 22:42:51.075237 6341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.2:55019
I20260807 22:42:51.077116 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 6637
I20260807 22:42:51.077342 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal/instance
I20260807 22:42:51.081477 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.3:0
--local_ip_for_outbound_sockets=127.4.103.3
--webserver_interface=127.4.103.3
--webserver_port=0
--tserver_master_addrs=127.4.103.62:36831
--builtin_ntp_servers=127.4.103.20:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
I20260807 22:42:51.493028 6634 heartbeater.cc:503] Master 127.4.103.62:36831 was elected leader, sending a full tablet report...
W20260807 22:42:51.513605 6820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:42:51.514061 6820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:42:51.514213 6820 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:42:51.514324 6820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:42:51.522812 6820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:42:51.523065 6820 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:42:51.523173 6820 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:42:51.523247 6820 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:42:51.523412 6820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.3
I20260807 22:42:51.534358 6820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.103.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:36831
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.3
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:42:51.536406 6820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:42:51.538908 6820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:42:51.556391 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:51.556378 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:42:51.557169 6830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:42:51.557507 6820 server_base.cc:1034] running on GCE node
I20260807 22:42:51.558475 6820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:42:51.559890 6820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:42:51.561182 6820 hybrid_clock.cc:648] HybridClock initialized: now 1786142571561093 us; error 95 us; skew 500 ppm
I20260807 22:42:51.561574 6820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:42:51.564901 6820 webserver.cc:460] Webserver started at http://127.4.103.3:33269/ using document root <none> and password file <none>
I20260807 22:42:51.565835 6820 fs_manager.cc:359] Metadata directory not provided
I20260807 22:42:51.565963 6820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:42:51.566299 6820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:42:51.569880 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data/instance:
uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
format_stamp: "Formatted at 2026-08-07 22:42:51 on dist-test-slave-kvfs"
I20260807 22:42:51.570874 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/wal/instance:
uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
format_stamp: "Formatted at 2026-08-07 22:42:51 on dist-test-slave-kvfs"
I20260807 22:42:51.577772 6820 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260807 22:42:51.582187 6840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:51.584107 6820 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:42:51.584313 6820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/wal
uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
format_stamp: "Formatted at 2026-08-07 22:42:51 on dist-test-slave-kvfs"
I20260807 22:42:51.584530 6820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:42:51.614580 6820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:42:51.615738 6820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:42:51.616120 6820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:42:51.619446 6820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:42:51.619553 6820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:51.619735 6820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:42:51.619834 6820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:42:51.679379 6820 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.3:42621
I20260807 22:42:51.679422 6993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.3:42621 every 8 connection(s)
I20260807 22:42:51.681105 6820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
I20260807 22:42:51.683712 6995 heartbeater.cc:347] Connected to a master server at 127.4.103.62:36831
I20260807 22:42:51.683979 6995 heartbeater.cc:464] Registering TS with master...
I20260807 22:42:51.684859 6995 heartbeater.cc:511] Master 127.4.103.62:36831 requested a full tablet report, sending...
I20260807 22:42:51.686094 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 6820
I20260807 22:42:51.686332 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/wal/instance
I20260807 22:42:51.686672 6341 ts_manager.cc:194] Registered new tserver with Master: d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:51.687893 6341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.3:33471
I20260807 22:42:51.690106 4508 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:42:51.959198 6341 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60560:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:42:51.960273 6341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:42:51.991648 6907 tablet_service.cc:1467] Processing CreateTablet for tablet c5fd68d798cc4cc68c2db34d14074bd4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1995bef3080f465e97a5680d79a8f97a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:42:51.991657 6738 tablet_service.cc:1467] Processing CreateTablet for tablet c5fd68d798cc4cc68c2db34d14074bd4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1995bef3080f465e97a5680d79a8f97a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:42:51.992323 6538 tablet_service.cc:1467] Processing CreateTablet for tablet c5fd68d798cc4cc68c2db34d14074bd4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1995bef3080f465e97a5680d79a8f97a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:42:51.993953 6738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fd68d798cc4cc68c2db34d14074bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:42:51.993956 6907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fd68d798cc4cc68c2db34d14074bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:42:51.994027 6538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5fd68d798cc4cc68c2db34d14074bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:42:52.008834 7027 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: Bootstrap starting.
I20260807 22:42:52.012967 7027 tablet_bootstrap.cc:654] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: Neither blocks nor log segments found. Creating new log.
I20260807 22:42:52.014552 7027 log.cc:824] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: Log is configured to *not* fsync() on all Append() calls
I20260807 22:42:52.016427 7028 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Bootstrap starting.
I20260807 22:42:52.017324 7029 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Bootstrap starting.
I20260807 22:42:52.018437 7027 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: No bootstrap required, opened a new log
I20260807 22:42:52.018761 7027 ts_tablet_manager.cc:1397] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260807 22:42:52.023885 7028 tablet_bootstrap.cc:654] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Neither blocks nor log segments found. Creating new log.
I20260807 22:42:52.025352 7029 tablet_bootstrap.cc:654] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Neither blocks nor log segments found. Creating new log.
I20260807 22:42:52.025452 7028 log.cc:824] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Log is configured to *not* fsync() on all Append() calls
I20260807 22:42:52.026616 7029 log.cc:824] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Log is configured to *not* fsync() on all Append() calls
I20260807 22:42:52.029203 7029 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: No bootstrap required, opened a new log
I20260807 22:42:52.029218 7028 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: No bootstrap required, opened a new log
I20260807 22:42:52.029536 7029 ts_tablet_manager.cc:1397] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260807 22:42:52.029645 7028 ts_tablet_manager.cc:1397] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.001s
I20260807 22:42:52.030536 7027 raft_consensus.cc:348] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.031322 7027 raft_consensus.cc:374] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:42:52.031512 7027 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6aee0f79fef46c5a7ba0d9714dc0a5e, State: Initialized, Role: FOLLOWER
I20260807 22:42:52.032435 7027 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.034008 6995 heartbeater.cc:503] Master 127.4.103.62:36831 was elected leader, sending a full tablet report...
I20260807 22:42:52.034083 7033 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:52.034543 7033 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:52.035187 7027 ts_tablet_manager.cc:1428] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: Time spent starting tablet: real 0.016s user 0.017s sys 0.001s
I20260807 22:42:52.040038 7029 raft_consensus.cc:348] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.040649 7029 raft_consensus.cc:374] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:42:52.040761 7029 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 753b63233367448f8f28e3a1719be1af, State: Initialized, Role: FOLLOWER
I20260807 22:42:52.041513 7029 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.043241 7028 raft_consensus.cc:348] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.043795 7028 raft_consensus.cc:374] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:42:52.043928 7028 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672a15bd3cdb41c29eb4863f63479a24, State: Initialized, Role: FOLLOWER
I20260807 22:42:52.043963 7029 ts_tablet_manager.cc:1428] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260807 22:42:52.044469 7034 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:52.044844 7034 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:52.044792 7028 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.046089 6818 heartbeater.cc:503] Master 127.4.103.62:36831 was elected leader, sending a full tablet report...
I20260807 22:42:52.046703 7028 ts_tablet_manager.cc:1428] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260807 22:42:52.049532 7035 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:52.049994 7035 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:52.051684 7034 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:42:52.051947 7034 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.054150 7034 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:52.064131 7035 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:42:52.064584 7035 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.067029 7035 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
W20260807 22:42:52.070628 6819 tablet.cc:2369] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:42:52.074847 7033 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:42:52.075201 7033 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.077476 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:42:52.078156 6933 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 753b63233367448f8f28e3a1719be1af in term 0.
I20260807 22:42:52.079109 6468 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:42:52.079629 7034 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:42:52.079783 7034 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:42:52.079878 7034 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:42:52.083691 7034 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.085216 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:42:52.085451 6933 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:42:52.088325 7033 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:42:52.088879 6933 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 753b63233367448f8f28e3a1719be1af in term 1.
I20260807 22:42:52.089426 6468 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:42:52.096192 7034 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 1 election: Requested vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:52.096562 7034 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:42:52.096765 7034 raft_consensus.cc:686] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 1 LEADER]: Becoming Leader. State: Replica: 753b63233367448f8f28e3a1719be1af, State: Running, Role: LEADER
I20260807 22:42:52.097630 7034 consensus_queue.cc:237] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.105293 6761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:42:52.105978 6761 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in term 0.
I20260807 22:42:52.107170 6841 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:42:52.107851 7033 raft_consensus.cc:2741] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260807 22:42:52.108372 7051 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 1 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:52.108855 7051 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:52.115808 6761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:42:52.116199 6761 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 753b63233367448f8f28e3a1719be1af in term 0.
I20260807 22:42:52.116854 6761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:42:52.117105 6761 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:42:52.121824 6761 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 753b63233367448f8f28e3a1719be1af in term 1.
I20260807 22:42:52.125634 7053 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:42:52.126088 7053 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:42:52.128630 6570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
W20260807 22:42:52.133002 6667 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.4.103.3:42621 for RPC RequestConsensusVote timed out after 0.065s (ON_OUTBOUND_QUEUE)
I20260807 22:42:52.136807 6570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:52.138024 6339 catalog_manager.cc:5495] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af reported cstate change: term changed from 0 to 1, leader changed from <none> to 753b63233367448f8f28e3a1719be1af (127.4.103.1). New cstate: current_term: 1 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260807 22:42:52.153955 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:42:52.154434 6933 raft_consensus.cc:2379] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 672a15bd3cdb41c29eb4863f63479a24 in current term 1: Already voted for candidate 753b63233367448f8f28e3a1719be1af in this term.
I20260807 22:42:52.155202 6667 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24; no voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:42:52.155720 7035 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260807 22:42:52.164223 7051 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:42:52.170096 7035 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 1 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:52.170394 7035 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:52.174012 7034 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:52.174386 7034 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:52.174507 7035 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:52.174846 7035 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:52.175683 7033 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:42:52.175827 7051 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:42:52.175901 7033 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.176787 7033 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:42:52.178305 6570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:52.179402 6760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:42:52.181859 6841 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af
W20260807 22:42:52.183776 6996 tablet.cc:2369] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:42:52.189313 7033 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 22:42:52.190073 7081 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:52.190443 7081 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:52.191156 7072 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 1 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:52.191565 7072 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:52.210796 7034 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:52.211256 7034 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:52.282568 7035 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:42:52.282888 7035 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.283617 7072 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:42:52.283913 7035 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:52.283933 7072 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.284765 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:42:52.284904 7072 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:42:52.285076 6933 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 1.
I20260807 22:42:52.285485 6760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:42:52.285615 6667 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:42:52.285821 6760 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in term 1.
I20260807 22:42:52.286019 7035 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 22:42:52.286183 7035 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:42:52.286303 7035 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:42:52.286336 6841 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:42:52.286639 7072 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 22:42:52.286777 7072 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:42:52.286911 7072 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 1 FOLLOWER]: Advancing to term 2
W20260807 22:42:52.287154 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
I20260807 22:42:52.289490 7072 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.289525 7035 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.290184 7072 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 2 election: Requested vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:42:52.290433 7035 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 2 election: Requested vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:52.290769 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:42:52.290768 6760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:42:52.291035 6933 raft_consensus.cc:2379] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 672a15bd3cdb41c29eb4863f63479a24 in current term 2: Already voted for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in this term.
I20260807 22:42:52.291102 6760 raft_consensus.cc:2379] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in current term 2: Already voted for candidate 672a15bd3cdb41c29eb4863f63479a24 in this term.
W20260807 22:42:52.297359 7006 meta_cache.cc:302] tablet c5fd68d798cc4cc68c2db34d14074bd4: replica 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211) has failed: Timed out: CoordinateTransaction RPC to 127.4.103.1:43211 timed out after 0.049s (SENT)
W20260807 22:42:52.298278 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 0) took 52 ms (client timeout 50 ms). Trace:
W20260807 22:42:52.298477 6384 rpcz_store.cc:269] 0807 22:42:52.245677 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:42:52.245764 (+ 87us) service_pool.cc:227] Handling call
0807 22:42:52.298182 (+ 52418us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:42:52.359143 6841 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.068s (SENT)
W20260807 22:42:52.429167 6841 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.068s (SENT)
I20260807 22:42:52.429448 6841 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af
I20260807 22:42:52.430011 7072 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260807 22:42:52.439059 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
W20260807 22:42:52.439674 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 1) took 50 ms (client timeout 50 ms). Trace:
W20260807 22:42:52.439851 6384 rpcz_store.cc:269] 0807 22:42:52.388811 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:42:52.388905 (+ 94us) service_pool.cc:227] Handling call
0807 22:42:52.439549 (+ 50644us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:42:52.440241 6662 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.074s (SENT)
I20260807 22:42:52.440515 6662 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24; no voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:42:52.440991 7035 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260807 22:42:52.458652 6841 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.086s (SENT)
W20260807 22:42:52.464565 6662 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.089s (SENT)
I20260807 22:42:52.513576 7072 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:42:52.514074 7072 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.514994 7072 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:42:52.515931 6760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:42:52.516270 6760 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in term 2.
I20260807 22:42:52.516772 6841 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:42:52.517107 7072 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 22:42:52.517207 7072 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:42:52.517354 7072 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 2 FOLLOWER]: Advancing to term 3
I20260807 22:42:52.519804 7072 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.520627 7072 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 3 election: Requested vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:42:52.521230 6760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:42:52.521492 6760 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 2 FOLLOWER]: Advancing to term 3
I20260807 22:42:52.523725 6760 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in term 3.
I20260807 22:42:52.524255 6841 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:42:52.524561 7072 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 3 FOLLOWER]: Leader election won for term 3
I20260807 22:42:52.524798 7072 raft_consensus.cc:686] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 3 LEADER]: Becoming Leader. State: Replica: d6aee0f79fef46c5a7ba0d9714dc0a5e, State: Running, Role: LEADER
I20260807 22:42:52.525403 7072 consensus_queue.cc:237] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:52.527029 7081 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: New leader d6aee0f79fef46c5a7ba0d9714dc0a5e. Latest consensus state: current_term: 3 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:52.527338 7081 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:52.528085 7100 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:42:52.528370 7100 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:42:52.529441 6341 catalog_manager.cc:5495] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e reported cstate change: term changed from 1 to 3, leader changed from 753b63233367448f8f28e3a1719be1af (127.4.103.1) to d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3). New cstate: current_term: 3 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } health_report { overall_health: HEALTHY } } }
I20260807 22:42:52.580212 6760 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 3 FOLLOWER]: Refusing update from remote peer d6aee0f79fef46c5a7ba0d9714dc0a5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260807 22:42:52.581255 7081 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 22:42:52.583515 7035 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: New leader d6aee0f79fef46c5a7ba0d9714dc0a5e. Latest consensus state: current_term: 3 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:52.583849 7035 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:52.586544 7035 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:52.586856 7035 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:52.586973 7033 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:52.587316 7033 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: LEADER
W20260807 22:42:52.652801 6841 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.068s (SENT)
W20260807 22:42:52.785117 6841 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.131s (SENT)
I20260807 22:42:53.252228 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:53.252622 6570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:53.252954 6565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:53.253683 6933 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 3 LEADER]: Rejecting Update request from peer 753b63233367448f8f28e3a1719be1af for earlier term 1. Current term is 3. Ops: []
I20260807 22:42:53.254797 6760 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 3 FOLLOWER]: Rejecting Update request from peer 753b63233367448f8f28e3a1719be1af for earlier term 1. Current term is 3. Ops: []
I20260807 22:42:53.255169 7082 consensus_queue.cc:1046] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 3, Time since last communication: 0.001s
I20260807 22:42:53.255514 7082 consensus_queue.cc:1225] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Peer d6aee0f79fef46c5a7ba0d9714dc0a5e log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260807 22:42:53.255775 7051 consensus_queue.cc:1046] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 22:42:53.260226 7082 raft_consensus.cc:3032] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 1 LEADER]: Stepping down as leader of term 1
I20260807 22:42:53.257972 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:42:53.260735 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:42:53.260782 6566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:42:53.261199 6565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:53.262189 7082 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 753b63233367448f8f28e3a1719be1af, State: Running, Role: LEADER
I20260807 22:42:53.262216 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:53.262627 6570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:42:53.263164 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:42:53.263082 7082 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:53.263962 7082 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 1 FOLLOWER]: Advancing to term 3
I20260807 22:42:53.264767 6565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:42:53.265549 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:53.272210 6570 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 3 FOLLOWER]: Refusing update from remote peer d6aee0f79fef46c5a7ba0d9714dc0a5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260807 22:42:53.273445 6471 txn_status_manager.cc:1396] failed to abort stale txn (ID 0) past 1.030s from last keepalive heartbeat (effective timeout is 0.500s): Illegal state: Replica 753b63233367448f8f28e3a1719be1af is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:53.273648 6471 txn_status_manager.cc:1406] skipping staleness check for the rest of in-flight txn records since this txn status tablet replica is no longer a leader or not running
I20260807 22:42:53.273890 7081 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 22:42:53.277590 7034 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:53.277943 7034 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:53.284137 7082 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: New leader d6aee0f79fef46c5a7ba0d9714dc0a5e. Latest consensus state: current_term: 3 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:53.284516 7082 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:53.325146 7081 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:53.325579 7081 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:54.341782 7109 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d6aee0f79fef46c5a7ba0d9714dc0a5e)
I20260807 22:42:54.341784 7111 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d6aee0f79fef46c5a7ba0d9714dc0a5e)
I20260807 22:42:54.342078 7111 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:54.342075 7109 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:54.342935 7111 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:54.343114 7109 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:54.343638 6570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:54.343767 6760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:42:54.344004 6570 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 3.
I20260807 22:42:54.344129 6760 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 753b63233367448f8f28e3a1719be1af in term 3.
I20260807 22:42:54.344594 6662 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af; no voters:
I20260807 22:42:54.344982 7109 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 22:42:54.345088 6464 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af; no voters:
I20260807 22:42:54.345218 7109 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d6aee0f79fef46c5a7ba0d9714dc0a5e)
I20260807 22:42:54.345368 7109 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 3 FOLLOWER]: Advancing to term 4
I20260807 22:42:54.345450 7111 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260807 22:42:54.345607 7111 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 3 FOLLOWER]: Starting leader election (detected failure of leader d6aee0f79fef46c5a7ba0d9714dc0a5e)
I20260807 22:42:54.345724 7111 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 3 FOLLOWER]: Advancing to term 4
I20260807 22:42:54.347915 7109 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:54.348678 7109 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 4 election: Requested vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:54.348562 7111 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:54.349239 6570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:42:54.349581 6570 raft_consensus.cc:2379] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 672a15bd3cdb41c29eb4863f63479a24 in current term 4: Already voted for candidate 753b63233367448f8f28e3a1719be1af in this term.
I20260807 22:42:54.349651 6760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:42:54.349229 7111 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 4 election: Requested vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:54.349952 6760 raft_consensus.cc:2379] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 753b63233367448f8f28e3a1719be1af in current term 4: Already voted for candidate 672a15bd3cdb41c29eb4863f63479a24 in this term.
W20260807 22:42:54.369665 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
W20260807 22:42:54.369894 7010 meta_cache.cc:302] tablet c5fd68d798cc4cc68c2db34d14074bd4: replica d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621) has failed: Timed out: CoordinateTransaction RPC to 127.4.103.3:42621 timed out after 0.049s (SENT)
W20260807 22:42:54.370609 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 19) took 51 ms (client timeout 50 ms). Trace:
W20260807 22:42:54.370769 6384 rpcz_store.cc:269] 0807 22:42:54.319248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:42:54.319374 (+ 126us) service_pool.cc:227] Handling call
0807 22:42:54.370449 (+ 51075us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:42:54.417809 6468 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.103.3:42621 timed out after 0.074s (SENT)
W20260807 22:42:54.475342 6667 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621): Timed out: RequestConsensusVote RPC to 127.4.103.3:42621 timed out after 0.062s (SENT)
I20260807 22:42:54.475581 6667 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24; no voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:42:54.476150 7109 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260807 22:42:54.493451 6468 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621): Timed out: RequestConsensusVote RPC to 127.4.103.3:42621 timed out after 0.071s (SENT)
I20260807 22:42:54.493775 6468 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 753b63233367448f8f28e3a1719be1af; no voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:42:54.494329 7111 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260807 22:42:54.495635 6468 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621): Timed out: RequestConsensusVote RPC to 127.4.103.3:42621 timed out after 0.074s (SENT)
W20260807 22:42:54.498961 6667 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621): Timed out: RequestConsensusVote RPC to 127.4.103.3:42621 timed out after 0.077s (SENT)
W20260807 22:42:54.521353 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
W20260807 22:42:54.522781 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 20) took 51 ms (client timeout 50 ms). Trace:
W20260807 22:42:54.522919 6384 rpcz_store.cc:269] 0807 22:42:54.471134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:42:54.471259 (+ 125us) service_pool.cc:227] Handling call
0807 22:42:54.522674 (+ 51415us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:42:54.542143 7109 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:42:54.542547 7109 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:54.543391 7109 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:54.544267 6570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:54.544651 6570 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 4.
I20260807 22:42:54.545233 6662 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af; no voters:
I20260807 22:42:54.545553 7109 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260807 22:42:54.545639 7109 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:42:54.545724 7109 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 4 FOLLOWER]: Advancing to term 5
I20260807 22:42:54.548581 7109 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:54.549332 7109 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 5 election: Requested vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:54.549933 6570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:42:54.550203 6570 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 4 FOLLOWER]: Advancing to term 5
I20260807 22:42:54.553032 6570 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 5.
I20260807 22:42:54.553574 6662 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af; no voters:
I20260807 22:42:54.553882 7109 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 5 FOLLOWER]: Leader election won for term 5
I20260807 22:42:54.554461 7109 raft_consensus.cc:686] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 5 LEADER]: Becoming Leader. State: Replica: 672a15bd3cdb41c29eb4863f63479a24, State: Running, Role: LEADER
I20260807 22:42:54.555076 7109 consensus_queue.cc:237] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:54.557173 7114 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 5 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:54.557571 7114 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:54.558627 7116 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:42:54.558908 7116 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:42:54.559788 6341 catalog_manager.cc:5495] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 reported cstate change: term changed from 3 to 5, leader changed from d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3) to 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2). New cstate: current_term: 5 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260807 22:42:54.611761 6570 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 5 FOLLOWER]: Refusing update from remote peer 672a15bd3cdb41c29eb4863f63479a24: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 22:42:54.612843 7114 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 22:42:54.615868 7111 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 5 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:54.616317 7111 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:54.619593 7109 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:54.620116 7109 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:54.621089 7111 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:54.621469 7111 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: FOLLOWER
W20260807 22:42:54.749629 6667 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621): Timed out: RequestConsensusVote RPC to 127.4.103.3:42621 timed out after 0.102s (SENT)
W20260807 22:42:54.807629 6667 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621): Timed out: RequestConsensusVote RPC to 127.4.103.3:42621 timed out after 0.128s (SENT)
I20260807 22:42:55.303707 6927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:42:55.304164 6760 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 5 LEADER]: Rejecting Update request from peer d6aee0f79fef46c5a7ba0d9714dc0a5e for earlier term 3. Current term is 5. Ops: []
I20260807 22:42:55.304651 6923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:42:55.305171 7081 consensus_queue.cc:1046] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 22:42:55.305548 7081 consensus_queue.cc:1225] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [LEADER]: Peer 672a15bd3cdb41c29eb4863f63479a24 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260807 22:42:55.305645 6918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:42:55.306087 6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:42:55.306593 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:42:55.308596 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:42:55.309064 6919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:42:55.309376 6921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:42:55.309962 6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:42:55.310443 6925 raft_consensus.cc:3032] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 3 LEADER]: Stepping down as leader of term 3
I20260807 22:42:55.310571 6925 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d6aee0f79fef46c5a7ba0d9714dc0a5e, State: Running, Role: LEADER
I20260807 22:42:55.310869 6926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:42:55.305291 6928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:42:55.311669 6924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:42:55.312871 6925 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:55.314823 6925 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 3 FOLLOWER]: Advancing to term 5
I20260807 22:42:55.315212 6570 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 5 FOLLOWER]: Rejecting Update request from peer d6aee0f79fef46c5a7ba0d9714dc0a5e for earlier term 3. Current term is 5. Ops: []
I20260807 22:42:55.318210 6925 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 5 FOLLOWER]: Refusing update from remote peer 672a15bd3cdb41c29eb4863f63479a24: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260807 22:42:55.319011 7114 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260807 22:42:55.322321 7081 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 5 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:55.322844 7081 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:55.322522 7136 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:55.323107 7136 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:55.369469 7114 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:55.369910 7114 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:56.361128 7145 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:42:56.361474 7145 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:56.362428 7145 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:42:56.363502 6570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:56.383862 7146 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:42:56.384214 7146 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:56.385144 7146 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:56.386168 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:42:56.386627 6925 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 753b63233367448f8f28e3a1719be1af in term 5.
I20260807 22:42:56.387449 6468 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:42:56.387903 7146 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260807 22:42:56.388114 7146 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 5 FOLLOWER]: Starting leader election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:42:56.388255 7146 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 5 FOLLOWER]: Advancing to term 6
I20260807 22:42:56.391040 7146 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:56.391914 7146 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 6 election: Requested vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:56.392803 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:42:56.393221 6925 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 5 FOLLOWER]: Advancing to term 6
I20260807 22:42:56.397719 6925 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 753b63233367448f8f28e3a1719be1af in term 6.
I20260807 22:42:56.398525 6468 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:42:56.398989 7146 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 6 FOLLOWER]: Leader election won for term 6
I20260807 22:42:56.399717 7146 raft_consensus.cc:686] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 6 LEADER]: Becoming Leader. State: Replica: 753b63233367448f8f28e3a1719be1af, State: Running, Role: LEADER
I20260807 22:42:56.400209 7146 consensus_queue.cc:237] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:56.402536 7148 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 6 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:56.402953 7148 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:56.404484 7151 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:42:56.404735 7151 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:42:56.405691 6341 catalog_manager.cc:5495] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af reported cstate change: term changed from 5 to 6, leader changed from 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2) to 753b63233367448f8f28e3a1719be1af (127.4.103.1). New cstate: current_term: 6 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260807 22:42:56.447422 6925 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 6 FOLLOWER]: Refusing update from remote peer 753b63233367448f8f28e3a1719be1af: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 22:42:56.448662 7148 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260807 22:42:56.451615 7145 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 6 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:56.452152 7145 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260807 22:42:56.452713 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
W20260807 22:42:56.453265 7006 meta_cache.cc:302] tablet c5fd68d798cc4cc68c2db34d14074bd4: replica 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723) has failed: Timed out: CoordinateTransaction RPC to 127.4.103.2:38723 timed out after 0.049s (SENT)
I20260807 22:42:56.454805 7148 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:56.455155 7148 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: LEADER
W20260807 22:42:56.455235 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 38) took 53 ms (client timeout 50 ms). Trace:
W20260807 22:42:56.455382 6384 rpcz_store.cc:269] 0807 22:42:56.402148 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:42:56.402304 (+ 156us) service_pool.cc:227] Handling call
0807 22:42:56.455130 (+ 52826us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:42:56.457649 7145 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:56.459168 7145 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260807 22:42:56.528357 6464 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Timed out: RequestConsensusVote RPC to 127.4.103.2:38723 timed out after 0.066s (SENT)
W20260807 22:42:56.536366 6464 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Timed out: RequestConsensusVote RPC to 127.4.103.2:38723 timed out after 0.074s (SENT)
W20260807 22:42:56.539430 6841 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Timed out: RequestConsensusVote RPC to 127.4.103.2:38723 timed out after 0.087s (SENT)
I20260807 22:42:56.539696 6841 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af
I20260807 22:42:56.540310 7145 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260807 22:42:57.336853 6761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:42:57.337221 6760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:42:57.342450 6761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:42:57.343324 6761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:42:57.343611 6760 raft_consensus.cc:3032] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 5 LEADER]: Stepping down as leader of term 5
I20260807 22:42:57.343761 6760 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 672a15bd3cdb41c29eb4863f63479a24, State: Running, Role: LEADER
I20260807 22:42:57.343978 6759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:42:57.344233 6760 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:57.344617 6759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:42:57.344925 6760 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 5 FOLLOWER]: Advancing to term 6
I20260807 22:42:57.348188 6760 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 6 FOLLOWER]: Refusing update from remote peer 753b63233367448f8f28e3a1719be1af: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260807 22:42:57.348287 6925 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 6 FOLLOWER]: Rejecting Update request from peer 672a15bd3cdb41c29eb4863f63479a24 for earlier term 5. Current term is 6. Ops: []
I20260807 22:42:57.349206 7146 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260807 22:42:57.350329 6670 txn_status_manager.cc:1396] failed to abort stale txn (ID 0) past 1.033s from last keepalive heartbeat (effective timeout is 0.500s): Illegal state: Replica 672a15bd3cdb41c29eb4863f63479a24 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:57.350548 6670 txn_status_manager.cc:1406] skipping staleness check for the rest of in-flight txn records since this txn status tablet replica is no longer a leader or not running
I20260807 22:42:57.351775 7164 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 6 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:57.352201 7164 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:57.352111 7163 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:57.352574 7163 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:57.410676 7167 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:57.411144 7167 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:58.390877 7175 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:42:58.391179 7177 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:42:58.391211 7175 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:58.391484 7177 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:58.392225 7175 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:58.392380 7177 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:42:58.392968 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:42:58.393054 6760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:42:58.393378 6760 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in term 6.
I20260807 22:42:58.393380 6925 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 6.
I20260807 22:42:58.393896 6841 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:42:58.393936 6667 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:42:58.394268 7175 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 22:42:58.394253 7177 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260807 22:42:58.394428 7175 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:42:58.394526 7177 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:42:58.394556 7175 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 6 FOLLOWER]: Advancing to term 7
I20260807 22:42:58.394646 7177 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 6 FOLLOWER]: Advancing to term 7
I20260807 22:42:58.398203 7177 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:58.398260 7175 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:58.399142 7177 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 7 election: Requested vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:42:58.399123 7175 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 7 election: Requested vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:58.399912 6760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:42:58.399957 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:42:58.400269 6925 raft_consensus.cc:2379] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 672a15bd3cdb41c29eb4863f63479a24 in current term 7: Already voted for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in this term.
I20260807 22:42:58.400333 6760 raft_consensus.cc:2379] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in current term 7: Already voted for candidate 672a15bd3cdb41c29eb4863f63479a24 in this term.
W20260807 22:42:58.476370 6841 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.083s (SENT) [suppressed 4 similar messages]
W20260807 22:42:58.477914 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
W20260807 22:42:58.478274 7006 meta_cache.cc:302] tablet c5fd68d798cc4cc68c2db34d14074bd4: replica 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211) has failed: Timed out: CoordinateTransaction RPC to 127.4.103.1:43211 timed out after 0.049s (SENT)
W20260807 22:42:58.478787 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 57) took 51 ms (client timeout 50 ms). Trace:
W20260807 22:42:58.478936 6384 rpcz_store.cc:269] 0807 22:42:58.427574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:42:58.427718 (+ 144us) service_pool.cc:227] Handling call
0807 22:42:58.478689 (+ 50971us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:42:58.484967 6662 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.085s (SENT) [suppressed 6 similar messages]
W20260807 22:42:58.561398 6841 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.083s (SENT)
W20260807 22:42:58.563006 6841 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.081s (SENT)
I20260807 22:42:58.563136 6841 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af
I20260807 22:42:58.563545 7177 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260807 22:42:58.572281 6662 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.085s (SENT)
I20260807 22:42:58.572541 6662 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24; no voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:42:58.573100 7175 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260807 22:42:58.578644 6662 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.092s (SENT)
W20260807 22:42:58.629559 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
W20260807 22:42:58.631237 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 58) took 51 ms (client timeout 50 ms). Trace:
W20260807 22:42:58.631541 6384 rpcz_store.cc:269] 0807 22:42:58.579300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:42:58.579445 (+ 145us) service_pool.cc:227] Handling call
0807 22:42:58.630983 (+ 51538us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:42:58.637179 7175 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:42:58.637493 7175 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:58.638233 7175 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:58.639150 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:42:58.639420 6925 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 7.
I20260807 22:42:58.639966 6667 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:42:58.640426 7175 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260807 22:42:58.640550 7175 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:42:58.640730 7175 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 7 FOLLOWER]: Advancing to term 8
I20260807 22:42:58.643218 7175 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:58.644011 7175 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 8 election: Requested vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:42:58.644804 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:42:58.645152 6925 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 7 FOLLOWER]: Advancing to term 8
I20260807 22:42:58.647904 6925 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 8.
I20260807 22:42:58.648578 6667 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:42:58.648969 7175 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 8 FOLLOWER]: Leader election won for term 8
I20260807 22:42:58.649611 7175 raft_consensus.cc:686] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 8 LEADER]: Becoming Leader. State: Replica: 672a15bd3cdb41c29eb4863f63479a24, State: Running, Role: LEADER
I20260807 22:42:58.650010 7175 consensus_queue.cc:237] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:58.651877 7181 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 8 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:58.652256 7181 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:58.652969 7184 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:42:58.653162 7184 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:42:58.653676 6339 catalog_manager.cc:5495] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 reported cstate change: term changed from 6 to 8, leader changed from 753b63233367448f8f28e3a1719be1af (127.4.103.1) to 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2). New cstate: current_term: 8 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260807 22:42:58.697515 6925 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 8 FOLLOWER]: Refusing update from remote peer 672a15bd3cdb41c29eb4863f63479a24: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260807 22:42:58.698580 7181 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 22:42:58.701192 7177 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 8 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:58.701660 7177 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:58.703943 7175 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:58.704362 7175 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:42:58.705485 7177 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:58.705828 7177 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260807 22:42:58.825716 6662 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.090s (SENT)
W20260807 22:42:58.894841 6662 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.127s (SENT)
I20260807 22:42:59.355953 6565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:42:59.359304 6760 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 8 LEADER]: Rejecting Update request from peer 753b63233367448f8f28e3a1719be1af for earlier term 6. Current term is 8. Ops: []
I20260807 22:42:59.359877 6566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:59.360200 6568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:59.359747 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:42:59.360559 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:42:59.361407 6565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:42:59.361622 6570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:59.361644 6567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:42:59.360174 6564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:59.362854 6925 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 8 FOLLOWER]: Rejecting Update request from peer 753b63233367448f8f28e3a1719be1af for earlier term 6. Current term is 8. Ops: []
I20260807 22:42:59.362658 6558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:59.364202 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:42:59.363595 7174 consensus_queue.cc:1046] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 6, Time since last communication: 0.004s
I20260807 22:42:59.364517 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:42:59.366170 7174 consensus_queue.cc:1225] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Peer 672a15bd3cdb41c29eb4863f63479a24 log is divergent from this leader: its last log entry 8.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260807 22:42:59.366405 7197 consensus_queue.cc:1046] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 6, Time since last communication: 0.001s
I20260807 22:42:59.367657 6557 raft_consensus.cc:3032] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 6 LEADER]: Stepping down as leader of term 6
I20260807 22:42:59.367821 6557 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 753b63233367448f8f28e3a1719be1af, State: Running, Role: LEADER
I20260807 22:42:59.368468 6557 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:42:59.368902 6557 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 6 FOLLOWER]: Advancing to term 8
I20260807 22:42:59.371985 6557 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 8 FOLLOWER]: Refusing update from remote peer 672a15bd3cdb41c29eb4863f63479a24: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260807 22:42:59.372277 7174 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260807 22:42:59.373404 7181 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260807 22:42:59.374954 6471 txn_status_manager.cc:1396] failed to abort stale txn (ID 0) past 1.030s from last keepalive heartbeat (effective timeout is 0.500s): Illegal state: Replica 753b63233367448f8f28e3a1719be1af is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:59.377660 7174 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 8 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:59.378116 7174 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:59.378019 7197 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:59.378711 7197 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:42:59.379444 6471 txn_status_manager.cc:1406] skipping staleness check for the rest of in-flight txn records since this txn status tablet replica is no longer a leader or not running
I20260807 22:42:59.426141 7198 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:42:59.426659 7198 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:00.400456 7202 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:43:00.400776 7202 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:00.401788 7202 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:43:00.402655 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:00.421609 7203 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:43:00.422060 7203 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:00.423028 7203 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:00.424003 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:00.424430 6925 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 753b63233367448f8f28e3a1719be1af in term 8.
I20260807 22:43:00.425175 6468 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:43:00.425530 7203 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260807 22:43:00.425659 7203 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 8 FOLLOWER]: Starting leader election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:43:00.425751 7203 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 8 FOLLOWER]: Advancing to term 9
I20260807 22:43:00.428290 7203 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:00.428951 7203 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 9 election: Requested vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:00.429705 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:43:00.430085 6925 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 8 FOLLOWER]: Advancing to term 9
I20260807 22:43:00.433111 6925 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 753b63233367448f8f28e3a1719be1af in term 9.
I20260807 22:43:00.433792 6468 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:43:00.434232 7203 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 9 FOLLOWER]: Leader election won for term 9
I20260807 22:43:00.434767 7203 raft_consensus.cc:686] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 9 LEADER]: Becoming Leader. State: Replica: 753b63233367448f8f28e3a1719be1af, State: Running, Role: LEADER
I20260807 22:43:00.435146 7203 consensus_queue.cc:237] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:00.436815 7204 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 9 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:00.437155 7204 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:00.438252 7206 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:43:00.438478 7206 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:43:00.439277 6341 catalog_manager.cc:5495] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af reported cstate change: term changed from 8 to 9, leader changed from 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2) to 753b63233367448f8f28e3a1719be1af (127.4.103.1). New cstate: current_term: 9 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
W20260807 22:43:00.446820 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
W20260807 22:43:00.446931 7006 meta_cache.cc:302] tablet c5fd68d798cc4cc68c2db34d14074bd4: replica 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723) has failed: Timed out: CoordinateTransaction RPC to 127.4.103.2:38723 timed out after 0.049s (SENT)
W20260807 22:43:00.447506 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 75) took 51 ms (client timeout 50 ms). Trace:
W20260807 22:43:00.447647 6384 rpcz_store.cc:269] 0807 22:43:00.396384 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:00.396504 (+ 120us) service_pool.cc:227] Handling call
0807 22:43:00.447307 (+ 50803us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:00.480731 6925 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 9 FOLLOWER]: Refusing update from remote peer 753b63233367448f8f28e3a1719be1af: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260807 22:43:00.481642 7204 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 22:43:00.484483 7202 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 9 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:00.484936 7202 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:00.486987 7203 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:00.487534 7203 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:00.488268 7202 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:00.488596 7202 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260807 22:43:00.505930 6464 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.103.2:38723 timed out after 0.082s (SENT) [suppressed 3 similar messages]
W20260807 22:43:00.563230 6841 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Timed out: RequestConsensusVote RPC to 127.4.103.2:38723 timed out after 0.079s (SENT)
I20260807 22:43:00.563482 6841 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af
I20260807 22:43:00.563974 7202 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260807 22:43:00.590858 6464 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Timed out: RequestConsensusVote RPC to 127.4.103.2:38723 timed out after 0.082s (SENT)
W20260807 22:43:00.614868 6464 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Timed out: RequestConsensusVote RPC to 127.4.103.2:38723 timed out after 0.092s (SENT)
I20260807 22:43:01.400942 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:01.401365 6759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:43:01.401681 6760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:01.402137 6756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:43:01.402331 6761 raft_consensus.cc:3032] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 8 LEADER]: Stepping down as leader of term 8
I20260807 22:43:01.402464 6761 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 672a15bd3cdb41c29eb4863f63479a24, State: Running, Role: LEADER
I20260807 22:43:01.402956 6761 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:01.403901 6761 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 8 FOLLOWER]: Advancing to term 9
I20260807 22:43:01.404452 6925 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 9 FOLLOWER]: Rejecting Update request from peer 672a15bd3cdb41c29eb4863f63479a24 for earlier term 8. Current term is 9. Ops: []
I20260807 22:43:01.405730 6760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:01.405870 6756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
W20260807 22:43:01.407950 6670 txn_status_manager.cc:1396] failed to abort stale txn (ID 0) past 1.105s from last keepalive heartbeat (effective timeout is 0.500s): Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.005s ago.
I20260807 22:43:01.440385 6761 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 9 FOLLOWER]: Refusing update from remote peer 753b63233367448f8f28e3a1719be1af: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260807 22:43:01.440836 6670 txn_status_manager.cc:1406] skipping staleness check for the rest of in-flight txn records since this txn status tablet replica is no longer a leader or not running
I20260807 22:43:01.442063 7203 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 22:43:01.445410 7198 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 9 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:01.445793 7198 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:01.446218 7222 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 9 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:01.446669 7222 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:01.502017 7203 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 9 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:01.502449 7203 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:02.417964 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 6444
I20260807 22:43:02.419505 6628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:02.457592 7230 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:43:02.457930 7230 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:02.458945 7230 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:43:02.459721 6761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
W20260807 22:43:02.476512 7006 meta_cache.cc:302] tablet c5fd68d798cc4cc68c2db34d14074bd4: replica 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211) has failed: Timed out: CoordinateTransaction RPC to 127.4.103.1:43211 timed out after 0.049s (SENT)
W20260807 22:43:02.476612 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
W20260807 22:43:02.477200 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 94) took 51 ms (client timeout 50 ms). Trace:
W20260807 22:43:02.477304 6384 rpcz_store.cc:269] 0807 22:43:02.426030 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:02.426147 (+ 117us) service_pool.cc:227] Handling call
0807 22:43:02.477124 (+ 50977us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:02.481845 7232 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:43:02.482173 7232 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:02.483381 7232 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:02.484383 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:02.484737 6925 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 9.
I20260807 22:43:02.485401 6667 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:43:02.485886 7232 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260807 22:43:02.486194 7232 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:43:02.486326 7232 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 9 FOLLOWER]: Advancing to term 10
I20260807 22:43:02.548393 7232 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:02.549541 7232 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 10 election: Requested vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:02.550696 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:43:02.551088 6925 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 9 FOLLOWER]: Advancing to term 10
I20260807 22:43:02.580976 6557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:02.581753 6561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:43:02.582290 6559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:02.582799 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 6444
W20260807 22:43:02.603217 6841 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.070s (SENT)
I20260807 22:43:02.603497 6841 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af
I20260807 22:43:02.617506 6924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:43:02.618546 6667 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24; no voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:43:02.619098 7232 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260807 22:43:02.623948 7006 connection.cc:535] client connection to 127.4.103.1:43211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 22:43:02.624043 6841 connection.cc:535] client connection to 127.4.103.1:43211 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 22:43:02.628118 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
W20260807 22:43:02.634326 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 95) took 56 ms (client timeout 50 ms). Trace:
W20260807 22:43:02.634516 6384 rpcz_store.cc:269] 0807 22:43:02.578160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:02.578287 (+ 127us) service_pool.cc:227] Handling call
0807 22:43:02.634197 (+ 55910us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:02.653153 6925 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 10.
W20260807 22:43:02.653628 6925 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.4.103.2:58119 (request call id 75) took 103 ms (client timeout 65 ms). Trace:
I20260807 22:43:02.653854 7230 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260807 22:43:02.653959 6925 rpcz_store.cc:269] 0807 22:43:02.550288 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:02.550416 (+ 128us) service_pool.cc:227] Handling call
0807 22:43:02.653510 (+103094us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:02.683696 7232 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:43:02.684166 7232 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:02.685003 7232 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:02.685886 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:02.686308 6925 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 10.
I20260807 22:43:02.686833 6667 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:43:02.687220 7232 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260807 22:43:02.687338 7232 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:43:02.687506 7232 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 10 FOLLOWER]: Advancing to term 11
W20260807 22:43:02.687664 6662 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Network error: Client connection negotiation failed: client connection to 127.4.103.1:43211: connect: Connection refused (error 111)
I20260807 22:43:02.758384 7232 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:02.759632 7232 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 11 election: Requested vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
W20260807 22:43:02.762369 6662 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Network error: Client connection negotiation failed: client connection to 127.4.103.1:43211: connect: Connection refused (error 111)
I20260807 22:43:02.762801 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:43:02.763180 6925 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 10 FOLLOWER]: Advancing to term 11
W20260807 22:43:02.779806 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
W20260807 22:43:02.817795 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 96) took 88 ms (client timeout 50 ms). Trace:
W20260807 22:43:02.818070 6384 rpcz_store.cc:269] 0807 22:43:02.729490 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:02.729597 (+ 107us) service_pool.cc:227] Handling call
0807 22:43:02.817631 (+ 88034us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:02.864502 6925 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 11.
W20260807 22:43:02.864976 6925 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.4.103.2:58119 (request call id 78) took 104 ms (client timeout 137 ms). Trace:
W20260807 22:43:02.865221 6925 rpcz_store.cc:269] 0807 22:43:02.760346 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:02.760447 (+ 101us) service_pool.cc:227] Handling call
0807 22:43:02.864833 (+104386us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:02.866091 6906 rpcz_store.cc:267] Call kudu.tserver.TabletServerAdminService.CoordinateTransaction from 127.0.0.1:49414 (ReqId={client: 44f6daa64fdd461caaac788274f97fe6, seq_no=98, attempt_no=1}) took 98 ms (client timeout 49 ms). Trace:
I20260807 22:43:02.866155 6667 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters: 753b63233367448f8f28e3a1719be1af
I20260807 22:43:02.866716 7232 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 11 FOLLOWER]: Leader election won for term 11
W20260807 22:43:02.866321 6906 rpcz_store.cc:269] 0807 22:43:02.767144 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:02.767288 (+ 144us) service_pool.cc:227] Handling call
0807 22:43:02.865271 (+ 97983us) spinlock_profiling.cc:243] Waited 97.3 ms on lock 0x6150000a30a4. stack: 00007ff700a6c66e 00007ff700a6c430 00007ff6ffaf35ed 00000000008ea064 00000000008ea02d 00007ff70f18527d 00007ff70f185373 00007ff70bfa7361 00007ff70f167c01 00007ff70fb1bf28 00007ff70a14f446 00007ff70a14f175 00007ff7075a89be 00007ff7075a779e 00007ff7075adb25 00007ff7075ae54f
0807 22:43:02.866004 (+ 733us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":223797632}
I20260807 22:43:02.867435 7232 raft_consensus.cc:686] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 11 LEADER]: Becoming Leader. State: Replica: 672a15bd3cdb41c29eb4863f63479a24, State: Running, Role: LEADER
I20260807 22:43:02.867846 7232 consensus_queue.cc:237] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:02.869695 7237 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 11 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:02.870014 7237 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:02.870810 7241 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:43:02.871007 7241 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:43:02.871553 6341 catalog_manager.cc:5495] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 reported cstate change: term changed from 9 to 11, leader changed from 753b63233367448f8f28e3a1719be1af (127.4.103.1) to 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2). New cstate: current_term: 11 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
I20260807 22:43:02.924342 6924 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 11 FOLLOWER]: Refusing update from remote peer 672a15bd3cdb41c29eb4863f63479a24: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260807 22:43:02.925243 7237 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260807 22:43:02.926070 6662 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 -> Peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Couldn't send request to peer 753b63233367448f8f28e3a1719be1af. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.1:43211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 22:43:02.927528 7230 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 11 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:02.927874 7230 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:02.929616 7232 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 11 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:02.930078 7232 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:02.932924 7230 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:02.933281 7230 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:02.936165 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.1:43211
--local_ip_for_outbound_sockets=127.4.103.1
--tserver_master_addrs=127.4.103.62:36831
--webserver_port=42603
--webserver_interface=127.4.103.1
--builtin_ntp_servers=127.4.103.20:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
W20260807 22:43:03.145098 6662 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 -> Peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Couldn't send request to peer 753b63233367448f8f28e3a1719be1af. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.1:43211: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260807 22:43:03.386286 7256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:43:03.386718 7256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:43:03.386893 7256 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:43:03.387012 7256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:43:03.395785 7256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:43:03.396234 7256 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:43:03.396402 7256 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:43:03.396494 7256 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:43:03.396657 7256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.1
W20260807 22:43:03.403466 6662 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 -> Peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Couldn't send request to peer 753b63233367448f8f28e3a1719be1af. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.1:43211: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260807 22:43:03.412802 7256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.1:43211
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.103.1
--webserver_port=42603
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:36831
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.1
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:43:03.415187 7256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:43:03.418246 7256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:43:03.438109 7262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:03.438201 7263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:03.438078 7267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:43:03.438516 7256 server_base.cc:1034] running on GCE node
I20260807 22:43:03.439558 7256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:43:03.441200 7256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:43:03.442633 7256 hybrid_clock.cc:648] HybridClock initialized: now 1786142583442516 us; error 63 us; skew 500 ppm
I20260807 22:43:03.443099 7256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:43:03.450351 7256 webserver.cc:460] Webserver started at http://127.4.103.1:42603/ using document root <none> and password file <none>
I20260807 22:43:03.451494 7256 fs_manager.cc:359] Metadata directory not provided
I20260807 22:43:03.451726 7256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:43:03.461041 7256 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.002s
I20260807 22:43:03.465480 7274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:03.467572 7256 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 22:43:03.467929 7256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
uuid: "753b63233367448f8f28e3a1719be1af"
format_stamp: "Formatted at 2026-08-07 22:42:50 on dist-test-slave-kvfs"
I20260807 22:43:03.469390 7256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:43:03.507444 7256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:43:03.508994 7256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:43:03.509406 7256 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260807 22:43:03.513319 6662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.103.1:43211: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20260807 22:43:03.515443 7286 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 22:43:03.526117 7256 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 22:43:03.526295 7256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.002s sys 0.000s
I20260807 22:43:03.526547 7256 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 22:43:03.532490 7256 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 22:43:03.532652 7256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.003s sys 0.003s
I20260807 22:43:03.533397 7286 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Bootstrap starting.
I20260807 22:43:03.573908 7286 log.cc:824] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Log is configured to *not* fsync() on all Append() calls
I20260807 22:43:03.591787 7286 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:43:03.593185 7286 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Bootstrap complete.
I20260807 22:43:03.595785 7286 ts_tablet_manager.cc:1397] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Time spent bootstrapping tablet: real 0.063s user 0.038s sys 0.020s
I20260807 22:43:03.613782 7286 raft_consensus.cc:348] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 9 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:03.618598 7286 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 9 FOLLOWER]: Becoming Follower/Learner. State: Replica: 753b63233367448f8f28e3a1719be1af, State: Initialized, Role: FOLLOWER
I20260807 22:43:03.619575 7286 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:03.621145 7433 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:03.621711 7433 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:03.621893 7286 ts_tablet_manager.cc:1428] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Time spent starting tablet: real 0.026s user 0.018s sys 0.003s
I20260807 22:43:03.628814 7256 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.1:43211
I20260807 22:43:03.628867 7450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.1:43211 every 8 connection(s)
I20260807 22:43:03.630771 7256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
I20260807 22:43:03.633813 7451 heartbeater.cc:347] Connected to a master server at 127.4.103.62:36831
I20260807 22:43:03.634127 7451 heartbeater.cc:464] Registering TS with master...
I20260807 22:43:03.635070 7451 heartbeater.cc:511] Master 127.4.103.62:36831 requested a full tablet report, sending...
I20260807 22:43:03.638219 6341 ts_manager.cc:194] Re-registered known tserver with Master: 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:43:03.639518 6341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.1:50989
I20260807 22:43:03.641355 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 7256
I20260807 22:43:03.686014 7383 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 9 FOLLOWER]: Advancing to term 11
I20260807 22:43:03.692430 7451 heartbeater.cc:503] Master 127.4.103.62:36831 was elected leader, sending a full tablet report...
I20260807 22:43:03.694700 7433 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:43:03.695030 7433 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:03.697773 7454 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 11 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:03.697804 7458 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 11 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:03.698499 7454 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:03.701077 7433 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:03.704767 7458 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:03.718111 6761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:03.718726 6924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:03.719537 7279 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 753b63233367448f8f28e3a1719be1af; no voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:43:03.720110 7458 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260807 22:43:03.724042 7272 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 11 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:03.724521 7272 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:04.655514 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 6637
I20260807 22:43:04.656814 6810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:04.692553 7472 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:43:04.692863 7472 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:04.693735 7472 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:43:04.701524 7383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:04.716197 7475 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:43:04.716512 7475 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:04.717406 7475 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:04.718320 6924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:04.718719 6924 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 753b63233367448f8f28e3a1719be1af in term 11.
I20260807 22:43:04.719403 7279 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:43:04.719769 7475 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260807 22:43:04.719964 7475 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 11 FOLLOWER]: Starting leader election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:43:04.720121 7475 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 11 FOLLOWER]: Advancing to term 12
I20260807 22:43:04.722627 7475 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:04.723596 7475 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 12 election: Requested vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:04.724315 6924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:43:04.724589 6924 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 11 FOLLOWER]: Advancing to term 12
I20260807 22:43:04.727105 6924 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 753b63233367448f8f28e3a1719be1af in term 12.
I20260807 22:43:04.727799 7279 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:43:04.728261 7475 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 12 FOLLOWER]: Leader election won for term 12
I20260807 22:43:04.728847 7475 raft_consensus.cc:686] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 12 LEADER]: Becoming Leader. State: Replica: 753b63233367448f8f28e3a1719be1af, State: Running, Role: LEADER
I20260807 22:43:04.729527 7475 consensus_queue.cc:237] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:04.731995 7477 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 12 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:04.732358 7477 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:04.733613 7479 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:43:04.733912 7479 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:43:04.734331 6341 catalog_manager.cc:5495] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af reported cstate change: term changed from 11 to 12, leader changed from 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2) to 753b63233367448f8f28e3a1719be1af (127.4.103.1). New cstate: current_term: 12 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
W20260807 22:43:04.761471 6841 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.103.2:38723 timed out after 0.067s (SENT) [suppressed 3 similar messages]
I20260807 22:43:04.792510 6924 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 12 FOLLOWER]: Refusing update from remote peer 753b63233367448f8f28e3a1719be1af: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260807 22:43:04.793648 7477 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260807 22:43:04.796890 7472 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 12 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:04.797348 7472 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260807 22:43:04.801090 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
W20260807 22:43:04.802367 7006 meta_cache.cc:302] tablet c5fd68d798cc4cc68c2db34d14074bd4: replica 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723) has failed: Timed out: CoordinateTransaction RPC to 127.4.103.2:38723 timed out after 0.049s (SENT)
W20260807 22:43:04.802928 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 115) took 51 ms (client timeout 50 ms). Trace:
W20260807 22:43:04.803058 6384 rpcz_store.cc:269] 0807 22:43:04.751052 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:04.751174 (+ 122us) service_pool.cc:227] Handling call
0807 22:43:04.802847 (+ 51673us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:04.803898 7275 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.103.2:38723 timed out after 0.080s (SENT)
I20260807 22:43:04.815161 7472 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 12 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:04.815582 7472 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:04.821538 7477 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 12 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:04.821943 7477 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:04.835618 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:04.836773 6760 raft_consensus.cc:3032] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 11 LEADER]: Stepping down as leader of term 11
I20260807 22:43:04.836757 6756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:43:04.836977 6760 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 672a15bd3cdb41c29eb4863f63479a24, State: Running, Role: LEADER
I20260807 22:43:04.835593 6759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:43:04.837815 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 6637
I20260807 22:43:04.837872 6761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:04.842909 6924 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 12 FOLLOWER]: Rejecting Update request from peer 672a15bd3cdb41c29eb4863f63479a24 for earlier term 11. Current term is 12. Ops: []
W20260807 22:43:04.843575 6841 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Timed out: RequestConsensusVote RPC to 127.4.103.2:38723 timed out after 0.067s (SENT)
I20260807 22:43:04.843736 6841 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af
I20260807 22:43:04.844154 7472 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260807 22:43:04.881007 7006 connection.cc:535] client connection to 127.4.103.2:38723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 22:43:04.881052 6844 connection.cc:535] server connection from 127.4.103.2:58119 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 22:43:04.881191 7275 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Network error: recv got EOF from 127.4.103.2:38723 (error 108)
W20260807 22:43:04.882201 6315 connection.cc:535] server connection from 127.4.103.2:55019 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 22:43:04.884491 7275 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af -> Peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Couldn't send request to peer 672a15bd3cdb41c29eb4863f63479a24. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.2:38723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 22:43:04.889276 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.2:38723
--local_ip_for_outbound_sockets=127.4.103.2
--tserver_master_addrs=127.4.103.62:36831
--webserver_port=33969
--webserver_interface=127.4.103.2
--builtin_ntp_servers=127.4.103.20:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
W20260807 22:43:05.142134 7275 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af -> Peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Couldn't send request to peer 672a15bd3cdb41c29eb4863f63479a24. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.2:38723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260807 22:43:05.350597 7496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:43:05.351007 7496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:43:05.351107 7496 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:43:05.351182 7496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:43:05.359786 7496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:43:05.360121 7496 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:43:05.360214 7496 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:43:05.360281 7496 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:43:05.360406 7496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.2
I20260807 22:43:05.374224 7496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.2:38723
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.103.2
--webserver_port=33969
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:36831
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.2
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:43:05.376557 7496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:43:05.379096 7496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:43:05.397787 7275 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af -> Peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Couldn't send request to peer 672a15bd3cdb41c29eb4863f63479a24. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.2:38723: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260807 22:43:05.398497 7505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:05.399442 7504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:05.400357 7507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:43:05.400755 7496 server_base.cc:1034] running on GCE node
I20260807 22:43:05.402298 7496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:43:05.403831 7496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:43:05.405401 7496 hybrid_clock.cc:648] HybridClock initialized: now 1786142585405292 us; error 81 us; skew 500 ppm
I20260807 22:43:05.405821 7496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:43:05.409152 7496 webserver.cc:460] Webserver started at http://127.4.103.2:33969/ using document root <none> and password file <none>
I20260807 22:43:05.410221 7496 fs_manager.cc:359] Metadata directory not provided
I20260807 22:43:05.410423 7496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:43:05.420610 7496 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.000s
I20260807 22:43:05.434616 7515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:05.438797 7496 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.005s sys 0.000s
I20260807 22:43:05.439183 7496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
uuid: "672a15bd3cdb41c29eb4863f63479a24"
format_stamp: "Formatted at 2026-08-07 22:42:50 on dist-test-slave-kvfs"
I20260807 22:43:05.441128 7496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:43:05.479744 7496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:43:05.481293 7496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:43:05.481714 7496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:43:05.487279 7533 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 22:43:05.498389 7496 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 22:43:05.498621 7496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.002s sys 0.000s
I20260807 22:43:05.498804 7496 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 22:43:05.506624 7533 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Bootstrap starting.
I20260807 22:43:05.508695 7496 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 22:43:05.508986 7496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s user 0.007s sys 0.000s
I20260807 22:43:05.539028 7533 log.cc:824] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Log is configured to *not* fsync() on all Append() calls
I20260807 22:43:05.562721 7533 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:43:05.564123 7533 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Bootstrap complete.
I20260807 22:43:05.566272 7533 ts_tablet_manager.cc:1397] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Time spent bootstrapping tablet: real 0.060s user 0.028s sys 0.023s
I20260807 22:43:05.579690 7533 raft_consensus.cc:348] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 11 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:05.581431 7533 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 11 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672a15bd3cdb41c29eb4863f63479a24, State: Initialized, Role: FOLLOWER
I20260807 22:43:05.582733 7533 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:05.584174 7638 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 11 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:05.584744 7638 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:05.585323 7533 ts_tablet_manager.cc:1428] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Time spent starting tablet: real 0.019s user 0.017s sys 0.000s
I20260807 22:43:05.603713 7496 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.2:38723
I20260807 22:43:05.603760 7700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.2:38723 every 8 connection(s)
I20260807 22:43:05.605609 7496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
I20260807 22:43:05.609112 7701 heartbeater.cc:347] Connected to a master server at 127.4.103.62:36831
I20260807 22:43:05.609509 7701 heartbeater.cc:464] Registering TS with master...
I20260807 22:43:05.610486 7701 heartbeater.cc:511] Master 127.4.103.62:36831 requested a full tablet report, sending...
I20260807 22:43:05.613188 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 7496
I20260807 22:43:05.613631 6341 ts_manager.cc:194] Re-registered known tserver with Master: 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723)
I20260807 22:43:05.615442 6341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.2:33723
I20260807 22:43:05.634210 7638 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:43:05.634773 7638 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:05.637562 7638 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:05.651796 7637 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 11 FOLLOWER]: Advancing to term 12
I20260807 22:43:05.661921 7701 heartbeater.cc:503] Master 127.4.103.62:36831 was elected leader, sending a full tablet report...
I20260807 22:43:05.665412 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:05.666061 7383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:05.667351 7517 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24; no voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:43:05.668627 7710 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 12 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:05.669385 7710 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:05.669701 7710 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260807 22:43:05.669813 7638 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 12 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:05.670177 7638 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:05.681123 7475 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 12 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:05.681658 7475 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:06.615765 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 7256
I20260807 22:43:06.617170 7445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:06.633999 7720 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:43:06.634289 7720 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:06.635279 7720 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:06.636137 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:06.676898 7721 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:43:06.677238 7721 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:06.678242 7721 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:43:06.687983 7637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:06.688668 7637 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in term 12.
I20260807 22:43:06.689417 6841 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:43:06.689855 7721 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260807 22:43:06.690017 7721 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:43:06.690145 7721 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 12 FOLLOWER]: Advancing to term 13
I20260807 22:43:06.692741 7721 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:06.693687 7721 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 13 election: Requested vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:43:06.694273 7637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:43:06.694682 7637 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 12 FOLLOWER]: Advancing to term 13
I20260807 22:43:06.697166 7637 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in term 13.
I20260807 22:43:06.697656 6841 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:43:06.698009 7721 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 13 FOLLOWER]: Leader election won for term 13
I20260807 22:43:06.698549 7721 raft_consensus.cc:686] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 13 LEADER]: Becoming Leader. State: Replica: d6aee0f79fef46c5a7ba0d9714dc0a5e, State: Running, Role: LEADER
I20260807 22:43:06.699074 7721 consensus_queue.cc:237] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:06.700898 7724 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: New leader d6aee0f79fef46c5a7ba0d9714dc0a5e. Latest consensus state: current_term: 13 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:06.701475 7724 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: LEADER
W20260807 22:43:06.701834 7517 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.066s (SENT)
I20260807 22:43:06.702230 7726 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:43:06.702404 7726 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:43:06.704305 6341 catalog_manager.cc:5495] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e reported cstate change: term changed from 12 to 13, leader changed from 753b63233367448f8f28e3a1719be1af (127.4.103.1) to d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3). New cstate: current_term: 13 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } health_report { overall_health: HEALTHY } } }
W20260807 22:43:06.735739 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
W20260807 22:43:06.736562 7006 meta_cache.cc:302] tablet c5fd68d798cc4cc68c2db34d14074bd4: replica 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211) has failed: Timed out: CoordinateTransaction RPC to 127.4.103.1:43211 timed out after 0.049s (SENT)
W20260807 22:43:06.737190 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 133) took 51 ms (client timeout 50 ms). Trace:
W20260807 22:43:06.737349 6384 rpcz_store.cc:269] 0807 22:43:06.685279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:06.685379 (+ 100us) service_pool.cc:227] Handling call
0807 22:43:06.737090 (+ 51711us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:06.746723 7637 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 13 FOLLOWER]: Refusing update from remote peer d6aee0f79fef46c5a7ba0d9714dc0a5e: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260807 22:43:06.747987 7724 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260807 22:43:06.750900 7720 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader d6aee0f79fef46c5a7ba0d9714dc0a5e. Latest consensus state: current_term: 13 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:06.751274 7720 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:06.754293 7721 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 13 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:06.754675 7721 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:06.756239 7720 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 13 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:06.756595 7720 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:06.764683 7383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
W20260807 22:43:06.770498 7517 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Timed out: RequestConsensusVote RPC to 127.4.103.1:43211 timed out after 0.066s (SENT)
I20260807 22:43:06.770795 7517 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24; no voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:43:06.771520 7720 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260807 22:43:06.772567 7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:43:06.773684 7383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:43:06.774407 7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:06.775307 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:06.775924 7382 raft_consensus.cc:3032] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 12 LEADER]: Stepping down as leader of term 12
I20260807 22:43:06.776090 7382 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 753b63233367448f8f28e3a1719be1af, State: Running, Role: LEADER
I20260807 22:43:06.776461 7382 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:06.776806 7382 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 12 FOLLOWER]: Advancing to term 13
I20260807 22:43:06.778586 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:06.778872 7637 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 13 FOLLOWER]: Rejecting Update request from peer 753b63233367448f8f28e3a1719be1af for earlier term 12. Current term is 13. Ops: []
I20260807 22:43:06.783089 7382 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 13 FOLLOWER]: Refusing update from remote peer d6aee0f79fef46c5a7ba0d9714dc0a5e: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260807 22:43:06.783918 7721 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260807 22:43:06.787153 6925 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 13 LEADER]: Rejecting Update request from peer 753b63233367448f8f28e3a1719be1af for earlier term 12. Current term is 13. Ops: []
I20260807 22:43:06.787576 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 7256
W20260807 22:43:06.827004 6841 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e -> Peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Couldn't send request to peer 753b63233367448f8f28e3a1719be1af. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.1:43211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 22:43:06.846001 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.1:43211
--local_ip_for_outbound_sockets=127.4.103.1
--tserver_master_addrs=127.4.103.62:36831
--webserver_port=42603
--webserver_interface=127.4.103.1
--builtin_ntp_servers=127.4.103.20:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
W20260807 22:43:07.021999 6841 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e -> Peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Couldn't send request to peer 753b63233367448f8f28e3a1719be1af. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.1:43211: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260807 22:43:07.297304 6841 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e -> Peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Couldn't send request to peer 753b63233367448f8f28e3a1719be1af. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.1:43211: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260807 22:43:07.310544 7743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:43:07.311013 7743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:43:07.311183 7743 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:43:07.311316 7743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:43:07.320747 7743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:43:07.320967 7743 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:43:07.321080 7743 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:43:07.321164 7743 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:43:07.321341 7743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.1
I20260807 22:43:07.333189 7743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.1:43211
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.103.1
--webserver_port=42603
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:36831
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.1
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:43:07.335136 7743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:43:07.337666 7743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:43:07.355445 7752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:07.355392 7755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:07.355469 7750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:43:07.356268 7743 server_base.cc:1034] running on GCE node
I20260807 22:43:07.357102 7743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:43:07.358373 7743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:43:07.359709 7743 hybrid_clock.cc:648] HybridClock initialized: now 1786142587359622 us; error 41 us; skew 500 ppm
I20260807 22:43:07.360175 7743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:43:07.363663 7743 webserver.cc:460] Webserver started at http://127.4.103.1:42603/ using document root <none> and password file <none>
I20260807 22:43:07.364663 7743 fs_manager.cc:359] Metadata directory not provided
I20260807 22:43:07.364835 7743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:43:07.371978 7743 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 22:43:07.376010 7762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:07.377915 7743 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260807 22:43:07.378181 7743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
uuid: "753b63233367448f8f28e3a1719be1af"
format_stamp: "Formatted at 2026-08-07 22:42:50 on dist-test-slave-kvfs"
I20260807 22:43:07.379535 7743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:43:07.402776 7743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:43:07.403925 7743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:43:07.404318 7743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:43:07.408828 7775 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 22:43:07.417337 7743 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 22:43:07.417516 7743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20260807 22:43:07.417703 7743 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 22:43:07.424142 7743 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 22:43:07.424327 7743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.007s sys 0.000s
I20260807 22:43:07.424934 7775 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Bootstrap starting.
I20260807 22:43:07.459867 7775 log.cc:824] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Log is configured to *not* fsync() on all Append() calls
I20260807 22:43:07.478636 7775 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:43:07.480238 7775 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Bootstrap complete.
I20260807 22:43:07.482654 7775 ts_tablet_manager.cc:1397] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Time spent bootstrapping tablet: real 0.058s user 0.037s sys 0.015s
I20260807 22:43:07.501259 7743 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.1:43211
I20260807 22:43:07.501327 7932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.1:43211 every 8 connection(s)
I20260807 22:43:07.503481 7743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
I20260807 22:43:07.502358 7775 raft_consensus.cc:348] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 13 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:07.504151 7775 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 13 FOLLOWER]: Becoming Follower/Learner. State: Replica: 753b63233367448f8f28e3a1719be1af, State: Initialized, Role: FOLLOWER
I20260807 22:43:07.505384 7775 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:07.507580 7935 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:07.508453 7935 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:07.509690 7933 heartbeater.cc:347] Connected to a master server at 127.4.103.62:36831
I20260807 22:43:07.510162 7775 ts_tablet_manager.cc:1428] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af: Time spent starting tablet: real 0.027s user 0.019s sys 0.005s
I20260807 22:43:07.510166 7933 heartbeater.cc:464] Registering TS with master...
I20260807 22:43:07.511866 7933 heartbeater.cc:511] Master 127.4.103.62:36831 requested a full tablet report, sending...
I20260807 22:43:07.513098 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 7743
I20260807 22:43:07.515196 6341 ts_manager.cc:194] Re-registered known tserver with Master: 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:43:07.516657 6341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.1:36183
I20260807 22:43:07.520519 7933 heartbeater.cc:503] Master 127.4.103.62:36831 was elected leader, sending a full tablet report...
I20260807 22:43:07.566018 7942 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 13 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:07.566602 7942 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:07.567739 7935 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader d6aee0f79fef46c5a7ba0d9714dc0a5e. Latest consensus state: current_term: 13 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:07.568013 7935 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:07.605501 7721 tablet_replica.cc:439] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 13 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:07.605881 7721 tablet_replica.cc:442] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:08.537959 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 6820
I20260807 22:43:08.539269 6987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:08.560472 7954 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d6aee0f79fef46c5a7ba0d9714dc0a5e)
I20260807 22:43:08.560877 7954 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:08.563021 7954 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:08.583065 7637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:08.592656 7960 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader d6aee0f79fef46c5a7ba0d9714dc0a5e)
I20260807 22:43:08.592970 7960 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:08.593945 7960 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:08.602173 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:08.602705 7872 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 13.
I20260807 22:43:08.603519 7517 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af; no voters:
I20260807 22:43:08.603965 7960 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260807 22:43:08.604151 7960 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 13 FOLLOWER]: Starting leader election (detected failure of leader d6aee0f79fef46c5a7ba0d9714dc0a5e)
I20260807 22:43:08.604260 7960 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 13 FOLLOWER]: Advancing to term 14
I20260807 22:43:08.607139 7960 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:08.608126 7960 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 14 election: Requested vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:08.608736 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:43:08.609027 7872 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 13 FOLLOWER]: Advancing to term 14
I20260807 22:43:08.614602 7872 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 14.
I20260807 22:43:08.615439 7517 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af; no voters:
I20260807 22:43:08.616077 7960 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 14 FOLLOWER]: Leader election won for term 14
I20260807 22:43:08.616704 7960 raft_consensus.cc:686] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 14 LEADER]: Becoming Leader. State: Replica: 672a15bd3cdb41c29eb4863f63479a24, State: Running, Role: LEADER
I20260807 22:43:08.617349 7960 consensus_queue.cc:237] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:08.619566 7963 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 14 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:08.620086 7963 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:08.620826 7965 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:43:08.621097 7965 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:43:08.622228 6341 catalog_manager.cc:5495] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 reported cstate change: term changed from 13 to 14, leader changed from d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3) to 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2). New cstate: current_term: 14 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
W20260807 22:43:08.638772 7770 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.4.103.3:42621 for RPC RequestConsensusVote timed out after 0.075s (ON_OUTBOUND_QUEUE)
W20260807 22:43:08.654690 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
W20260807 22:43:08.654660 7010 meta_cache.cc:302] tablet c5fd68d798cc4cc68c2db34d14074bd4: replica d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621) has failed: Timed out: CoordinateTransaction RPC to 127.4.103.3:42621 timed out after 0.049s (SENT)
W20260807 22:43:08.655362 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 151) took 51 ms (client timeout 50 ms). Trace:
W20260807 22:43:08.655515 6384 rpcz_store.cc:269] 0807 22:43:08.604267 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:08.604375 (+ 108us) service_pool.cc:227] Handling call
0807 22:43:08.655257 (+ 50882us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:08.662923 7872 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 14 FOLLOWER]: Refusing update from remote peer 672a15bd3cdb41c29eb4863f63479a24: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260807 22:43:08.664062 7963 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260807 22:43:08.666909 7954 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 14 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:08.667325 7954 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:08.671408 7960 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 14 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:08.671802 7960 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:08.672483 7954 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 14 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:08.672809 7954 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
W20260807 22:43:08.716485 7770 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621): Timed out: connection negotiation to 127.4.103.3:42621 for RPC RequestConsensusVote timed out after 0.075s (ON_OUTBOUND_QUEUE)
I20260807 22:43:08.716862 7770 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 753b63233367448f8f28e3a1719be1af; no voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:43:08.717586 7954 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260807 22:43:08.722926 6924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:43:08.723832 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:08.726020 6924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:08.726476 6928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:43:08.726604 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 6820
I20260807 22:43:08.733357 7872 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 14 FOLLOWER]: Rejecting Update request from peer d6aee0f79fef46c5a7ba0d9714dc0a5e for earlier term 13. Current term is 14. Ops: []
I20260807 22:43:08.738749 7637 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 14 LEADER]: Rejecting Update request from peer d6aee0f79fef46c5a7ba0d9714dc0a5e for earlier term 13. Current term is 14. Ops: []
W20260807 22:43:08.766824 7523 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621): Timed out: RequestConsensusVote RPC to 127.4.103.3:42621 timed out after 0.085s (SENT)
W20260807 22:43:08.779801 7767 connection.cc:535] server connection from 127.4.103.3:60267 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 22:43:08.780661 6311 connection.cc:535] server connection from 127.4.103.3:33471 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 22:43:08.781015 7523 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621): Network error: recv got EOF from 127.4.103.3:42621 (error 108)
W20260807 22:43:08.781859 7956 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:43:08.563463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.103.3:42621 (local address 127.4.103.1:43763)
0807 22:43:08.575543 (+ 12080us) negotiation.cc:107] Waiting for socket to connect
0807 22:43:08.575571 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0807 22:43:08.575803 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:43:08.780353 (+204550us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.103.3:42621: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":11914,"thread_start_us":12351,"threads_started":1}
W20260807 22:43:08.781872 7523 connection.cc:535] server connection from 127.4.103.3:57019 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 22:43:08.783522 7523 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 -> Peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621): Couldn't send request to peer d6aee0f79fef46c5a7ba0d9714dc0a5e. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.3:42621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 22:43:08.791081 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.3:42621
--local_ip_for_outbound_sockets=127.4.103.3
--tserver_master_addrs=127.4.103.62:36831
--webserver_port=33269
--webserver_interface=127.4.103.3
--builtin_ntp_servers=127.4.103.20:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
W20260807 22:43:09.032399 7523 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 -> Peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621): Couldn't send request to peer d6aee0f79fef46c5a7ba0d9714dc0a5e. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.3:42621: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260807 22:43:09.046012 7954 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:43:09.046319 7954 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:09.047253 7954 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:09.047915 7637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
W20260807 22:43:09.049651 7770 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621): Network error: Client connection negotiation failed: client connection to 127.4.103.3:42621: connect: Connection refused (error 111)
I20260807 22:43:09.049841 7770 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 753b63233367448f8f28e3a1719be1af; no voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:43:09.050295 7954 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260807 22:43:09.266958 7523 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 -> Peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621): Couldn't send request to peer d6aee0f79fef46c5a7ba0d9714dc0a5e. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.3:42621: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260807 22:43:09.266997 7984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:43:09.267517 7984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:43:09.267700 7984 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:43:09.267825 7984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:43:09.277179 7984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:43:09.277436 7984 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:43:09.277539 7984 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:43:09.277611 7984 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:43:09.277757 7984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.3
I20260807 22:43:09.288362 7984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.3:42621
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.103.3
--webserver_port=33269
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:36831
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.3
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:43:09.290216 7984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:43:09.292832 7984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:43:09.311053 7990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:09.311133 7991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:09.311071 7997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:43:09.317866 7984 server_base.cc:1034] running on GCE node
I20260807 22:43:09.318821 7984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:43:09.320513 7984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:43:09.321844 7984 hybrid_clock.cc:648] HybridClock initialized: now 1786142589321737 us; error 77 us; skew 500 ppm
I20260807 22:43:09.322292 7984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:43:09.325915 7984 webserver.cc:460] Webserver started at http://127.4.103.3:33269/ using document root <none> and password file <none>
I20260807 22:43:09.327235 7984 fs_manager.cc:359] Metadata directory not provided
I20260807 22:43:09.327436 7984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:43:09.334733 7984 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 22:43:09.338732 8004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:09.340564 7984 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:43:09.340792 7984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/wal
uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
format_stamp: "Formatted at 2026-08-07 22:42:51 on dist-test-slave-kvfs"
I20260807 22:43:09.342056 7984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:43:09.359807 7984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:43:09.361047 7984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:43:09.361424 7984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:43:09.369349 8013 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 22:43:09.379050 7984 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 22:43:09.379279 7984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.003s
I20260807 22:43:09.379467 7984 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 22:43:09.385825 7984 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 22:43:09.386063 7984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.005s sys 0.000s
I20260807 22:43:09.386456 8013 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: Bootstrap starting.
I20260807 22:43:09.418258 8013 log.cc:824] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: Log is configured to *not* fsync() on all Append() calls
I20260807 22:43:09.436224 8013 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:43:09.437597 8013 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: Bootstrap complete.
I20260807 22:43:09.440258 8013 ts_tablet_manager.cc:1397] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: Time spent bootstrapping tablet: real 0.054s user 0.043s sys 0.008s
I20260807 22:43:09.458491 7984 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.3:42621
I20260807 22:43:09.458523 8176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.3:42621 every 8 connection(s)
I20260807 22:43:09.458491 8013 raft_consensus.cc:348] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 13 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:09.460207 8013 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 13 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6aee0f79fef46c5a7ba0d9714dc0a5e, State: Initialized, Role: FOLLOWER
I20260807 22:43:09.460734 7984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
I20260807 22:43:09.461181 8013 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:09.462812 8179 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 13 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:09.463352 8179 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:09.464371 8178 heartbeater.cc:347] Connected to a master server at 127.4.103.62:36831
I20260807 22:43:09.464569 8013 ts_tablet_manager.cc:1428] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e: Time spent starting tablet: real 0.024s user 0.023s sys 0.000s
I20260807 22:43:09.464816 8178 heartbeater.cc:464] Registering TS with master...
I20260807 22:43:09.465842 8178 heartbeater.cc:511] Master 127.4.103.62:36831 requested a full tablet report, sending...
I20260807 22:43:09.468587 6341 ts_manager.cc:194] Re-registered known tserver with Master: d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:09.470216 6341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.3:47663
I20260807 22:43:09.471410 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 7984
I20260807 22:43:09.474936 8178 heartbeater.cc:503] Master 127.4.103.62:36831 was elected leader, sending a full tablet report...
I20260807 22:43:09.482117 8113 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 13 FOLLOWER]: Advancing to term 14
I20260807 22:43:09.491123 8188 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 14 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:09.491671 8188 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:09.492934 8179 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 14 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:09.493182 8179 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:09.521600 7963 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 14 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:09.522030 7963 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:10.476306 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 7496
I20260807 22:43:10.477777 7695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:10.485479 8194 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:43:10.485838 8194 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:10.487986 8194 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:43:10.497761 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:10.525002 8201 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:43:10.525269 8201 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:10.526252 8201 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:10.543394 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:10.543964 8113 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 753b63233367448f8f28e3a1719be1af in term 14.
I20260807 22:43:10.544620 7770 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:43:10.545025 8201 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260807 22:43:10.545207 8201 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 14 FOLLOWER]: Starting leader election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:43:10.545341 8201 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 14 FOLLOWER]: Advancing to term 15
I20260807 22:43:10.548178 8201 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:10.549024 8201 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 15 election: Requested vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:10.549665 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:43:10.549984 8113 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 14 FOLLOWER]: Advancing to term 15
I20260807 22:43:10.553416 8113 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 753b63233367448f8f28e3a1719be1af in term 15.
I20260807 22:43:10.554162 7770 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:43:10.554603 8201 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 15 FOLLOWER]: Leader election won for term 15
I20260807 22:43:10.555186 8201 raft_consensus.cc:686] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 15 LEADER]: Becoming Leader. State: Replica: 753b63233367448f8f28e3a1719be1af, State: Running, Role: LEADER
I20260807 22:43:10.555861 8201 consensus_queue.cc:237] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:10.558854 8205 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 15 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:10.559252 8205 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:10.560534 8212 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:43:10.560820 8212 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:43:10.560937 6341 catalog_manager.cc:5495] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af reported cstate change: term changed from 14 to 15, leader changed from 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2) to 753b63233367448f8f28e3a1719be1af (127.4.103.1). New cstate: current_term: 15 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
W20260807 22:43:10.563916 8005 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.4.103.2:38723 for RPC RequestConsensusVote timed out after 0.075s (ON_OUTBOUND_QUEUE)
W20260807 22:43:10.589630 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
W20260807 22:43:10.589870 7006 meta_cache.cc:302] tablet c5fd68d798cc4cc68c2db34d14074bd4: replica 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723) has failed: Timed out: CoordinateTransaction RPC to 127.4.103.2:38723 timed out after 0.049s (SENT)
W20260807 22:43:10.590426 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 169) took 51 ms (client timeout 50 ms). Trace:
W20260807 22:43:10.590566 6384 rpcz_store.cc:269] 0807 22:43:10.539284 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:10.539402 (+ 118us) service_pool.cc:227] Handling call
0807 22:43:10.590327 (+ 50925us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:10.604272 8113 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 15 FOLLOWER]: Refusing update from remote peer 753b63233367448f8f28e3a1719be1af: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260807 22:43:10.605724 8205 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260807 22:43:10.608538 8194 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 15 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:10.608950 8194 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:10.625861 8194 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 15 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:10.626307 8194 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:10.626473 8201 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 15 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:10.626890 8201 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
W20260807 22:43:10.650190 8005 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Timed out: connection negotiation to 127.4.103.2:38723 for RPC RequestConsensusVote timed out after 0.075s (ON_OUTBOUND_QUEUE)
I20260807 22:43:10.650581 8005 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af
I20260807 22:43:10.651201 8194 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260807 22:43:10.660388 7637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:10.661629 7633 raft_consensus.cc:3032] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 14 LEADER]: Stepping down as leader of term 14
I20260807 22:43:10.661674 8113 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 15 FOLLOWER]: Rejecting Update request from peer 672a15bd3cdb41c29eb4863f63479a24 for earlier term 14. Current term is 15. Ops: []
I20260807 22:43:10.661811 7633 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 672a15bd3cdb41c29eb4863f63479a24, State: Running, Role: LEADER
I20260807 22:43:10.662237 7630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:43:10.662570 7632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:10.663115 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 7496
I20260807 22:43:10.663089 7636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
W20260807 22:43:10.701015 8195 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:43:10.488316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.103.2:38723 (local address 127.4.103.3:59743)
0807 22:43:10.488958 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0807 22:43:10.488984 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 22:43:10.489187 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:43:10.699480 (+210293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.103.2:38723: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":306,"threads_started":1}
W20260807 22:43:10.702418 7764 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af -> Peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Couldn't send request to peer 672a15bd3cdb41c29eb4863f63479a24. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.2:38723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260807 22:43:10.702783 7006 connection.cc:535] client connection to 127.4.103.2:38723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 22:43:10.706657 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.2:38723
--local_ip_for_outbound_sockets=127.4.103.2
--tserver_master_addrs=127.4.103.62:36831
--webserver_port=33969
--webserver_interface=127.4.103.2
--builtin_ntp_servers=127.4.103.20:39311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
W20260807 22:43:10.914088 7764 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af -> Peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Couldn't send request to peer 672a15bd3cdb41c29eb4863f63479a24. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.2:38723: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260807 22:43:11.172822 8229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:43:11.173331 8229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:43:11.173440 8229 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:43:11.173501 8229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:43:11.182422 8229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:43:11.182629 8229 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:43:11.182703 8229 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:43:11.182749 8229 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:43:11.182849 8229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.2
I20260807 22:43:11.194388 8229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:39311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.2:38723
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.103.2
--webserver_port=33969
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:36831
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.2
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:43:11.196466 8229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:43:11.199352 8229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:43:11.201026 7764 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af -> Peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Couldn't send request to peer 672a15bd3cdb41c29eb4863f63479a24. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.2:38723: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260807 22:43:11.218963 8238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:11.218963 8243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:11.218993 8235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:43:11.220153 8229 server_base.cc:1034] running on GCE node
I20260807 22:43:11.220965 8229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:43:11.222177 8229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:43:11.223469 8229 hybrid_clock.cc:648] HybridClock initialized: now 1786142591223396 us; error 47 us; skew 500 ppm
I20260807 22:43:11.223857 8229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:43:11.227095 8229 webserver.cc:460] Webserver started at http://127.4.103.2:33969/ using document root <none> and password file <none>
I20260807 22:43:11.228214 8229 fs_manager.cc:359] Metadata directory not provided
I20260807 22:43:11.228421 8229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:43:11.236166 8229 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 22:43:11.239979 8255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:11.242202 8229 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:43:11.242455 8229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
uuid: "672a15bd3cdb41c29eb4863f63479a24"
format_stamp: "Formatted at 2026-08-07 22:42:50 on dist-test-slave-kvfs"
I20260807 22:43:11.244321 8229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:43:11.281155 8229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:43:11.282414 8229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:43:11.282789 8229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:43:11.287488 8269 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 22:43:11.295722 8229 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 22:43:11.295902 8229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20260807 22:43:11.296137 8229 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 22:43:11.301785 8229 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 22:43:11.301983 8229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.004s sys 0.000s
I20260807 22:43:11.302630 8269 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Bootstrap starting.
I20260807 22:43:11.333086 8269 log.cc:824] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Log is configured to *not* fsync() on all Append() calls
I20260807 22:43:11.349285 8269 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:43:11.350155 8269 tablet_bootstrap.cc:492] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Bootstrap complete.
I20260807 22:43:11.351691 8269 ts_tablet_manager.cc:1397] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Time spent bootstrapping tablet: real 0.050s user 0.023s sys 0.024s
I20260807 22:43:11.364725 8269 raft_consensus.cc:348] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 14 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:11.366582 8269 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 14 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672a15bd3cdb41c29eb4863f63479a24, State: Initialized, Role: FOLLOWER
I20260807 22:43:11.367565 8269 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:11.369508 8405 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 14 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:11.369992 8405 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:11.370563 8269 ts_tablet_manager.cc:1428] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24: Time spent starting tablet: real 0.019s user 0.006s sys 0.011s
I20260807 22:43:11.383394 8229 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.2:38723
I20260807 22:43:11.383450 8448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.2:38723 every 8 connection(s)
I20260807 22:43:11.385241 8229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
I20260807 22:43:11.388470 8449 heartbeater.cc:347] Connected to a master server at 127.4.103.62:36831
I20260807 22:43:11.388769 8449 heartbeater.cc:464] Registering TS with master...
I20260807 22:43:11.389614 8449 heartbeater.cc:511] Master 127.4.103.62:36831 requested a full tablet report, sending...
I20260807 22:43:11.392323 6341 ts_manager.cc:194] Re-registered known tserver with Master: 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723)
I20260807 22:43:11.392850 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 8229
I20260807 22:43:11.393800 6341 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.2:36947
I20260807 22:43:11.417405 8405 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:43:11.417798 8405 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:11.420182 8405 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:11.430073 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:11.430778 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:11.431702 8256 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24; no voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:43:11.432333 8405 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260807 22:43:11.439625 8374 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 14 FOLLOWER]: Advancing to term 15
I20260807 22:43:11.446026 8449 heartbeater.cc:503] Master 127.4.103.62:36831 was elected leader, sending a full tablet report...
I20260807 22:43:11.448702 8460 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 15 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:11.449209 8460 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:11.450106 8405 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 753b63233367448f8f28e3a1719be1af. Latest consensus state: current_term: 15 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:11.450444 8405 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:11.473270 8201 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 15 leader_uuid: "753b63233367448f8f28e3a1719be1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:11.473698 8201 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:12.136200 8465 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:43:12.136914 8465 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:12.138474 8465 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:43:12.141906 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:12.169749 8374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:12.171096 8374 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in term 15.
I20260807 22:43:12.176916 8469 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:43:12.177246 8005 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters: 753b63233367448f8f28e3a1719be1af
I20260807 22:43:12.177412 8469 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:12.178258 8465 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260807 22:43:12.178547 8465 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:43:12.178747 8465 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 15 FOLLOWER]: Advancing to term 16
I20260807 22:43:12.179155 8469 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:12.180387 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:12.181679 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:12.182785 8256 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24; no voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:43:12.183352 8469 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260807 22:43:12.184345 8465 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:12.185959 8465 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 16 election: Requested vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:43:12.186880 8374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:43:12.188410 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:43:12.188694 8113 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 16 FOLLOWER]: Rejecting Update request from peer 753b63233367448f8f28e3a1719be1af for earlier term 15. Current term is 16. Ops: []
I20260807 22:43:12.189551 8005 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af
I20260807 22:43:12.189719 8201 consensus_queue.cc:1046] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260807 22:43:12.190265 8465 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260807 22:43:12.190969 8471 raft_consensus.cc:3032] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 15 LEADER]: Stepping down as leader of term 15
I20260807 22:43:12.191150 8471 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 753b63233367448f8f28e3a1719be1af, State: Running, Role: LEADER
I20260807 22:43:12.191854 8471 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:12.193253 8471 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 15 FOLLOWER]: Advancing to term 16
I20260807 22:43:12.253978 8469 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:43:12.254007 8465 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:43:12.254464 8469 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:12.254621 8465 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:12.255271 8471 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:43:12.255720 8471 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:12.256263 8469 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:12.257105 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:12.257233 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:12.257531 7872 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 16.
I20260807 22:43:12.257440 8374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:12.257905 8113 raft_consensus.cc:2379] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 672a15bd3cdb41c29eb4863f63479a24 in current term 16: Already voted for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in this term.
I20260807 22:43:12.257872 8374 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in term 15.
I20260807 22:43:12.258651 8256 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af; no voters:
I20260807 22:43:12.259109 8465 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:43:12.259275 8005 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters:
I20260807 22:43:12.259646 8469 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260807 22:43:12.259918 8469 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 753b63233367448f8f28e3a1719be1af)
I20260807 22:43:12.260332 8469 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 15 FOLLOWER]: Advancing to term 16
I20260807 22:43:12.260751 8465 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260807 22:43:12.261015 8465 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:43:12.261085 8374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:12.261124 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:12.261436 8471 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:12.261264 8465 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 16 FOLLOWER]: Advancing to term 17
I20260807 22:43:12.262293 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:12.262764 7872 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in term 16.
I20260807 22:43:12.267686 8465 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:12.268965 8113 raft_consensus.cc:2379] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 753b63233367448f8f28e3a1719be1af in current term 17: Already voted for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in this term.
I20260807 22:43:12.270020 8465 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 17 election: Requested vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:43:12.272274 8373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24"
I20260807 22:43:12.272490 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:43:12.273190 7872 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 16 FOLLOWER]: Advancing to term 17
I20260807 22:43:12.273705 8469 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:12.274993 8374 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 753b63233367448f8f28e3a1719be1af in term 16.
I20260807 22:43:12.276580 7764 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af; no voters: d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:43:12.280375 7871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
I20260807 22:43:12.280607 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:43:12.281242 8113 raft_consensus.cc:2354] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 672a15bd3cdb41c29eb4863f63479a24 for earlier term 16. Current term is 17.
I20260807 22:43:12.281539 7872 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6aee0f79fef46c5a7ba0d9714dc0a5e in term 17.
I20260807 22:43:12.281935 8471 raft_consensus.cc:2741] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260807 22:43:12.282363 8259 leader_election.cc:398] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 16 election: Vote denied by peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621) with higher term. Message: Invalid argument: T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 672a15bd3cdb41c29eb4863f63479a24 for earlier term 16. Current term is 17.
I20260807 22:43:12.282534 8259 leader_election.cc:401] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260807 22:43:12.283790 8005 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters: 672a15bd3cdb41c29eb4863f63479a24
I20260807 22:43:12.284478 8472 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 16 FOLLOWER]: Advancing to term 17
I20260807 22:43:12.285007 8465 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 17 FOLLOWER]: Leader election won for term 17
I20260807 22:43:12.286273 8465 raft_consensus.cc:686] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 17 LEADER]: Becoming Leader. State: Replica: d6aee0f79fef46c5a7ba0d9714dc0a5e, State: Running, Role: LEADER
I20260807 22:43:12.287254 8465 consensus_queue.cc:237] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:12.291324 8472 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621) with higher term. Message: Invalid argument: T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 672a15bd3cdb41c29eb4863f63479a24 for earlier term 16. Current term is 17.
I20260807 22:43:12.292366 8469 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 16 election: Requested vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:12.299019 8473 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader d6aee0f79fef46c5a7ba0d9714dc0a5e. Latest consensus state: current_term: 17 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:12.299643 8473 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:12.299081 6341 catalog_manager.cc:5495] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e reported cstate change: term changed from 15 to 17, leader changed from 753b63233367448f8f28e3a1719be1af (127.4.103.1) to d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3). New cstate: current_term: 17 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } health_report { overall_health: HEALTHY } } }
I20260807 22:43:12.302379 8476 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:43:12.302927 8476 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260807 22:43:12.325515 7086 txn_status_manager-itest.cc:376] KeepTransactionAlive() returned non-OK status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
I20260807 22:43:12.337988 8374 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 17 FOLLOWER]: Refusing update from remote peer d6aee0f79fef46c5a7ba0d9714dc0a5e: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260807 22:43:12.340888 8465 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260807 22:43:12.343984 7872 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 17 FOLLOWER]: Refusing update from remote peer d6aee0f79fef46c5a7ba0d9714dc0a5e: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260807 22:43:12.345604 8473 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260807 22:43:12.352577 6384 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:60582 (request call id 185) took 77 ms (client timeout 50 ms). Trace:
W20260807 22:43:12.353044 6384 rpcz_store.cc:269] 0807 22:43:12.275085 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:12.276172 (+ 1087us) service_pool.cc:227] Handling call
0807 22:43:12.352381 (+ 76209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:12.357738 8469 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader d6aee0f79fef46c5a7ba0d9714dc0a5e. Latest consensus state: current_term: 17 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:12.358579 8469 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:12.363368 8471 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader d6aee0f79fef46c5a7ba0d9714dc0a5e. Latest consensus state: current_term: 17 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:12.364401 8471 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:12.378715 8473 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 17 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:12.379637 8473 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:12.383994 8473 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 17 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:12.384717 8473 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/txn_status_manager-itest.cc:449: Failure
Failed
Bad status: Timed out: KeepTransactionAlive RPC to 127.4.103.62:36831 timed out after 0.050s (SENT)
I20260807 22:43:12.406697 4508 external_mini_cluster-itest-base.cc:80] Found fatal failure
I20260807 22:43:12.407075 4508 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID 753b63233367448f8f28e3a1719be1af and pid 7743
I20260807 22:43:12.413906 8471 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 17 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:12.414593 8471 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:12.414701 8469 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 17 leader_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:12.415299 8469 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:12.426815 8469 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d6aee0f79fef46c5a7ba0d9714dc0a5e)
I20260807 22:43:12.427138 8469 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:12.428601 8469 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:12.430943 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:12.432852 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:12.434126 8256 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24; no voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:43:12.434958 8469 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260807 22:43:12.937716 8011 txn_status_manager.cc:1390] automatically aborted stale txn (ID 0) past 0.552s from last keepalive heartbeat (effective timeout is 0.500s)
************************ BEGIN STACKS **************************
[New LWP 7744]
[New LWP 7746]
[New LWP 7747]
[New LWP 7748]
[New LWP 7757]
[New LWP 7758]
[New LWP 7759]
[New LWP 7764]
[New LWP 7767]
[New LWP 7769]
[New LWP 7770]
[New LWP 7771]
[New LWP 7773]
[New LWP 7779]
[New LWP 7780]
[New LWP 7781]
[New LWP 7782]
[New LWP 7783]
[New LWP 7784]
[New LWP 7785]
[New LWP 7786]
[New LWP 7787]
[New LWP 7788]
[New LWP 7789]
[New LWP 7791]
[New LWP 7793]
[New LWP 7795]
[New LWP 7796]
[New LWP 7800]
[New LWP 7801]
[New LWP 7802]
[New LWP 7803]
[New LWP 7805]
[New LWP 7806]
[New LWP 7809]
[New LWP 7811]
[New LWP 7813]
[New LWP 7818]
[New LWP 7819]
[New LWP 7820]
[New LWP 7821]
[New LWP 7822]
[New LWP 7823]
[New LWP 7824]
[New LWP 7826]
[New LWP 7827]
[New LWP 7828]
[New LWP 7829]
[New LWP 7830]
[New LWP 7833]
[New LWP 7834]
[New LWP 7835]
[New LWP 7837]
[New LWP 7838]
[New LWP 7840]
[New LWP 7841]
[New LWP 7843]
[New LWP 7846]
[New LWP 7847]
[New LWP 7848]
[New LWP 7849]
[New LWP 7850]
[New LWP 7851]
[New LWP 7853]
[New LWP 7855]
[New LWP 7856]
[New LWP 7857]
[New LWP 7858]
[New LWP 7859]
[New LWP 7860]
[New LWP 7861]
[New LWP 7862]
[New LWP 7863]
[New LWP 7864]
[New LWP 7865]
[New LWP 7867]
[New LWP 7868]
[New LWP 7871]
[New LWP 7872]
[New LWP 7873]
[New LWP 7874]
[New LWP 7875]
[New LWP 7876]
[New LWP 7877]
[New LWP 7879]
[New LWP 7882]
[New LWP 7883]
[New LWP 7884]
[New LWP 7885]
[New LWP 7886]
[New LWP 7887]
[New LWP 7888]
[New LWP 7891]
[New LWP 7892]
[New LWP 7893]
[New LWP 7894]
[New LWP 7896]
[New LWP 7898]
[New LWP 7899]
[New LWP 7901]
[New LWP 7902]
[New LWP 7903]
[New LWP 7905]
[New LWP 7907]
[New LWP 7908]
[New LWP 7909]
[New LWP 7910]
[New LWP 7911]
[New LWP 7913]
[New LWP 7915]
[New LWP 7916]
[New LWP 7917]
[New LWP 7918]
[New LWP 7919]
[New LWP 7921]
[New LWP 7922]
[New LWP 7923]
[New LWP 7926]
[New LWP 7927]
[New LWP 7928]
[New LWP 7929]
[New LWP 7931]
[New LWP 7932]
[New LWP 7933]
[New LWP 7934]
[New LWP 8486]
warning: Error reading shared library list entry at 0x4
warning: Corrupted shared library list: 0x0 != 0x74696172745f7261
warning: Error reading shared library list entry at 0x4
warning: Corrupted shared library list: 0x0 != 0x74696172745f7261
0x00007f1617393d50 in ?? ()
Id Target Id Frame
* 1 LWP 7743 "kudu" 0x00007f1617393d50 in ?? ()
2 LWP 7744 "kudu" 0x00007f161738ffb9 in ?? ()
3 LWP 7746 "kudu" 0x00007f161738ffb9 in ?? ()
4 LWP 7747 "kudu" 0x00007f161738ffb9 in ?? ()
5 LWP 7748 "kernel-watcher-" 0x00007f161738ffb9 in ?? ()
6 LWP 7757 "ntp client-7757" 0x00007f16173939e2 in ?? ()
7 LWP 7758 "file cache-evic" 0x00007f161738ffb9 in ?? ()
8 LWP 7759 "sq_acceptor" 0x00007f1610312bb9 in ?? ()
9 LWP 7764 "rpc reactor-776" 0x00007f161031f947 in ?? ()
10 LWP 7767 "rpc reactor-776" 0x00007f161031f947 in ?? ()
11 LWP 7769 "rpc reactor-776" 0x00007f161031f947 in ?? ()
12 LWP 7770 "rpc reactor-777" 0x00007f161031f947 in ?? ()
13 LWP 7771 "MaintenanceMgr " 0x00007f161738fad3 in ?? ()
14 LWP 7773 "txn-status-mana" 0x00007f161738ffb9 in ?? ()
15 LWP 7779 "collect_and_rem" 0x00007f161738ffb9 in ?? ()
16 LWP 7780 "tc-session-exp-" 0x00007f161738ffb9 in ?? ()
17 LWP 7781 "rpc reactor-778" 0x00007f161031f947 in ?? ()
18 LWP 7782 "rpc worker-7782" 0x00007f161738fad3 in ?? ()
19 LWP 7783 "rpc worker-7783" 0x00007f161738fad3 in ?? ()
20 LWP 7784 "rpc reactor-778" 0x00007f161031f947 in ?? ()
21 LWP 7785 "rpc reactor-778" 0x00007f161031f947 in ?? ()
22 LWP 7786 "rpc reactor-778" 0x00007f161031f947 in ?? ()
23 LWP 7787 "rpc worker-7787" 0x00007f161738fad3 in ?? ()
24 LWP 7788 "rpc worker-7788" 0x00007f161738fad3 in ?? ()
25 LWP 7789 "rpc worker-7789" 0x00007f161738fad3 in ?? ()
26 LWP 7791 "rpc worker-7791" 0x00007f161738fad3 in ?? ()
27 LWP 7793 "rpc worker-7793" 0x00007f161738fad3 in ?? ()
28 LWP 7795 "rpc worker-7795" 0x00007f161738fad3 in ?? ()
29 LWP 7796 "rpc worker-7796" 0x00007f161738fad3 in ?? ()
30 LWP 7800 "rpc worker-7800" 0x00007f161738fad3 in ?? ()
31 LWP 7801 "rpc worker-7801" 0x00007f161738fad3 in ?? ()
32 LWP 7802 "rpc worker-7802" 0x00007f161738fad3 in ?? ()
33 LWP 7803 "rpc worker-7803" 0x00007f161738fad3 in ?? ()
34 LWP 7805 "rpc worker-7805" 0x00007f161738fad3 in ?? ()
35 LWP 7806 "rpc worker-7806" 0x00007f161738fad3 in ?? ()
36 LWP 7809 "rpc worker-7809" 0x00007f161738fad3 in ?? ()
37 LWP 7811 "rpc worker-7811" 0x00007f161738fad3 in ?? ()
38 LWP 7813 "rpc worker-7813" 0x00007f161738fad3 in ?? ()
39 LWP 7818 "rpc worker-7818" 0x00007f161738fad3 in ?? ()
40 LWP 7819 "rpc worker-7819" 0x00007f161738fad3 in ?? ()
41 LWP 7820 "rpc worker-7820" 0x00007f161738fad3 in ?? ()
42 LWP 7821 "rpc worker-7821" 0x00007f161738fad3 in ?? ()
43 LWP 7822 "rpc worker-7822" 0x00007f161738fad3 in ?? ()
44 LWP 7823 "rpc worker-7823" 0x00007f161738fad3 in ?? ()
45 LWP 7824 "rpc worker-7824" 0x00007f161738fad3 in ?? ()
46 LWP 7826 "rpc worker-7826" 0x00007f161738fad3 in ?? ()
47 LWP 7827 "rpc worker-7827" 0x00007f161738fad3 in ?? ()
48 LWP 7828 "rpc worker-7828" 0x00007f161738fad3 in ?? ()
49 LWP 7829 "rpc worker-7829" 0x00007f161738fad3 in ?? ()
50 LWP 7830 "rpc worker-7830" 0x00007f161738fad3 in ?? ()
51 LWP 7833 "rpc worker-7833" 0x00007f161738fad3 in ?? ()
52 LWP 7834 "rpc worker-7834" 0x00007f161738fad3 in ?? ()
53 LWP 7835 "rpc worker-7835" 0x00007f161738fad3 in ?? ()
54 LWP 7837 "rpc worker-7837" 0x00007f161738fad3 in ?? ()
55 LWP 7838 "rpc worker-7838" 0x00007f161738fad3 in ?? ()
56 LWP 7840 "rpc worker-7840" 0x00007f161738fad3 in ?? ()
57 LWP 7841 "rpc worker-7841" 0x00007f161738fad3 in ?? ()
58 LWP 7843 "rpc worker-7843" 0x00007f161738fad3 in ?? ()
59 LWP 7846 "rpc worker-7846" 0x00007f161738fad3 in ?? ()
60 LWP 7847 "rpc worker-7847" 0x00007f161738fad3 in ?? ()
61 LWP 7848 "rpc worker-7848" 0x00007f161738fad3 in ?? ()
62 LWP 7849 "rpc worker-7849" 0x00007f161738fad3 in ?? ()
63 LWP 7850 "rpc worker-7850" 0x00007f161738fad3 in ?? ()
64 LWP 7851 "rpc worker-7851" 0x00007f161738fad3 in ?? ()
65 LWP 7853 "rpc worker-7853" 0x00007f161738fad3 in ?? ()
66 LWP 7855 "rpc worker-7855" 0x00007f161738fad3 in ?? ()
67 LWP 7856 "rpc worker-7856" 0x00007f161738fad3 in ?? ()
68 LWP 7857 "rpc worker-7857" 0x00007f161738fad3 in ?? ()
69 LWP 7858 "rpc worker-7858" 0x00007f161738fad3 in ?? ()
70 LWP 7859 "rpc worker-7859" 0x00007f161738fad3 in ?? ()
71 LWP 7860 "rpc worker-7860" 0x00007f161738fad3 in ?? ()
72 LWP 7861 "rpc worker-7861" 0x00007f161738fad3 in ?? ()
73 LWP 7862 "rpc worker-7862" 0x00007f161738fad3 in ?? ()
74 LWP 7863 "rpc worker-7863" 0x00007f161738fad3 in ?? ()
75 LWP 7864 "rpc worker-7864" 0x00007f161738fad3 in ?? ()
76 LWP 7865 "rpc worker-7865" 0x00007f161738fad3 in ?? ()
77 LWP 7867 "rpc worker-7867" 0x00007f161738fad3 in ?? ()
78 LWP 7868 "rpc worker-7868" 0x00007f161738fad3 in ?? ()
79 LWP 7871 "rpc worker-7871" 0x00007f161738fad3 in ?? ()
80 LWP 7872 "rpc worker-7872" 0x00007f161738fad3 in ?? ()
81 LWP 7873 "rpc worker-7873" 0x00007f161738fad3 in ?? ()
82 LWP 7874 "rpc worker-7874" 0x00007f161738fad3 in ?? ()
83 LWP 7875 "rpc worker-7875" 0x00007f161738fad3 in ?? ()
84 LWP 7876 "rpc worker-7876" 0x00007f161738fad3 in ?? ()
85 LWP 7877 "rpc worker-7877" 0x00007f161738fad3 in ?? ()
86 LWP 7879 "rpc worker-7879" 0x00007f161738fad3 in ?? ()
87 LWP 7882 "rpc worker-7882" 0x00007f161738fad3 in ?? ()
88 LWP 7883 "rpc worker-7883" 0x00007f161738fad3 in ?? ()
89 LWP 7884 "rpc worker-7884" 0x00007f161738fad3 in ?? ()
90 LWP 7885 "rpc worker-7885" 0x00007f161738fad3 in ?? ()
91 LWP 7886 "rpc worker-7886" 0x00007f161738fad3 in ?? ()
92 LWP 7887 "rpc worker-7887" 0x00007f161738fad3 in ?? ()
93 LWP 7888 "rpc worker-7888" 0x00007f161738fad3 in ?? ()
94 LWP 7891 "rpc worker-7891" 0x00007f161738fad3 in ?? ()
95 LWP 7892 "rpc worker-7892" 0x00007f161738fad3 in ?? ()
96 LWP 7893 "rpc worker-7893" 0x00007f161738fad3 in ?? ()
97 LWP 7894 "rpc worker-7894" 0x00007f161738fad3 in ?? ()
98 LWP 7896 "rpc worker-7896" 0x00007f161738fad3 in ?? ()
99 LWP 7898 "rpc worker-7898" 0x00007f161738fad3 in ?? ()
100 LWP 7899 "rpc worker-7899" 0x00007f161738fad3 in ?? ()
101 LWP 7901 "rpc worker-7901" 0x00007f161738fad3 in ?? ()
102 LWP 7902 "rpc worker-7902" 0x00007f161738fad3 in ?? ()
103 LWP 7903 "rpc worker-7903" 0x00007f161738fad3 in ?? ()
104 LWP 7905 "rpc worker-7905" 0x00007f161738fad3 in ?? ()
105 LWP 7907 "rpc worker-7907" 0x00007f161738fad3 in ?? ()
106 LWP 7908 "rpc worker-7908" 0x00007f161738fad3 in ?? ()
107 LWP 7909 "rpc worker-7909" 0x00007f161738fad3 in ?? ()
108 LWP 7910 "rpc worker-7910" 0x00007f161738fad3 in ?? ()
109 LWP 7911 "rpc worker-7911" 0x00007f161738fad3 in ?? ()
110 LWP 7913 "rpc worker-7913" 0x00007f161738fad3 in ?? ()
111 LWP 7915 "rpc worker-7915" 0x00007f161738fad3 in ?? ()
112 LWP 7916 "rpc worker-7916" 0x00007f161738fad3 in ?? ()
113 LWP 7917 "rpc worker-7917" 0x00007f161738fad3 in ?? ()
114 LWP 7918 "rpc worker-7918" 0x00007f161738fad3 in ?? ()
115 LWP 7919 "rpc worker-7919" 0x00007f161738fad3 in ?? ()
116 LWP 7921 "rpc worker-7921" 0x00007f161738fad3 in ?? ()
117 LWP 7922 "rpc worker-7922" 0x00007f161738fad3 in ?? ()
118 LWP 7923 "rpc worker-7923" 0x00007f161738fad3 in ?? ()
119 LWP 7926 "rpc worker-7926" 0x00007f161738fad3 in ?? ()
120 LWP 7927 "rpc worker-7927" 0x00007f161738fad3 in ?? ()
121 LWP 7928 "diag-logger-792" 0x00007f161738ffb9 in ?? ()
122 LWP 7929 "result-tracker-" 0x00007f161738ffb9 in ?? ()
123 LWP 7931 "excess-log-dele" 0x00007f161738ffb9 in ?? ()
124 LWP 7932 "acceptor-7932" 0x00007f1610320fc7 in ?? ()
125 LWP 7933 "heartbeat-7933" 0x00007f161738ffb9 in ?? ()
126 LWP 7934 "maintenance_sch" 0x00007f161738ffb9 in ?? ()
127 LWP 8486 "wal-append [wor" 0x00007f161738ffb9 in ?? ()
Thread 127 (LWP 8486):
#0 0x00007f161738ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 126 (LWP 7934):
#0 0x00007f161738ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 125 (LWP 7933):
#0 0x00007f161738ffb9 in ?? ()
#1 0x4008000000000000 in ?? ()
#2 0x000000000000000b in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000613000017d28 in ?? ()
#5 0x00007f15c8d4a770 in ?? ()
#6 0x0000000000000016 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 124 (LWP 7932):
#0 0x00007f1610320fc7 in ?? ()
#1 0x0000614000009490 in ?? ()
#2 0x00007f15c9571fa0 in ?? ()
#3 0x00007f15c9571fc0 in ?? ()
#4 0x00007f15c95720e0 in ?? ()
#5 0x00007f15c9571fd0 in ?? ()
#6 0x00007f15c9572040 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x000000000082c9d2 in __sanitizer::theDepot ()
#9 0x00000000008c3cc9 in __sanitizer::theDepot ()
#10 0x00000000008f35b0 in __sanitizer::theDepot ()
#11 0x0000000000000032 in ?? ()
#12 0x000060c00007ffd8 in ?? ()
#13 0x000060c000080032 in ?? ()
#14 0x00000000008875e1 in __sanitizer::theDepot ()
#15 0x000000000000005a in ?? ()
#16 0x000000000088348c in __sanitizer::theDepot ()
#17 0x0000000000000380 in ?? ()
#18 0x0000000000897749 in __sanitizer::theDepot ()
#19 0x0000000000010000 in ?? ()
#20 0x0000000000895799 in __sanitizer::theDepot ()
#21 0x0000000000e377a0 in ?? ()
#22 0x00000000007f9413 in __sanitizer::theDepot ()
#23 0x00000000009edb7d in __sanitizer::theDepot ()
#24 0x0000000000000700 in ?? ()
#25 0x00007f16173896db in ?? ()
#26 0x00007f16102d1346 in ?? ()
#27 0x00007f15c9571842 in ?? ()
#28 0x000000006a765f7b in ?? ()
#29 0x000000006a765f7b in ?? ()
#30 0x0000640000001140 in ?? ()
#31 0x00007f15c8d6be60 in ?? ()
#32 0x0000000000000049 in ?? ()
#33 0x0000017000000000 in ?? ()
#34 0x0000000000000017 in ?? ()
#35 0x0000640000001140 in ?? ()
#36 0x00000000008b1acd in __sanitizer::theDepot ()
#37 0x0000000000e377a0 in ?? ()
#38 0x00007f15c74ebfe8 in ?? ()
#39 0x0000000000000000 in ?? ()
Thread 123 (LWP 7931):
#0 0x00007f161738ffb9 in ?? ()
#1 0x00007fff7e7cbcf0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 122 (LWP 7929):
#0 0x00007f161738ffb9 in ?? ()
#1 0x00007f15ca5a3160 in ?? ()
#2 0x0000000000000006 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061100006ec30 in ?? ()
#5 0x00007f15ca5a3110 in ?? ()
#6 0x000000000000000c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 7928):
#0 0x00007f161738ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 7927):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 119 (LWP 7926):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 118 (LWP 7923):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 117 (LWP 7922):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 7921):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 7919):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 7918):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 7917):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 7916):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 7915):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 7913):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 7911):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 7910):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 7909):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 7908):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 7907):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 7905):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 7903):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 7902):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 7901):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 7899):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 7898):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 7896):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 7894):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 7893):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 7892):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 7891):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 7888):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 7887):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 7886):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 7885):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 7884):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 7883):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 7882):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 7879):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 7877):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 7876):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 7875):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 7874):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 7873):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 7872):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000068 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001d6858 in ?? ()
#4 0x00007f15df9a4eb0 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f15df9a4ed0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 79 (LWP 7871):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001d0058 in ?? ()
#4 0x00007f15e01bceb0 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f15e01bced0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 78 (LWP 7868):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 7867):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 7865):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 75 (LWP 7864):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 7863):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 7862):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 7861):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 7860):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 7859):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 7858):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 7857):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 7856):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 7855):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 7853):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 7851):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 7850):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 7849):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 7848):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 7847):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000010 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000150008 in ?? ()
#4 0x00007f15e9b95eb0 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f15e9b95ed0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 59 (LWP 7846):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 7843):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 7841):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 7840):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 7838):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 7837):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 7835):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 7834):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 7833):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 7830):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 7829):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 7828):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 7827):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 7826):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 7824):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 7823):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 7822):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 7821):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 7820):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 7819):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 7818):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 7813):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 7811):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 7809):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 7806):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 7805):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 7803):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 7802):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 7801):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 7800):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 7796):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 7795):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 7793):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 7791):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 7789):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 7788):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 7787):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 7786):
#0 0x00007f161031f947 in ?? ()
#1 0x0000614000041280 in ?? ()
#2 0x00006190000a0f80 in ?? ()
#3 0x00007f15fd878340 in ?? ()
#4 0x00007f15fd878540 in ?? ()
#5 0x00007f15fd878390 in ?? ()
#6 0x0000614000041280 in ?? ()
#7 0x00007f15fd878400 in ?? ()
#8 0x00007f1612926bda in ?? ()
#9 0x3fb98e040a000000 in ?? ()
#10 0x00006190000a0f80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x00006190000a0f80 in ?? ()
#13 0x00007f161ab3b340 in ?? ()
#14 0x00007f15fd878540 in ?? ()
#15 0x41da9797a7133371 in ?? ()
#16 0x000000003fb07080 in ?? ()
#17 0x00007f15fd8783e0 in ?? ()
#18 0x00007f161292a228 in ?? ()
#19 0x00000000fd8783c0 in ?? ()
#20 0x00006190000a0f80 in ?? ()
#21 0x0000614000041280 in ?? ()
#22 0x00007f15fd878400 in ?? ()
#23 0x3fb98e040a000000 in ?? ()
#24 0x413800e53a63e7f6 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 21 (LWP 7785):
#0 0x00007f161031f947 in ?? ()
#1 0x0000614000041080 in ?? ()
#2 0x00006190000a0a80 in ?? ()
#3 0x00007f15fe8aa340 in ?? ()
#4 0x00007f15fe8aa540 in ?? ()
#5 0x00007f15fe8aa390 in ?? ()
#6 0x0000614000041080 in ?? ()
#7 0x00007f15fe8aa400 in ?? ()
#8 0x00007f1612926bda in ?? ()
#9 0x3fb98cfb32000000 in ?? ()
#10 0x00006190000a0a80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x00006190000a0a80 in ?? ()
#13 0x00007f161ab3b340 in ?? ()
#14 0x00007f15fe8aa540 in ?? ()
#15 0x41da9797a7133373 in ?? ()
#16 0x000000003fd0d480 in ?? ()
#17 0x00007f15fe8aa3e0 in ?? ()
#18 0x00007f161292a228 in ?? ()
#19 0x00000000fe8aa3c0 in ?? ()
#20 0x00006190000a0a80 in ?? ()
#21 0x0000614000041080 in ?? ()
#22 0x00007f15fe8aa400 in ?? ()
#23 0x3fb98cfb32000000 in ?? ()
#24 0x413800e53a63db42 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 20 (LWP 7784):
#0 0x00007f161031f947 in ?? ()
#1 0x0000614000040e80 in ?? ()
#2 0x00006190000a0580 in ?? ()
#3 0x00007f15ff0c1340 in ?? ()
#4 0x00007f15ff0c1540 in ?? ()
#5 0x00007f15ff0c1390 in ?? ()
#6 0x0000614000040e80 in ?? ()
#7 0x00007f15ff0c1400 in ?? ()
#8 0x00007f1612926bda in ?? ()
#9 0x3fb987e4c7000000 in ?? ()
#10 0x00006190000a0580 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x00006190000a0580 in ?? ()
#13 0x00007f161ab3b340 in ?? ()
#14 0x00007f15ff0c1540 in ?? ()
#15 0x41da9797a7133373 in ?? ()
#16 0x000000003fe10280 in ?? ()
#17 0x00007f15ff0c13e0 in ?? ()
#18 0x00007f161292a228 in ?? ()
#19 0x00000000ff0c13c0 in ?? ()
#20 0x00006190000a0580 in ?? ()
#21 0x0000614000040e80 in ?? ()
#22 0x00007f15ff0c1400 in ?? ()
#23 0x3fb987e4c7000000 in ?? ()
#24 0x413800e53a671b65 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 19 (LWP 7783):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 18 (LWP 7782):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 17 (LWP 7781):
#0 0x00007f161031f947 in ?? ()
#1 0x0000614000040c80 in ?? ()
#2 0x00006190000a0080 in ?? ()
#3 0x00007f16008da340 in ?? ()
#4 0x00007f16008da540 in ?? ()
#5 0x00007f16008da390 in ?? ()
#6 0x0000614000040c80 in ?? ()
#7 0x00007f16008da400 in ?? ()
#8 0x00007f1612926bda in ?? ()
#9 0x3fb98072b4000000 in ?? ()
#10 0x00006190000a0080 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x00006190000a0080 in ?? ()
#13 0x00007f161ab3b340 in ?? ()
#14 0x00007f16008da540 in ?? ()
#15 0x41da9797a7133373 in ?? ()
#16 0x0000000040113480 in ?? ()
#17 0x00007f16008da3e0 in ?? ()
#18 0x00007f161292a228 in ?? ()
#19 0x00000000008da3c0 in __sanitizer::theDepot ()
#20 0x00006190000a0080 in ?? ()
#21 0x0000614000040c80 in ?? ()
#22 0x00007f16008da400 in ?? ()
#23 0x3fb98072b4000000 in ?? ()
#24 0x413800e53a67a13e in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 16 (LWP 7780):
#0 0x00007f161738ffb9 in ?? ()
#1 0x00000000ffffffff in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 15 (LWP 7779):
#0 0x00007f161738ffb9 in ?? ()
#1 0x4008000000000000 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001a618 in ?? ()
#5 0x00007f16018dc270 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 7773):
#0 0x00007f161738ffb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000003c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000618000004da8 in ?? ()
#5 0x00007f16030de250 in ?? ()
#6 0x0000000000000078 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 7771):
#0 0x00007f161738fad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 7770):
#0 0x00007f161031f947 in ?? ()
#1 0x0000614000009a80 in ?? ()
#2 0x0000619000059680 in ?? ()
#3 0x00007f16048e2340 in ?? ()
#4 0x00007f16048e2540 in ?? ()
#5 0x00007f16048e2390 in ?? ()
#6 0x0000614000009a80 in ?? ()
#7 0x00007f16048e2400 in ?? ()
#8 0x00007f1612926bda in ?? ()
#9 0x3fb9850a7c000000 in ?? ()
#10 0x0000619000059680 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059680 in ?? ()
#13 0x4222fd5ad6000000 in ?? ()
#14 0x00007f16048e2540 in ?? ()
#15 0x41da9797a7133372 in ?? ()
#16 0x0000000040914480 in ?? ()
#17 0x00007f16048e23e0 in ?? ()
#18 0x00007f161292a228 in ?? ()
#19 0x00000000048e23c0 in ?? ()
#20 0x0000619000059680 in ?? ()
#21 0x0000614000009a80 in ?? ()
#22 0x00007f16048e2400 in ?? ()
#23 0x3fb9850a7c000000 in ?? ()
#24 0x413800e52f687c11 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 7769):
#0 0x00007f161031f947 in ?? ()
#1 0x0000614000009880 in ?? ()
#2 0x0000619000059180 in ?? ()
#3 0x00007f16050e3340 in ?? ()
#4 0x00007f16050e3540 in ?? ()
#5 0x00007f16050e3390 in ?? ()
#6 0x0000614000009880 in ?? ()
#7 0x00007f16050e3400 in ?? ()
#8 0x00007f1612926bda in ?? ()
#9 0x3fa786db96000000 in ?? ()
#10 0x0000619000059180 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059180 in ?? ()
#13 0x4209d3d2f4000000 in ?? ()
#14 0x00007f16050e3540 in ?? ()
#15 0x41da9797a7133377 in ?? ()
#16 0x0000000040a14680 in ?? ()
#17 0x00007f16050e33e0 in ?? ()
#18 0x00007f161292a228 in ?? ()
#19 0x00000000050e33c0 in ?? ()
#20 0x0000619000059180 in ?? ()
#21 0x0000614000009880 in ?? ()
#22 0x00007f16050e3400 in ?? ()
#23 0x3fa786db96000000 in ?? ()
#24 0x413800e536451cd5 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 7767):
#0 0x00007f161031f947 in ?? ()
#1 0x0000614000009680 in ?? ()
#2 0x0000619000058c80 in ?? ()
#3 0x00007f16058e4340 in ?? ()
#4 0x00007f16058e4540 in ?? ()
#5 0x00007f16058e4390 in ?? ()
#6 0x0000614000009680 in ?? ()
#7 0x00007f16058e4400 in ?? ()
#8 0x00007f1612926bda in ?? ()
#9 0x3fb95ebb7f000000 in ?? ()
#10 0x0000619000058c80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058c80 in ?? ()
#13 0x422375c197000000 in ?? ()
#14 0x00007f16058e4540 in ?? ()
#15 0x41da9797a7133376 in ?? ()
#16 0x0000000040b14880 in ?? ()
#17 0x00007f16058e43e0 in ?? ()
#18 0x00007f161292a228 in ?? ()
#19 0x00000000058e43c0 in ?? ()
#20 0x0000619000058c80 in ?? ()
#21 0x0000614000009680 in ?? ()
#22 0x00007f16058e4400 in ?? ()
#23 0x3fb95ebb7f000000 in ?? ()
#24 0x413800e52f8aa75f in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 7764):
#0 0x00007f161031f947 in ?? ()
#1 0x0000614000009480 in ?? ()
#2 0x0000619000058780 in ?? ()
#3 0x00007f1606ead340 in ?? ()
#4 0x00007f1606ead540 in ?? ()
#5 0x00007f1606ead390 in ?? ()
#6 0x0000614000009480 in ?? ()
#7 0x00007f1606ead400 in ?? ()
#8 0x00007f1612926bda in ?? ()
#9 0x3fa3141bf2000000 in ?? ()
#10 0x0000619000058780 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058780 in ?? ()
#13 0x42561d6d8fc00000 in ?? ()
#14 0x00007f1606ead540 in ?? ()
#15 0x41da9797a7133377 in ?? ()
#16 0x0000000040dcda80 in ?? ()
#17 0x00007f1606ead3e0 in ?? ()
#18 0x00007f161292a228 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 7759):
#0 0x00007f1610312bb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f16088b77d8 in ?? ()
#3 0x000060200004d5b0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x0000000000823a92 in __sanitizer::theDepot ()
#7 0x00007f16088b7800 in ?? ()
#8 0x00000000000fe000 in ?? ()
#9 0x0000000000000000 in ?? ()
Thread 7 (LWP 7758):
#0 0x00007f161738ffb9 in ?? ()
#1 0x00007f16080b6140 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 7757):
#0 0x00007f16173939e2 in ?? ()
#1 0x00007f16078b4430 in ?? ()
#2 0x00007f16078b3f20 in ?? ()
#3 0x00007f16078b3650 in ?? ()
#4 0x00007f16078b51c0 in ?? ()
#5 0x00007f16078b3ed0 in ?? ()
#6 0x0000000000000030 in ?? ()
#7 0x00007f16078b3f30 in ?? ()
#8 0x000061100005a2d0 in ?? ()
#9 0x00007f16078b3f20 in ?? ()
#10 0x000000000081de90 in __sanitizer::theDepot ()
#11 0x00007f161cb57269 in ?? ()
#12 0x00000000009edb7d in __sanitizer::theDepot ()
#13 0x00007f16140afeaf in ?? ()
#14 0x00007f16173896db in ?? ()
#15 0x3a73250000000000 in ?? ()
#16 0x642520656e696c20 in ?? ()
#17 0x0000000000000000 in ?? ()
Thread 5 (LWP 7748):
#0 0x00007f161738ffb9 in ?? ()
#1 0x00007f16098b90a0 in ?? ()
#2 0x000000000000001e in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001ab28 in ?? ()
#5 0x00007f16098b8cd0 in ?? ()
#6 0x000000000000003c in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 7747):
#0 0x00007f161738ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 7746):
#0 0x00007f161738ffb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 7744):
#0 0x00007f161738ffb9 in ?? ()
#1 0x0000000000008000 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000009838 in ?? ()
#5 0x00007f160b0bc450 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 7743):
#0 0x00007f1617393d50 in ?? ()
#1 0x0000000000000000 in ?? ()
I20260807 22:43:13.656526 8522 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d6aee0f79fef46c5a7ba0d9714dc0a5e)
I20260807 22:43:13.656817 8522 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:13.657823 8522 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:13.658632 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:13.658632 8374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:13.659585 7764 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 753b63233367448f8f28e3a1719be1af; no voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:43:13.660010 8522 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
************************* END STACKS ***************************
I20260807 22:43:13.661480 4508 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID 672a15bd3cdb41c29eb4863f63479a24 and pid 8229
************************ BEGIN STACKS **************************
[New LWP 8231]
[New LWP 8232]
[New LWP 8233]
[New LWP 8234]
[New LWP 8245]
[New LWP 8246]
[New LWP 8250]
[New LWP 8256]
[New LWP 8257]
[New LWP 8258]
[New LWP 8259]
[New LWP 8263]
[New LWP 8268]
[New LWP 8272]
[New LWP 8274]
[New LWP 8276]
[New LWP 8280]
[New LWP 8281]
[New LWP 8282]
[New LWP 8283]
[New LWP 8284]
[New LWP 8286]
[New LWP 8287]
[New LWP 8288]
[New LWP 8289]
[New LWP 8290]
[New LWP 8291]
[New LWP 8292]
[New LWP 8293]
[New LWP 8294]
[New LWP 8295]
[New LWP 8297]
[New LWP 8298]
[New LWP 8302]
[New LWP 8305]
[New LWP 8306]
[New LWP 8308]
[New LWP 8309]
[New LWP 8310]
[New LWP 8311]
[New LWP 8312]
[New LWP 8313]
[New LWP 8315]
[New LWP 8320]
[New LWP 8321]
[New LWP 8322]
[New LWP 8323]
[New LWP 8324]
[New LWP 8325]
[New LWP 8326]
[New LWP 8330]
[New LWP 8332]
[New LWP 8335]
[New LWP 8336]
[New LWP 8338]
[New LWP 8340]
[New LWP 8342]
[New LWP 8344]
[New LWP 8345]
[New LWP 8349]
[New LWP 8350]
[New LWP 8351]
[New LWP 8352]
[New LWP 8354]
[New LWP 8355]
[New LWP 8357]
[New LWP 8358]
[New LWP 8360]
[New LWP 8362]
[New LWP 8363]
[New LWP 8364]
[New LWP 8366]
[New LWP 8367]
[New LWP 8368]
[New LWP 8369]
[New LWP 8371]
[New LWP 8372]
[New LWP 8373]
[New LWP 8374]
[New LWP 8375]
[New LWP 8376]
[New LWP 8377]
[New LWP 8383]
[New LWP 8384]
[New LWP 8385]
[New LWP 8387]
[New LWP 8388]
[New LWP 8390]
[New LWP 8391]
[New LWP 8392]
[New LWP 8393]
[New LWP 8395]
[New LWP 8396]
[New LWP 8397]
[New LWP 8398]
[New LWP 8402]
[New LWP 8403]
[New LWP 8406]
[New LWP 8407]
[New LWP 8408]
[New LWP 8409]
[New LWP 8413]
[New LWP 8415]
[New LWP 8416]
[New LWP 8418]
[New LWP 8419]
[New LWP 8420]
[New LWP 8423]
[New LWP 8425]
[New LWP 8426]
[New LWP 8429]
[New LWP 8431]
[New LWP 8433]
[New LWP 8434]
[New LWP 8435]
[New LWP 8436]
[New LWP 8439]
[New LWP 8441]
[New LWP 8444]
[New LWP 8445]
[New LWP 8446]
[New LWP 8447]
[New LWP 8448]
[New LWP 8449]
[New LWP 8450]
warning: Error reading shared library list entry at 0x4
warning: Corrupted shared library list: 0x0 != 0x74696172745f7261
warning: Error reading shared library list entry at 0x4
warning: Corrupted shared library list: 0x0 != 0x74696172745f7261
0x00007f2a1cf2ad50 in ?? ()
Id Target Id Frame
* 1 LWP 8229 "kudu" 0x00007f2a1cf2ad50 in ?? ()
2 LWP 8231 "kudu" 0x00007f2a1cf26fb9 in ?? ()
3 LWP 8232 "kudu" 0x00007f2a1cf26fb9 in ?? ()
4 LWP 8233 "kudu" 0x00007f2a1cf26fb9 in ?? ()
5 LWP 8234 "kernel-watcher-" 0x00007f2a1cf26fb9 in ?? ()
6 LWP 8245 "ntp client-8245" 0x00007f2a1cf2a9e2 in ?? ()
7 LWP 8246 "file cache-evic" 0x00007f2a1cf26fb9 in ?? ()
8 LWP 8250 "sq_acceptor" 0x00007f2a15ea9bb9 in ?? ()
9 LWP 8256 "rpc reactor-825" 0x00007f2a15eb6947 in ?? ()
10 LWP 8257 "rpc reactor-825" 0x00007f2a15eb6947 in ?? ()
11 LWP 8258 "rpc reactor-825" 0x00007f2a15eb6947 in ?? ()
12 LWP 8259 "rpc reactor-825" 0x00007f2a15eb6947 in ?? ()
13 LWP 8263 "MaintenanceMgr " 0x00007f2a1cf26ad3 in ?? ()
14 LWP 8268 "txn-status-mana" 0x00007f2a1cf26fb9 in ?? ()
15 LWP 8272 "collect_and_rem" 0x00007f2a1cf26fb9 in ?? ()
16 LWP 8274 "rpc reactor-827" 0x00007f2a15eb6947 in ?? ()
17 LWP 8276 "rpc reactor-827" 0x00007f2a15eb6947 in ?? ()
18 LWP 8280 "rpc reactor-828" 0x00007f2a15eb6947 in ?? ()
19 LWP 8281 "rpc reactor-828" 0x00007f2a15eb6947 in ?? ()
20 LWP 8282 "tc-session-exp-" 0x00007f2a1cf26fb9 in ?? ()
21 LWP 8283 "rpc worker-8283" 0x00007f2a1cf26ad3 in ?? ()
22 LWP 8284 "rpc worker-8284" 0x00007f2a1cf26ad3 in ?? ()
23 LWP 8286 "rpc worker-8286" 0x00007f2a1cf26ad3 in ?? ()
24 LWP 8287 "rpc worker-8287" 0x00007f2a1cf26ad3 in ?? ()
25 LWP 8288 "rpc worker-8288" 0x00007f2a1cf26ad3 in ?? ()
26 LWP 8289 "rpc worker-8289" 0x00007f2a1cf26ad3 in ?? ()
27 LWP 8290 "rpc worker-8290" 0x00007f2a1cf26ad3 in ?? ()
28 LWP 8291 "rpc worker-8291" 0x00007f2a1cf26ad3 in ?? ()
29 LWP 8292 "rpc worker-8292" 0x00007f2a1cf26ad3 in ?? ()
30 LWP 8293 "rpc worker-8293" 0x00007f2a1cf26ad3 in ?? ()
31 LWP 8294 "rpc worker-8294" 0x00007f2a1cf26ad3 in ?? ()
32 LWP 8295 "rpc worker-8295" 0x00007f2a1cf26ad3 in ?? ()
33 LWP 8297 "rpc worker-8297" 0x00007f2a1cf26ad3 in ?? ()
34 LWP 8298 "rpc worker-8298" 0x00007f2a1cf26ad3 in ?? ()
35 LWP 8302 "rpc worker-8302" 0x00007f2a1cf26ad3 in ?? ()
36 LWP 8305 "rpc worker-8305" 0x00007f2a1cf26ad3 in ?? ()
37 LWP 8306 "rpc worker-8306" 0x00007f2a1cf26ad3 in ?? ()
38 LWP 8308 "rpc worker-8308" 0x00007f2a1cf26ad3 in ?? ()
39 LWP 8309 "rpc worker-8309" 0x00007f2a1cf26ad3 in ?? ()
40 LWP 8310 "rpc worker-8310" 0x00007f2a1cf26ad3 in ?? ()
41 LWP 8311 "rpc worker-8311" 0x00007f2a1cf26ad3 in ?? ()
42 LWP 8312 "rpc worker-8312" 0x00007f2a1cf26ad3 in ?? ()
43 LWP 8313 "rpc worker-8313" 0x00007f2a1cf26ad3 in ?? ()
44 LWP 8315 "rpc worker-8315" 0x00007f2a1cf26ad3 in ?? ()
45 LWP 8320 "rpc worker-8320" 0x00007f2a1cf26ad3 in ?? ()
46 LWP 8321 "rpc worker-8321" 0x00007f2a1cf26ad3 in ?? ()
47 LWP 8322 "rpc worker-8322" 0x00007f2a1cf26ad3 in ?? ()
48 LWP 8323 "rpc worker-8323" 0x00007f2a1cf26ad3 in ?? ()
49 LWP 8324 "rpc worker-8324" 0x00007f2a1cf26ad3 in ?? ()
50 LWP 8325 "rpc worker-8325" 0x00007f2a1cf26ad3 in ?? ()
51 LWP 8326 "rpc worker-8326" 0x00007f2a1cf26ad3 in ?? ()
52 LWP 8330 "rpc worker-8330" 0x00007f2a1cf26ad3 in ?? ()
53 LWP 8332 "rpc worker-8332" 0x00007f2a1cf26ad3 in ?? ()
54 LWP 8335 "rpc worker-8335" 0x00007f2a1cf26ad3 in ?? ()
55 LWP 8336 "rpc worker-8336" 0x00007f2a1cf26ad3 in ?? ()
56 LWP 8338 "rpc worker-8338" 0x00007f2a1cf26ad3 in ?? ()
57 LWP 8340 "rpc worker-8340" 0x00007f2a1cf26ad3 in ?? ()
58 LWP 8342 "rpc worker-8342" 0x00007f2a1cf26ad3 in ?? ()
59 LWP 8344 "rpc worker-8344" 0x00007f2a1cf26ad3 in ?? ()
60 LWP 8345 "rpc worker-8345" 0x00007f2a1cf26ad3 in ?? ()
61 LWP 8349 "rpc worker-8349" 0x00007f2a1cf26ad3 in ?? ()
62 LWP 8350 "rpc worker-8350" 0x00007f2a1cf26ad3 in ?? ()
63 LWP 8351 "rpc worker-8351" 0x00007f2a1cf26ad3 in ?? ()
64 LWP 8352 "rpc worker-8352" 0x00007f2a1cf26ad3 in ?? ()
65 LWP 8354 "rpc worker-8354" 0x00007f2a1cf26ad3 in ?? ()
66 LWP 8355 "rpc worker-8355" 0x00007f2a1cf26ad3 in ?? ()
67 LWP 8357 "rpc worker-8357" 0x00007f2a1cf26ad3 in ?? ()
68 LWP 8358 "rpc worker-8358" 0x00007f2a1cf26ad3 in ?? ()
69 LWP 8360 "rpc worker-8360" 0x00007f2a1cf26ad3 in ?? ()
70 LWP 8362 "rpc worker-8362" 0x00007f2a1cf26ad3 in ?? ()
71 LWP 8363 "rpc worker-8363" 0x00007f2a1cf26ad3 in ?? ()
72 LWP 8364 "rpc worker-8364" 0x00007f2a1cf26ad3 in ?? ()
73 LWP 8366 "rpc worker-8366" 0x00007f2a1cf26ad3 in ?? ()
74 LWP 8367 "rpc worker-8367" 0x00007f2a1cf26ad3 in ?? ()
75 LWP 8368 "rpc worker-8368" 0x00007f2a1cf26ad3 in ?? ()
76 LWP 8369 "rpc worker-8369" 0x00007f2a1cf26ad3 in ?? ()
77 LWP 8371 "rpc worker-8371" 0x00007f2a1cf26ad3 in ?? ()
78 LWP 8372 "rpc worker-8372" 0x00007f2a1cf26ad3 in ?? ()
79 LWP 8373 "rpc worker-8373" 0x00007f2a1cf26ad3 in ?? ()
80 LWP 8374 "rpc worker-8374" 0x00007f2a1cf26ad3 in ?? ()
81 LWP 8375 "rpc worker-8375" 0x00007f2a1cf26ad3 in ?? ()
82 LWP 8376 "rpc worker-8376" 0x00007f2a1cf26ad3 in ?? ()
83 LWP 8377 "rpc worker-8377" 0x00007f2a1cf26ad3 in ?? ()
84 LWP 8383 "rpc worker-8383" 0x00007f2a1cf26ad3 in ?? ()
85 LWP 8384 "rpc worker-8384" 0x00007f2a1cf26ad3 in ?? ()
86 LWP 8385 "rpc worker-8385" 0x00007f2a1cf26ad3 in ?? ()
87 LWP 8387 "rpc worker-8387" 0x00007f2a1cf26ad3 in ?? ()
88 LWP 8388 "rpc worker-8388" 0x00007f2a1cf26ad3 in ?? ()
89 LWP 8390 "rpc worker-8390" 0x00007f2a1cf26ad3 in ?? ()
90 LWP 8391 "rpc worker-8391" 0x00007f2a1cf26ad3 in ?? ()
91 LWP 8392 "rpc worker-8392" 0x00007f2a1cf26ad3 in ?? ()
92 LWP 8393 "rpc worker-8393" 0x00007f2a1cf26ad3 in ?? ()
93 LWP 8395 "rpc worker-8395" 0x00007f2a1cf26ad3 in ?? ()
94 LWP 8396 "rpc worker-8396" 0x00007f2a1cf26ad3 in ?? ()
95 LWP 8397 "rpc worker-8397" 0x00007f2a1cf26ad3 in ?? ()
96 LWP 8398 "rpc worker-8398" 0x00007f2a1cf26ad3 in ?? ()
97 LWP 8402 "rpc worker-8402" 0x00007f2a1cf26ad3 in ?? ()
98 LWP 8403 "rpc worker-8403" 0x00007f2a1cf26ad3 in ?? ()
99 LWP 8406 "rpc worker-8406" 0x00007f2a1cf26ad3 in ?? ()
100 LWP 8407 "rpc worker-8407" 0x00007f2a1cf26ad3 in ?? ()
101 LWP 8408 "rpc worker-8408" 0x00007f2a1cf26ad3 in ?? ()
102 LWP 8409 "rpc worker-8409" 0x00007f2a1cf26ad3 in ?? ()
103 LWP 8413 "rpc worker-8413" 0x00007f2a1cf26ad3 in ?? ()
104 LWP 8415 "rpc worker-8415" 0x00007f2a1cf26ad3 in ?? ()
105 LWP 8416 "rpc worker-8416" 0x00007f2a1cf26ad3 in ?? ()
106 LWP 8418 "rpc worker-8418" 0x00007f2a1cf26ad3 in ?? ()
107 LWP 8419 "rpc worker-8419" 0x00007f2a1cf26ad3 in ?? ()
108 LWP 8420 "rpc worker-8420" 0x00007f2a1cf26ad3 in ?? ()
109 LWP 8423 "rpc worker-8423" 0x00007f2a1cf26ad3 in ?? ()
110 LWP 8425 "rpc worker-8425" 0x00007f2a1cf26ad3 in ?? ()
111 LWP 8426 "rpc worker-8426" 0x00007f2a1cf26ad3 in ?? ()
112 LWP 8429 "rpc worker-8429" 0x00007f2a1cf26ad3 in ?? ()
113 LWP 8431 "rpc worker-8431" 0x00007f2a1cf26ad3 in ?? ()
114 LWP 8433 "rpc worker-8433" 0x00007f2a1cf26ad3 in ?? ()
115 LWP 8434 "rpc worker-8434" 0x00007f2a1cf26ad3 in ?? ()
116 LWP 8435 "rpc worker-8435" 0x00007f2a1cf26ad3 in ?? ()
117 LWP 8436 "rpc worker-8436" 0x00007f2a1cf26ad3 in ?? ()
118 LWP 8439 "rpc worker-8439" 0x00007f2a1cf26ad3 in ?? ()
119 LWP 8441 "rpc worker-8441" 0x00007f2a1cf26ad3 in ?? ()
120 LWP 8444 "rpc worker-8444" 0x00007f2a1cf26ad3 in ?? ()
121 LWP 8445 "diag-logger-844" 0x00007f2a1cf26fb9 in ?? ()
122 LWP 8446 "result-tracker-" 0x00007f2a1cf26fb9 in ?? ()
123 LWP 8447 "excess-log-dele" 0x00007f2a1cf26fb9 in ?? ()
124 LWP 8448 "acceptor-8448" 0x00007f2a15eb7fc7 in ?? ()
125 LWP 8449 "heartbeat-8449" 0x00007f2a1cf26fb9 in ?? ()
126 LWP 8450 "maintenance_sch" 0x00007f2a1cf26fb9 in ?? ()
Thread 126 (LWP 8450):
#0 0x00007f2a1cf26fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 125 (LWP 8449):
#0 0x00007f2a1cf26fb9 in ?? ()
#1 0xffffffffffffffff in ?? ()
#2 0x0000000000000004 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x0000613000017d2c in ?? ()
#5 0x00007f29ce0b9770 in ?? ()
#6 0x0000000000000009 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f29ce0b9790 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007f29ce0b9810 in ?? ()
#12 0x00000fe539c17304 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 124 (LWP 8448):
#0 0x00007f2a15eb7fc7 in ?? ()
#1 0x0000614000009a90 in ?? ()
#2 0x00007f29ce8e0fa0 in ?? ()
#3 0x00007f29ce8e0fc0 in ?? ()
#4 0x00007f29ce8e10e0 in ?? ()
#5 0x00007f29ce8e0fd0 in ?? ()
#6 0x00007f29ce8e1040 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x000000000082c9d2 in __sanitizer::theDepot ()
#9 0x00000000008c3cc9 in __sanitizer::theDepot ()
#10 0x00000000008f35b0 in __sanitizer::theDepot ()
#11 0x0000000000000032 in ?? ()
#12 0x000060c00007ffd8 in ?? ()
#13 0x000060c000080032 in ?? ()
#14 0x00000000008875e1 in __sanitizer::theDepot ()
#15 0x000000000000005a in ?? ()
#16 0x000000000088348c in __sanitizer::theDepot ()
#17 0x0000000000000380 in ?? ()
#18 0x0000000000897749 in __sanitizer::theDepot ()
#19 0x0000000000010000 in ?? ()
#20 0x0000000000895799 in __sanitizer::theDepot ()
#21 0x0000000000e377a0 in ?? ()
#22 0x00000000007f9413 in __sanitizer::theDepot ()
#23 0x00000000009edb7d in __sanitizer::theDepot ()
#24 0x0000000000000700 in ?? ()
#25 0x00007f2a1cf206db in ?? ()
#26 0x00007f2a15e68346 in ?? ()
#27 0x00007f29ce8e0842 in ?? ()
#28 0x000000006a765f7f in ?? ()
#29 0x000000006a765f7f in ?? ()
#30 0x0000640000001140 in ?? ()
#31 0x00007f29ce0dae60 in ?? ()
#32 0x0000000000000049 in ?? ()
#33 0x0000017000000000 in ?? ()
#34 0x0000000000000017 in ?? ()
#35 0x0000640000001140 in ?? ()
#36 0x00000000008b1acd in __sanitizer::theDepot ()
#37 0x0000000000e377a0 in ?? ()
#38 0x00007f29cd08b058 in ?? ()
#39 0x0000000000000000 in ?? ()
Thread 123 (LWP 8447):
#0 0x00007f2a1cf26fb9 in ?? ()
#1 0x00007fffb03e9bd0 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 122 (LWP 8446):
#0 0x00007f2a1cf26fb9 in ?? ()
#1 0x00007f29cf912160 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061100006ec30 in ?? ()
#5 0x00007f29cf912110 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 8445):
#0 0x00007f2a1cf26fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 120 (LWP 8444):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 119 (LWP 8441):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 118 (LWP 8439):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 117 (LWP 8436):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 8435):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 8434):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 8433):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 8431):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 8429):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 8426):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 8425):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 8423):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 8420):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 8419):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 8418):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 8416):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 8415):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 8413):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 8409):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 8408):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 8407):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 8406):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 8403):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 8402):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 8398):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 8397):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 8396):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 8395):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 8393):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 8392):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 8391):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 8390):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 8388):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 8387):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 8385):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 8384):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 8383):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 8377):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 8376):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 81 (LWP 8375):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 8374):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x000000000000003c in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001d6858 in ?? ()
#4 0x00007f29e553beb0 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f29e553bed0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 79 (LWP 8373):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d0001d005c in ?? ()
#4 0x00007f29e5d53eb0 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f29e5d53ed0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d0001d0010 in ?? ()
#9 0x00007f2a1cf26770 in ?? ()
#10 0x00007f29e5d53ed0 in ?? ()
#11 0x00007f29e5d53fa0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 78 (LWP 8372):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 8371):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 8369):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 75 (LWP 8368):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 8367):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 8366):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 8364):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 8363):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 8362):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 8360):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 8358):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 8357):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 8355):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 8354):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 8352):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 8351):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 8350):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 8349):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 8345):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000001 in ?? ()
#2 0x0000000100000081 in ?? ()
#3 0x000060d00015000c in ?? ()
#4 0x00007f29ef71ceb0 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f29ef71ced0 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x000060d00014ffc0 in ?? ()
#9 0x00007f2a1cf26770 in ?? ()
#10 0x00007f29ef71ced0 in ?? ()
#11 0x00007f29ef71cfa0 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 59 (LWP 8344):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 8342):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 8340):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 8338):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 8336):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 8335):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 8332):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 8330):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 8326):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 8325):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 8324):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 8323):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 8322):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 8321):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 8320):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 8315):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 8313):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 8312):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 8311):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 8310):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 8309):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 8308):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 8306):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 8305):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 8302):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 8298):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 8297):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 8295):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 8294):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 8293):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 8292):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 8291):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 8290):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 8289):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 8288):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 8287):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 8286):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 8284):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 8283):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 8282):
#0 0x00007f2a1cf26fb9 in ?? ()
#1 0x00000000ffffffff in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 19 (LWP 8281):
#0 0x00007f2a15eb6947 in ?? ()
#1 0x0000614000021280 in ?? ()
#2 0x00006190000a0f80 in ?? ()
#3 0x00007f2a05486340 in ?? ()
#4 0x00007f2a05486540 in ?? ()
#5 0x00007f2a05486390 in ?? ()
#6 0x0000614000021280 in ?? ()
#7 0x00007f2a05486400 in ?? ()
#8 0x00007f2a184bdbda in ?? ()
#9 0x3fb978b718000000 in ?? ()
#10 0x00006190000a0f80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x00006190000a0f80 in ?? ()
#13 0x00007f2a206d2340 in ?? ()
#14 0x00007f2a05486540 in ?? ()
#15 0x41da9797a7133376 in ?? ()
#16 0x00000000c0a88c80 in ?? ()
#17 0x00007f2a054863e0 in ?? ()
#18 0x00007f2a184c1228 in ?? ()
#19 0x00000000054863c0 in ?? ()
#20 0x00006190000a0f80 in ?? ()
#21 0x0000614000021280 in ?? ()
#22 0x00007f2a05486400 in ?? ()
#23 0x3fb978b718000000 in ?? ()
#24 0x413800e5e7852c69 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 18 (LWP 8280):
#0 0x00007f2a15eb6947 in ?? ()
#1 0x0000614000021080 in ?? ()
#2 0x00006190000a0a80 in ?? ()
#3 0x00007f2a05cad340 in ?? ()
#4 0x00007f2a05cad540 in ?? ()
#5 0x00007f2a05cad390 in ?? ()
#6 0x0000614000021080 in ?? ()
#7 0x00007f2a05cad400 in ?? ()
#8 0x00007f2a184bdbda in ?? ()
#9 0x3fb9789368000000 in ?? ()
#10 0x00006190000a0a80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x00006190000a0a80 in ?? ()
#13 0x00007f2a206d2340 in ?? ()
#14 0x00007f2a05cad540 in ?? ()
#15 0x41da9797a7133374 in ?? ()
#16 0x00000000c0b8da80 in ?? ()
#17 0x00007f2a05cad3e0 in ?? ()
#18 0x00007f2a184c1228 in ?? ()
#19 0x0000000005cad3c0 in ?? ()
#20 0x00006190000a0a80 in ?? ()
#21 0x0000614000021080 in ?? ()
#22 0x00007f2a05cad400 in ?? ()
#23 0x3fb9789368000000 in ?? ()
#24 0x413800e5e7834139 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 17 (LWP 8276):
#0 0x00007f2a15eb6947 in ?? ()
#1 0x0000614000020e80 in ?? ()
#2 0x00006190000a0580 in ?? ()
#3 0x00007f2a064c4340 in ?? ()
#4 0x00007f2a064c4540 in ?? ()
#5 0x00007f2a064c4390 in ?? ()
#6 0x0000614000020e80 in ?? ()
#7 0x00007f2a064c4400 in ?? ()
#8 0x00007f2a184bdbda in ?? ()
#9 0x3fb9788840000000 in ?? ()
#10 0x00006190000a0580 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x00006190000a0580 in ?? ()
#13 0x00007f2a206d2340 in ?? ()
#14 0x00007f2a064c4540 in ?? ()
#15 0x41da9797a7133374 in ?? ()
#16 0x00000000c0c90880 in ?? ()
#17 0x00007f2a064c43e0 in ?? ()
#18 0x00007f2a184c1228 in ?? ()
#19 0x00000000064c43c0 in ?? ()
#20 0x00006190000a0580 in ?? ()
#21 0x0000614000020e80 in ?? ()
#22 0x00007f2a064c4400 in ?? ()
#23 0x3fb9788840000000 in ?? ()
#24 0x413800e5e77d473e in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 16 (LWP 8274):
#0 0x00007f2a15eb6947 in ?? ()
#1 0x0000614000020c80 in ?? ()
#2 0x00006190000a0080 in ?? ()
#3 0x00007f2a06cdb340 in ?? ()
#4 0x00007f2a06cdb540 in ?? ()
#5 0x00007f2a06cdb390 in ?? ()
#6 0x0000614000020c80 in ?? ()
#7 0x00007f2a06cdb400 in ?? ()
#8 0x00007f2a184bdbda in ?? ()
#9 0x3fb9713c8b000000 in ?? ()
#10 0x00006190000a0080 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x00006190000a0080 in ?? ()
#13 0x00007f2a206d2340 in ?? ()
#14 0x00007f2a06cdb540 in ?? ()
#15 0x41da9797a7133374 in ?? ()
#16 0x00000000c0d93680 in ?? ()
#17 0x00007f2a06cdb3e0 in ?? ()
#18 0x00007f2a184c1228 in ?? ()
#19 0x0000000006cdb3c0 in ?? ()
#20 0x00006190000a0080 in ?? ()
#21 0x0000614000020c80 in ?? ()
#22 0x00007f2a06cdb400 in ?? ()
#23 0x3fb9713c8b000000 in ?? ()
#24 0x413800e5e783a214 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 15 (LWP 8272):
#0 0x00007f2a1cf26fb9 in ?? ()
#1 0x4008000000000000 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 14 (LWP 8268):
#0 0x00007f2a1cf26fb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x000000000000001d in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000618000004da8 in ?? ()
#5 0x00007f2a08cde250 in ?? ()
#6 0x000000000000003a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 8263):
#0 0x00007f2a1cf26ad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 8259):
#0 0x00007f2a15eb6947 in ?? ()
#1 0x0000614000009a80 in ?? ()
#2 0x0000619000059680 in ?? ()
#3 0x00007f2a0a4e2340 in ?? ()
#4 0x00007f2a0a4e2540 in ?? ()
#5 0x00007f2a0a4e2390 in ?? ()
#6 0x0000614000009a80 in ?? ()
#7 0x00007f2a0a4e2400 in ?? ()
#8 0x00007f2a184bdbda in ?? ()
#9 0x3fb96f0223000000 in ?? ()
#10 0x0000619000059680 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059680 in ?? ()
#13 0x421e211457000000 in ?? ()
#14 0x00007f2a0a4e2540 in ?? ()
#15 0x41da9797a7133373 in ?? ()
#16 0x00000000c1494480 in ?? ()
#17 0x00007f2a0a4e23e0 in ?? ()
#18 0x00007f2a184c1228 in ?? ()
#19 0x000000000a4e23c0 in ?? ()
#20 0x0000619000059680 in ?? ()
#21 0x0000614000009a80 in ?? ()
#22 0x00007f2a0a4e2400 in ?? ()
#23 0x3fb96f0223000000 in ?? ()
#24 0x413800e5f3c8c36d in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 8258):
#0 0x00007f2a15eb6947 in ?? ()
#1 0x0000614000009880 in ?? ()
#2 0x0000619000059180 in ?? ()
#3 0x00007f2a0ace3340 in ?? ()
#4 0x00007f2a0ace3540 in ?? ()
#5 0x00007f2a0ace3390 in ?? ()
#6 0x0000614000009880 in ?? ()
#7 0x00007f2a0ace3400 in ?? ()
#8 0x00007f2a184bdbda in ?? ()
#9 0x3fb4340fe7000000 in ?? ()
#10 0x0000619000059180 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059180 in ?? ()
#13 0x425331a49bd00000 in ?? ()
#14 0x00007f2a0ace3540 in ?? ()
#15 0x41da9797a7133374 in ?? ()
#16 0x00000000c1594680 in ?? ()
#17 0x00007f2a0ace33e0 in ?? ()
#18 0x00007f2a184c1228 in ?? ()
#19 0x000000000ace33c0 in ?? ()
#20 0x0000619000059180 in ?? ()
#21 0x0000614000009880 in ?? ()
#22 0x00007f2a0ace3400 in ?? ()
#23 0x3fb4340fe7000000 in ?? ()
#24 0x413800e5f8f9a732 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 10 (LWP 8257):
#0 0x00007f2a15eb6947 in ?? ()
#1 0x0000614000009680 in ?? ()
#2 0x0000619000058c80 in ?? ()
#3 0x00007f2a0b4e4340 in ?? ()
#4 0x00007f2a0b4e4540 in ?? ()
#5 0x00007f2a0b4e4390 in ?? ()
#6 0x0000614000009680 in ?? ()
#7 0x00007f2a0b4e4400 in ?? ()
#8 0x00007f2a184bdbda in ?? ()
#9 0x3fa78c6a20000000 in ?? ()
#10 0x0000619000058c80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058c80 in ?? ()
#13 0x420791e5bc000000 in ?? ()
#14 0x00007f2a0b4e4540 in ?? ()
#15 0x41da9797a7133374 in ?? ()
#16 0x00000000c1694880 in ?? ()
#17 0x00007f2a0b4e43e0 in ?? ()
#18 0x00007f2a184c1228 in ?? ()
#19 0x000000000b4e43c0 in ?? ()
#20 0x0000619000058c80 in ?? ()
#21 0x0000614000009680 in ?? ()
#22 0x00007f2a0b4e4400 in ?? ()
#23 0x3fa78c6a20000000 in ?? ()
#24 0x413800e5f9200da7 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 8256):
#0 0x00007f2a15eb6947 in ?? ()
#1 0x0000614000009480 in ?? ()
#2 0x0000619000058780 in ?? ()
#3 0x00007f2a0caad340 in ?? ()
#4 0x00007f2a0caad540 in ?? ()
#5 0x00007f2a0caad390 in ?? ()
#6 0x0000614000009480 in ?? ()
#7 0x00007f2a0caad400 in ?? ()
#8 0x00007f2a184bdbda in ?? ()
#9 0x3fb96abc0b000000 in ?? ()
#10 0x0000619000058780 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058780 in ?? ()
#13 0x422c18d9a1000000 in ?? ()
#14 0x00007f2a0caad540 in ?? ()
#15 0x41da9797a7133375 in ?? ()
#16 0x00000000c194da80 in ?? ()
#17 0x00007f2a0caad3e0 in ?? ()
#18 0x00007f2a184c1228 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 8250):
#0 0x00007f2a15ea9bb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f2a0e4b77d8 in ?? ()
#3 0x000060200004d530 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x0000000000823a92 in __sanitizer::theDepot ()
#7 0x00007f2a0e4b7800 in ?? ()
#8 0x00000000000fe000 in ?? ()
#9 0x0000000000000000 in ?? ()
Thread 7 (LWP 8246):
#0 0x00007f2a1cf26fb9 in ?? ()
#1 0x00007f2a0dcb6140 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 8245):
#0 0x00007f2a1cf2a9e2 in ?? ()
#1 0x00007f2a0d4b4430 in ?? ()
#2 0x00007f2a0d4b3f20 in ?? ()
#3 0x00007f2a0d4b3650 in ?? ()
#4 0x00007f2a0d4b51c0 in ?? ()
#5 0x00007f2a0d4b3ed0 in ?? ()
#6 0x0000000000000030 in ?? ()
#7 0x00007f2a0d4b3f30 in ?? ()
#8 0x000061100005a2d0 in ?? ()
#9 0x00007f2a0d4b3f20 in ?? ()
#10 0x000000000081de90 in __sanitizer::theDepot ()
#11 0x00007f2a226ee269 in ?? ()
#12 0x00000000009edb7d in __sanitizer::theDepot ()
#13 0x00007f2a19c46eaf in ?? ()
#14 0x00007f2a1cf206db in ?? ()
#15 0xbcaf48ad494ea400 in ?? ()
#16 0x0000000000000000 in ?? ()
Thread 5 (LWP 8234):
#0 0x00007f2a1cf26fb9 in ?? ()
#1 0x00007f2a0f4b90a0 in ?? ()
#2 0x000000000000000f in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001ab28 in ?? ()
#5 0x00007f2a0f4b8cd0 in ?? ()
#6 0x000000000000001e in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 8233):
#0 0x00007f2a1cf26fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 8232):
#0 0x00007f2a1cf26fb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 8231):
#0 0x00007f2a1cf26fb9 in ?? ()
#1 0x0000000000008000 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000009838 in ?? ()
#5 0x00007f2a10cbc450 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 8229):
#0 0x00007f2a1cf2ad50 in ?? ()
#1 0x0000000000000000 in ?? ()
I20260807 22:43:14.403532 8536 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d6aee0f79fef46c5a7ba0d9714dc0a5e)
I20260807 22:43:14.403810 8536 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:14.404839 8536 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
I20260807 22:43:14.405726 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:14.405754 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:14.406670 8259 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24; no voters: 753b63233367448f8f28e3a1719be1af, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:43:14.407035 8536 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
************************* END STACKS ***************************
I20260807 22:43:14.413931 4508 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID d6aee0f79fef46c5a7ba0d9714dc0a5e and pid 7984
************************ BEGIN STACKS **************************
[New LWP 7985]
[New LWP 7986]
[New LWP 7988]
[New LWP 7989]
[New LWP 7999]
[New LWP 8000]
[New LWP 8001]
[New LWP 8005]
[New LWP 8006]
[New LWP 8007]
[New LWP 8008]
[New LWP 8009]
[New LWP 8011]
[New LWP 8018]
[New LWP 8019]
[New LWP 8020]
[New LWP 8021]
[New LWP 8022]
[New LWP 8023]
[New LWP 8028]
[New LWP 8029]
[New LWP 8031]
[New LWP 8032]
[New LWP 8033]
[New LWP 8035]
[New LWP 8036]
[New LWP 8037]
[New LWP 8038]
[New LWP 8039]
[New LWP 8041]
[New LWP 8042]
[New LWP 8044]
[New LWP 8045]
[New LWP 8046]
[New LWP 8047]
[New LWP 8049]
[New LWP 8051]
[New LWP 8052]
[New LWP 8053]
[New LWP 8054]
[New LWP 8056]
[New LWP 8057]
[New LWP 8058]
[New LWP 8060]
[New LWP 8062]
[New LWP 8063]
[New LWP 8064]
[New LWP 8066]
[New LWP 8067]
[New LWP 8068]
[New LWP 8069]
[New LWP 8070]
[New LWP 8073]
[New LWP 8076]
[New LWP 8078]
[New LWP 8080]
[New LWP 8081]
[New LWP 8084]
[New LWP 8085]
[New LWP 8086]
[New LWP 8087]
[New LWP 8088]
[New LWP 8089]
[New LWP 8090]
[New LWP 8092]
[New LWP 8093]
[New LWP 8095]
[New LWP 8097]
[New LWP 8098]
[New LWP 8101]
[New LWP 8103]
[New LWP 8104]
[New LWP 8106]
[New LWP 8107]
[New LWP 8109]
[New LWP 8110]
[New LWP 8111]
[New LWP 8112]
[New LWP 8113]
[New LWP 8114]
[New LWP 8115]
[New LWP 8116]
[New LWP 8118]
[New LWP 8119]
[New LWP 8120]
[New LWP 8121]
[New LWP 8124]
[New LWP 8125]
[New LWP 8126]
[New LWP 8127]
[New LWP 8130]
[New LWP 8131]
[New LWP 8134]
[New LWP 8135]
[New LWP 8137]
[New LWP 8139]
[New LWP 8142]
[New LWP 8143]
[New LWP 8145]
[New LWP 8146]
[New LWP 8147]
[New LWP 8148]
[New LWP 8150]
[New LWP 8151]
[New LWP 8152]
[New LWP 8153]
[New LWP 8154]
[New LWP 8155]
[New LWP 8156]
[New LWP 8158]
[New LWP 8159]
[New LWP 8161]
[New LWP 8162]
[New LWP 8164]
[New LWP 8165]
[New LWP 8166]
[New LWP 8168]
[New LWP 8169]
[New LWP 8170]
[New LWP 8171]
[New LWP 8172]
[New LWP 8175]
[New LWP 8176]
[New LWP 8178]
[New LWP 8181]
[New LWP 8465]
[New LWP 8544]
warning: Error reading shared library list entry at 0x4
warning: Corrupted shared library list: 0x0 != 0x74696172745f7261
warning: Error reading shared library list entry at 0x4
warning: Corrupted shared library list: 0x0 != 0x74696172745f7261
I20260807 22:43:15.082953 8551 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d6aee0f79fef46c5a7ba0d9714dc0a5e)
0x00007f8058e21d50 in ?? ()
I20260807 22:43:15.083277 8551 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
Id Target Id Frame
* 1 LWP 7984 "kudu" 0x00007f8058e21d50 in ?? ()
2 LWP 7985 "kudu" 0x00007f8058e1dfb9 in ?? ()
3 LWP 7986 "kudu" 0x00007f8058e1dfb9 in ?? ()
4 LWP 7988 "kudu" 0x00007f8058e1dfb9 in ?? ()
5 LWP 7989 "kernel-watcher-" 0x00007f8058e1dfb9 in ?? ()
6 LWP 7999 "ntp client-7999" 0x00007f8058e219e2 in ?? ()
I20260807 22:43:15.084444 8551 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
7 LWP 8000 "file cache-evic" 0x00007f8058e1dfb9 in ?? ()
8 LWP 8001 "sq_acceptor" 0x00007f8051da0bb9 in ?? ()
9 LWP 8005 "rpc reactor-800" 0x00007f8051dad947 in ?? ()
10 LWP 8006 "rpc reactor-800" 0x00007f8051dad947 in ?? ()
11 LWP 8007 "rpc reactor-800" 0x00007f8051dad947 in ?? ()
12 LWP 8008 "rpc reactor-800" 0x00007f8051dad947 in ?? ()
13 LWP 8009 "MaintenanceMgr " 0x00007f8058e1dad3 in ?? ()
14 LWP 8011 "txn-status-mana" 0x00007f8058e1dfb9 in ?? ()
I20260807 22:43:15.085263 8374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
15 LWP 8018 "collect_and_rem" 0x00007f8058e1dfb9 in ?? ()
16 LWP 8019 "rpc reactor-801" 0x00007f8051dad947 in ?? ()
17 LWP 8020 "rpc reactor-802" 0x00007f8051dad947 in ?? ()
18 LWP 8021 "rpc reactor-802" 0x00007f8051dad947 in ?? ()
19 LWP 8022 "rpc reactor-802" 0x00007f8051dad947 in ?? ()
20 LWP 8023 "tc-session-exp-" 0x00007f8058e1dfb9 in ?? ()
21 LWP 8028 "rpc worker-8028" 0x00007f8058e1dad3 in ?? ()
22 LWP 8029 "rpc worker-8029" 0x00007f8058e1dad3 in ?? ()
23 LWP 8031 "rpc worker-8031" 0x00007f8058e1dad3 in ?? ()
24 LWP 8032 "rpc worker-8032" 0x00007f8058e1dad3 in ?? ()
25 LWP 8033 "rpc worker-8033" 0x00007f8058e1dad3 in ?? ()
26 LWP 8035 "rpc worker-8035" 0x00007f8058e1dad3 in ?? ()
27 LWP 8036 "rpc worker-8036" 0x00007f8058e1dad3 in ?? ()
28 LWP 8037 "rpc worker-8037" 0x00007f8058e1dad3 in ?? ()
29 LWP 8038 "rpc worker-8038" 0x00007f8058e1dad3 in ?? ()
30 LWP 8039 "rpc worker-8039" 0x00007f8058e1dad3 in ?? ()
31 LWP 8041 "rpc worker-8041" 0x00007f8058e1dad3 in ?? ()
32 LWP 8042 "rpc worker-8042" 0x00007f8058e1dad3 in ?? ()
33 LWP 8044 "rpc worker-8044" 0x00007f8058e1dad3 in ?? ()
34 LWP 8045 "rpc worker-8045" 0x00007f8058e1dad3 in ?? ()
I20260807 22:43:15.087954 8552 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader d6aee0f79fef46c5a7ba0d9714dc0a5e)
35 LWP 8046 "rpc worker-8046" 0x00007f8058e1dad3 in ?? ()
36 LWP 8047 "rpc worker-8047" 0x00007f8058e1dad3 in ?? ()
37 LWP 8049 "rpc worker-8049" 0x00007f8058e1dad3 in ?? ()
I20260807 22:43:15.088233 8552 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
38 LWP 8051 "rpc worker-8051" 0x00007f8058e1dad3 in ?? ()
39 LWP 8052 "rpc worker-8052" 0x00007f8058e1dad3 in ?? ()
40 LWP 8053 "rpc worker-8053" 0x00007f8058e1dad3 in ?? ()
41 LWP 8054 "rpc worker-8054" 0x00007f8058e1dad3 in ?? ()
42 LWP 8056 "rpc worker-8056" 0x00007f8058e1dad3 in ?? ()
43 LWP 8057 "rpc worker-8057" 0x00007f8058e1dad3 in ?? ()
44 LWP 8058 "rpc worker-8058" 0x00007f8058e1dad3 in ?? ()
I20260807 22:43:15.089237 8552 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
45 LWP 8060 "rpc worker-8060" 0x00007f8058e1dad3 in ?? ()
46 LWP 8062 "rpc worker-8062" 0x00007f8058e1dad3 in ?? ()
47 LWP 8063 "rpc worker-8063" 0x00007f8058e1dad3 in ?? ()
48 LWP 8064 "rpc worker-8064" 0x00007f8058e1dad3 in ?? ()
49 LWP 8066 "rpc worker-8066" 0x00007f8058e1dad3 in ?? ()
I20260807 22:43:15.089972 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
50 LWP 8067 "rpc worker-8067" 0x00007f8058e1dad3 in ?? ()
51 LWP 8068 "rpc worker-8068" 0x00007f8058e1dad3 in ?? ()
I20260807 22:43:15.090297 7872 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 17.
52 LWP 8069 "rpc worker-8069" 0x00007f8058e1dad3 in ?? ()
53 LWP 8070 "rpc worker-8070" 0x00007f8058e1dad3 in ?? ()
54 LWP 8073 "rpc worker-8073" 0x00007f8058e1dad3 in ?? ()
55 LWP 8076 "rpc worker-8076" 0x00007f8058e1dad3 in ?? ()
I20260807 22:43:15.090832 8256 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af; no voters:
56 LWP 8078 "rpc worker-8078" 0x00007f8058e1dad3 in ?? ()
57 LWP 8080 "rpc worker-8080" 0x00007f8058e1dad3 in ?? ()
I20260807 22:43:15.091149 8552 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 17 FOLLOWER]: Leader pre-election won for term 18
58 LWP 8081 "rpc worker-8081" 0x00007f8058e1dad3 in ?? ()
I20260807 22:43:15.091280 8552 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 17 FOLLOWER]: Starting leader election (detected failure of leader d6aee0f79fef46c5a7ba0d9714dc0a5e)
59 LWP 8084 "rpc worker-8084" 0x00007f8058e1dad3 in ?? ()
I20260807 22:43:15.091401 8552 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 17 FOLLOWER]: Advancing to term 18
60 LWP 8085 "rpc worker-8085" 0x00007f8058e1dad3 in ?? ()
61 LWP 8086 "rpc worker-8086" 0x00007f8058e1dad3 in ?? ()
62 LWP 8087 "rpc worker-8087" 0x00007f8058e1dad3 in ?? ()
63 LWP 8088 "rpc worker-8088" 0x00007f8058e1dad3 in ?? ()
64 LWP 8089 "rpc worker-8089" 0x00007f8058e1dad3 in ?? ()
65 LWP 8090 "rpc worker-8090" 0x00007f8058e1dad3 in ?? ()
66 LWP 8092 "rpc worker-8092" 0x00007f8058e1dad3 in ?? ()
67 LWP 8093 "rpc worker-8093" 0x00007f8058e1dad3 in ?? ()
68 LWP 8095 "rpc worker-8095" 0x00007f8058e1dad3 in ?? ()
69 LWP 8097 "rpc worker-8097" 0x00007f8058e1dad3 in ?? ()
70 LWP 8098 "rpc worker-8098" 0x00007f8058e1dad3 in ?? ()
71 LWP 8101 "rpc worker-8101" 0x00007f8058e1dad3 in ?? ()
72 LWP 8103 "rpc worker-8103" 0x00007f8058e1dad3 in ?? ()
73 LWP 8104 "rpc worker-8104" 0x00007f8058e1dad3 in ?? ()
74 LWP 8106 "rpc worker-8106" 0x00007f8058e1dad3 in ?? ()
75 LWP 8107 "rpc worker-8107" 0x00007f8058e1dad3 in ?? ()
76 LWP 8109 "rpc worker-8109" 0x00007f8058e1dad3 in ?? ()
77 LWP 8110 "rpc worker-8110" 0x00007f8058e1dad3 in ?? ()
78 LWP 8111 "rpc worker-8111" 0x00007f8058e1dad3 in ?? ()
79 LWP 8112 "rpc worker-8112" 0x00007f8058e1dad3 in ?? ()
80 LWP 8113 "rpc worker-8113" 0x00007f8058e1dad3 in ?? ()
81 LWP 8114 "rpc worker-8114" 0x00007f8058e1dad3 in ?? ()
82 LWP 8115 "rpc worker-8115" 0x00007f8058e1dad3 in ?? ()
83 LWP 8116 "rpc worker-8116" 0x00007f8058e1dad3 in ?? ()
84 LWP 8118 "rpc worker-8118" 0x00007f8058e1dad3 in ?? ()
85 LWP 8119 "rpc worker-8119" 0x00007f8058e1dad3 in ?? ()
86 LWP 8120 "rpc worker-8120" 0x00007f8058e1dad3 in ?? ()
87 LWP 8121 "rpc worker-8121" 0x00007f8058e1dad3 in ?? ()
I20260807 22:43:15.095044 8552 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
88 LWP 8124 "rpc worker-8124" 0x00007f8058e1dad3 in ?? ()
89 LWP 8125 "rpc worker-8125" 0x00007f8058e1dad3 in ?? ()
90 LWP 8126 "rpc worker-8126" 0x00007f8058e1dad3 in ?? ()
91 LWP 8127 "rpc worker-8127" 0x00007f8058e1dad3 in ?? ()
92 LWP 8130 "rpc worker-8130" 0x00007f8058e1dad3 in ?? ()
I20260807 22:43:15.095947 8552 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 18 election: Requested vote from peers 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211), d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3:42621)
93 LWP 8131 "rpc worker-8131" 0x00007f8058e1dad3 in ?? ()
94 LWP 8134 "rpc worker-8134" 0x00007f8058e1dad3 in ?? ()
95 LWP 8135 "rpc worker-8135" 0x00007f8058e1dad3 in ?? ()
96 LWP 8137 "rpc worker-8137" 0x00007f8058e1dad3 in ?? ()
97 LWP 8139 "rpc worker-8139" 0x00007f8058e1dad3 in ?? ()
I20260807 22:43:15.096581 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af"
98 LWP 8142 "rpc worker-8142" 0x00007f8058e1dad3 in ?? ()
99 LWP 8143 "rpc worker-8143" 0x00007f8058e1dad3 in ?? ()
I20260807 22:43:15.096894 7872 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 17 FOLLOWER]: Advancing to term 18
100 LWP 8145 "rpc worker-8145" 0x00007f8058e1dad3 in ?? ()
101 LWP 8146 "rpc worker-8146" 0x00007f8058e1dad3 in ?? ()
102 LWP 8147 "rpc worker-8147" 0x00007f8058e1dad3 in ?? ()
103 LWP 8148 "rpc worker-8148" 0x00007f8058e1dad3 in ?? ()
104 LWP 8150 "rpc worker-8150" 0x00007f8058e1dad3 in ?? ()
105 LWP 8151 "rpc worker-8151" 0x00007f8058e1dad3 in ?? ()
106 LWP 8152 "rpc worker-8152" 0x00007f8058e1dad3 in ?? ()
107 LWP 8153 "rpc worker-8153" 0x00007f8058e1dad3 in ?? ()
108 LWP 8154 "rpc worker-8154" 0x00007f8058e1dad3 in ?? ()
109 LWP 8155 "rpc worker-8155" 0x00007f8058e1dad3 in ?? ()
110 LWP 8156 "rpc worker-8156" 0x00007f8058e1dad3 in ?? ()
111 LWP 8158 "rpc worker-8158" 0x00007f8058e1dad3 in ?? ()
112 LWP 8159 "rpc worker-8159" 0x00007f8058e1dad3 in ?? ()
113 LWP 8161 "rpc worker-8161" 0x00007f8058e1dad3 in ?? ()
114 LWP 8162 "rpc worker-8162" 0x00007f8058e1dad3 in ?? ()
115 LWP 8164 "rpc worker-8164" 0x00007f8058e1dad3 in ?? ()
116 LWP 8165 "rpc worker-8165" 0x00007f8058e1dad3 in ?? ()
117 LWP 8166 "rpc worker-8166" 0x00007f8058e1dad3 in ?? ()
118 LWP 8168 "rpc worker-8168" 0x00007f8058e1dad3 in ?? ()
119 LWP 8169 "rpc worker-8169" 0x00007f8058e1dad3 in ?? ()
120 LWP 8170 "rpc worker-8170" 0x00007f8058e1dad3 in ?? ()
121 LWP 8171 "diag-logger-817" 0x00007f8058e1dfb9 in ?? ()
122 LWP 8172 "result-tracker-" 0x00007f8058e1dfb9 in ?? ()
123 LWP 8175 "excess-log-dele" 0x00007f8058e1dfb9 in ?? ()
124 LWP 8176 "acceptor-8176" 0x00007f8051daefc7 in ?? ()
I20260807 22:43:15.100397 7872 raft_consensus.cc:2454] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 672a15bd3cdb41c29eb4863f63479a24 in term 18.
125 LWP 8178 "heartbeat-8178" 0x00007f8058e1dfb9 in ?? ()
126 LWP 8181 "maintenance_sch" 0x00007f8058e1dfb9 in ?? ()
127 LWP 8465 "raft [worker]-8" 0x00007f8058e1dfb9 in ?? ()
128 LWP 8544 "raft [worker]-8" 0x00007f8058e1dfb9 in ?? ()
I20260807 22:43:15.101006 8256 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af; no voters:
I20260807 22:43:15.101431 8552 raft_consensus.cc:2781] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 18 FOLLOWER]: Leader election won for term 18
Thread 128 (LWP 8544):
#0 0x00007f8058e1dfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000000 in ?? ()
I20260807 22:43:15.102068 8552 raft_consensus.cc:686] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 18 LEADER]: Becoming Leader. State: Replica: 672a15bd3cdb41c29eb4863f63479a24, State: Running, Role: LEADER
Thread 127 (LWP 8465):
#0 0x00007f8058e1dfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x00000000000000c2 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x00007f8007708ba0 in ?? ()
#5 0x00007f8007708830 in ?? ()
#6 0x0000000000000184 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 126 (LWP 8181):
#0 0x00007f8058e1dfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 125 (LWP 8178):
#0 0x00007f8058e1dfb9 in ?? ()
I20260807 22:43:15.102825 8552 consensus_queue.cc:237] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
#1 0x4008000000000000 in ?? ()
#2 0x0000000000000009 in ?? ()
#3 0x0000000100000081 in ?? ()
#4 0x0000613000017d2c in ?? ()
#5 0x00007f800a7d8770 in ?? ()
#6 0x0000000000000013 in ?? ()
#7 0x0000000000000001 in ?? ()
#8 0x00007f800a7d8790 in ?? ()
#9 0x0000000200000000 in ?? ()
#10 0x0000008000000189 in ?? ()
#11 0x00007f800a7d8810 in ?? ()
#12 0x00000ff0014fb104 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 124 (LWP 8176):
#0 0x00007f8051daefc7 in ?? ()
#1 0x0000614000009490 in ?? ()
#2 0x00007f800affffa0 in ?? ()
#3 0x00007f800affffc0 in ?? ()
#4 0x00007f800b0000e0 in ?? ()
#5 0x00007f800affffd0 in ?? ()
#6 0x00007f800b000040 in ?? ()
#7 0x0000000000000080 in ?? ()
#8 0x000000000082c9d2 in __sanitizer::theDepot ()
#9 0x00000000008c3cc9 in __sanitizer::theDepot ()
#10 0x00000000008f35b0 in __sanitizer::theDepot ()
#11 0x0000000000000032 in ?? ()
#12 0x000060c00007ffd8 in ?? ()
#13 0x000060c000080032 in ?? ()
#14 0x00000000008875e1 in __sanitizer::theDepot ()
#15 0x000000000000005a in ?? ()
#16 0x000000000088348c in __sanitizer::theDepot ()
#17 0x0000000000000380 in ?? ()
#18 0x0000000000897749 in __sanitizer::theDepot ()
#19 0x0000000000010000 in ?? ()
#20 0x0000000000895799 in __sanitizer::theDepot ()
#21 0x0000000000e377a0 in ?? ()
#22 0x00000000007f9413 in __sanitizer::theDepot ()
#23 0x00000000009edb7d in __sanitizer::theDepot ()
#24 0x0000000000000700 in ?? ()
#25 0x00007f8058e176db in ?? ()
#26 0x00007f8051d5f346 in ?? ()
#27 0x00007f800afff842 in ?? ()
#28 0x000000006a765f7d in ?? ()
#29 0x000000006a765f7d in ?? ()
#30 0x0000640000001140 in ?? ()
#31 0x00007f800a7f9e60 in ?? ()
#32 0x0000000000000049 in ?? ()
#33 0x0000017000000000 in ?? ()
#34 0x0000000000000017 in ?? ()
#35 0x0000640000001140 in ?? ()
#36 0x00000000008b1acd in __sanitizer::theDepot ()
#37 0x0000000000e377a0 in ?? ()
#38 0x00007f8008f74de8 in ?? ()
#39 0x0000000000000000 in ?? ()
Thread 123 (LWP 8175):
#0 0x00007f8058e1dfb9 in ?? ()
#1 0x00007ffdbf613d50 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 122 (LWP 8172):
#0 0x00007f8058e1dfb9 in ?? ()
I20260807 22:43:15.106557 8553 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 18 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
#1 0x00007f800c031160 in ?? ()
#2 0x0000000000000005 in ?? ()
#3 0x0000000000000081 in ?? ()
I20260807 22:43:15.107048 8553 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
#4 0x000061100006ec30 in ?? ()
#5 0x00007f800c031110 in ?? ()
#6 0x000000000000000a in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 121 (LWP 8171):
#0 0x00007f8058e1dfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
I20260807 22:43:15.107714 8555 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:43:15.107905 8555 txn_status_manager.cc:929] Loading transaction status metadata into memory...
Thread 120 (LWP 8170):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 119 (LWP 8169):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 118 (LWP 8168):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
I20260807 22:43:15.108150 6341 catalog_manager.cc:5495] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 reported cstate change: term changed from 17 to 18, leader changed from d6aee0f79fef46c5a7ba0d9714dc0a5e (127.4.103.3) to 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2). New cstate: current_term: 18 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } health_report { overall_health: UNKNOWN } } }
Thread 117 (LWP 8166):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 116 (LWP 8165):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 115 (LWP 8164):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 114 (LWP 8162):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 113 (LWP 8161):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 112 (LWP 8159):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 111 (LWP 8158):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 110 (LWP 8156):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 109 (LWP 8155):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 108 (LWP 8154):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 107 (LWP 8153):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 106 (LWP 8152):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 105 (LWP 8151):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 104 (LWP 8150):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 103 (LWP 8148):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 102 (LWP 8147):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 101 (LWP 8146):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 100 (LWP 8145):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 99 (LWP 8143):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 98 (LWP 8142):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 97 (LWP 8139):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 96 (LWP 8137):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 95 (LWP 8135):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 94 (LWP 8134):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 93 (LWP 8131):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 92 (LWP 8130):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 91 (LWP 8127):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 90 (LWP 8126):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 89 (LWP 8125):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 88 (LWP 8124):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 87 (LWP 8121):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 86 (LWP 8120):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 85 (LWP 8119):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 84 (LWP 8118):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 83 (LWP 8116):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 82 (LWP 8115):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
I20260807 22:43:15.118527 8555 txn_status_manager.cc:727] Starting 1 aborts task
Thread 81 (LWP 8114):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 80 (LWP 8113):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000042 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d0001d6858 in ?? ()
#4 0x00007f8021432eb0 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f8021432ed0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 79 (LWP 8112):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 78 (LWP 8111):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 77 (LWP 8110):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 76 (LWP 8109):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 75 (LWP 8107):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 74 (LWP 8106):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 73 (LWP 8104):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 72 (LWP 8103):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 71 (LWP 8101):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 70 (LWP 8098):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 69 (LWP 8097):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 68 (LWP 8095):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 67 (LWP 8093):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 66 (LWP 8092):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 65 (LWP 8090):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 64 (LWP 8089):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 63 (LWP 8088):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 62 (LWP 8087):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 61 (LWP 8086):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 60 (LWP 8085):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000002 in ?? ()
#2 0x0000000000000081 in ?? ()
#3 0x000060d000150008 in ?? ()
#4 0x00007f802b613eb0 in ?? ()
#5 0x0000008000000000 in ?? ()
#6 0x00007f802b613ed0 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 59 (LWP 8084):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 58 (LWP 8081):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 57 (LWP 8080):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 56 (LWP 8078):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 55 (LWP 8076):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 54 (LWP 8073):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 53 (LWP 8070):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 52 (LWP 8069):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 51 (LWP 8068):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 50 (LWP 8067):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 49 (LWP 8066):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 48 (LWP 8064):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 47 (LWP 8063):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 46 (LWP 8062):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 45 (LWP 8060):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 44 (LWP 8058):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 43 (LWP 8057):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 42 (LWP 8056):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 41 (LWP 8054):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 40 (LWP 8053):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 39 (LWP 8052):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 38 (LWP 8051):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 37 (LWP 8049):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 36 (LWP 8047):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 35 (LWP 8046):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 34 (LWP 8045):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 33 (LWP 8044):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 32 (LWP 8042):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 31 (LWP 8041):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 30 (LWP 8039):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 29 (LWP 8038):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 28 (LWP 8037):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 27 (LWP 8036):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 26 (LWP 8035):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 25 (LWP 8033):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 24 (LWP 8032):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 23 (LWP 8031):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 22 (LWP 8029):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 21 (LWP 8028):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 20 (LWP 8023):
#0 0x00007f8058e1dfb9 in ?? ()
#1 0x00000000ffffffff in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 19 (LWP 8022):
#0 0x00007f8051dad947 in ?? ()
#1 0x0000614000050880 in ?? ()
#2 0x0000619000090b80 in ?? ()
#3 0x00007f8041393340 in ?? ()
#4 0x00007f8041393540 in ?? ()
#5 0x00007f8041393390 in ?? ()
#6 0x0000614000050880 in ?? ()
#7 0x00007f8041393400 in ?? ()
#8 0x00007f80543b4bda in ?? ()
#9 0x3fb959f1f2000000 in ?? ()
#10 0x0000619000090b80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000090b80 in ?? ()
#13 0x00007f805c5c9340 in ?? ()
#14 0x00007f8041393540 in ?? ()
#15 0x41da9797a7133374 in ?? ()
#16 0x000000008826a680 in ?? ()
#17 0x00007f80413933e0 in ?? ()
#18 0x00007f80543b8228 in ?? ()
#19 0x00000000413933c0 in ?? ()
#20 0x0000619000090b80 in ?? ()
#21 0x0000614000050880 in ?? ()
#22 0x00007f8041393400 in ?? ()
#23 0x3fb959f1f2000000 in ?? ()
#24 0x413800e6afc14578 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 18 (LWP 8021):
#0 0x00007f8051dad947 in ?? ()
#1 0x0000614000050680 in ?? ()
#2 0x0000619000090680 in ?? ()
#3 0x00007f8041bab340 in ?? ()
#4 0x00007f8041bab540 in ?? ()
#5 0x00007f8041bab390 in ?? ()
#6 0x0000614000050680 in ?? ()
#7 0x00007f8041bab400 in ?? ()
#8 0x00007f80543b4bda in ?? ()
#9 0x3fb981a424000000 in ?? ()
#10 0x0000619000090680 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000090680 in ?? ()
#13 0x00007f805c5c9340 in ?? ()
#14 0x00007f8041bab540 in ?? ()
#15 0x41da9797a7133373 in ?? ()
#16 0x000000008836d680 in ?? ()
#17 0x00007f8041bab3e0 in ?? ()
#18 0x00007f80543b8228 in ?? ()
#19 0x0000000041bab3c0 in ?? ()
#20 0x0000619000090680 in ?? ()
#21 0x0000614000050680 in ?? ()
#22 0x00007f8041bab400 in ?? ()
#23 0x3fb981a424000000 in ?? ()
#24 0x413800e6af98e4cf in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 17 (LWP 8020):
#0 0x00007f8051dad947 in ?? ()
#1 0x0000614000050480 in ?? ()
#2 0x0000619000090180 in ?? ()
#3 0x00007f80423d2340 in ?? ()
#4 0x00007f80423d2540 in ?? ()
#5 0x00007f80423d2390 in ?? ()
#6 0x0000614000050480 in ?? ()
#7 0x00007f80423d2400 in ?? ()
#8 0x00007f80543b4bda in ?? ()
#9 0x3fb9804099000000 in ?? ()
#10 0x0000619000090180 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000090180 in ?? ()
#13 0x00007f805c5c9340 in ?? ()
#14 0x00007f80423d2540 in ?? ()
#15 0x41da9797a7133376 in ?? ()
#16 0x0000000088472480 in ?? ()
#17 0x00007f80423d23e0 in ?? ()
#18 0x00007f80543b8228 in ?? ()
#19 0x00000000423d23c0 in ?? ()
#20 0x0000619000090180 in ?? ()
#21 0x0000614000050480 in ?? ()
#22 0x00007f80423d2400 in ?? ()
#23 0x3fb9804099000000 in ?? ()
#24 0x413800e6af98a4df in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 16 (LWP 8019):
#0 0x00007f8051dad947 in ?? ()
#1 0x0000614000050280 in ?? ()
#2 0x000061900008fc80 in ?? ()
#3 0x00007f8042bdb340 in ?? ()
#4 0x00007f8042bdb540 in ?? ()
#5 0x00007f8042bdb390 in ?? ()
#6 0x0000614000050280 in ?? ()
#7 0x00007f8042bdb400 in ?? ()
#8 0x00007f80543b4bda in ?? ()
#9 0x3fb980060f000000 in ?? ()
#10 0x000061900008fc80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x000061900008fc80 in ?? ()
#13 0x00007f805c5c9340 in ?? ()
#14 0x00007f8042bdb540 in ?? ()
#15 0x41da9797a7133374 in ?? ()
#16 0x0000000088573680 in ?? ()
#17 0x00007f8042bdb3e0 in ?? ()
#18 0x00007f80543b8228 in ?? ()
#19 0x0000000042bdb3c0 in ?? ()
#20 0x000061900008fc80 in ?? ()
#21 0x0000614000050280 in ?? ()
#22 0x00007f8042bdb400 in ?? ()
#23 0x3fb980060f000000 in ?? ()
#24 0x413800e6af9311b8 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 15 (LWP 8018):
#0 0x00007f8058e1dfb9 in ?? ()
#1 0x4008000000000000 in ?? ()
#2 0x0000000000000001 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001a618 in ?? ()
#5 0x00007f80433dc270 in ?? ()
#6 0x0000000000000002 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 14 (LWP 8011):
#0 0x00007f8058e1dfb9 in ?? ()
#1 0x0000000045e0360e in ?? ()
#2 0x0000000000000038 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000618000004da8 in ?? ()
#5 0x00007f8044bde250 in ?? ()
#6 0x0000000000000070 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 13 (LWP 8009):
#0 0x00007f8058e1dad3 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 12 (LWP 8008):
#0 0x00007f8051dad947 in ?? ()
#1 0x0000614000009a80 in ?? ()
#2 0x0000619000059680 in ?? ()
#3 0x00007f80463e2340 in ?? ()
#4 0x00007f80463e2540 in ?? ()
#5 0x00007f80463e2390 in ?? ()
#6 0x0000614000009a80 in ?? ()
#7 0x00007f80463e2400 in ?? ()
#8 0x00007f80543b4bda in ?? ()
#9 0x3f767f25c0000000 in ?? ()
#10 0x0000619000059680 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059680 in ?? ()
#13 0x420e043b16000000 in ?? ()
#14 0x00007f80463e2540 in ?? ()
#15 0x41da9797a7133374 in ?? ()
#16 0x0000000088c74480 in ?? ()
#17 0x00007f80463e23e0 in ?? ()
#18 0x00007f80543b8228 in ?? ()
#19 0x00000000463e23c0 in ?? ()
#20 0x0000619000059680 in ?? ()
#21 0x0000614000009a80 in ?? ()
#22 0x00007f80463e2400 in ?? ()
#23 0x3f767f25c0000000 in ?? ()
#24 0x413800e6c29d9ac6 in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 11 (LWP 8007):
#0 0x00007f8051dad947 in ?? ()
#1 0x0000614000009880 in ?? ()
#2 0x0000619000059180 in ?? ()
#3 0x00007f8046be3340 in ?? ()
#4 0x00007f8046be3540 in ?? ()
#5 0x00007f8046be3390 in ?? ()
#6 0x0000614000009880 in ?? ()
I20260807 22:43:15.146711 7872 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 18 FOLLOWER]: Refusing update from remote peer 672a15bd3cdb41c29eb4863f63479a24: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
#7 0x00007f8046be3400 in ?? ()
#8 0x00007f80543b4bda in ?? ()
#9 0x3fb9901d0e000000 in ?? ()
#10 0x0000619000059180 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000059180 in ?? ()
#13 0x4229aad0e8000000 in ?? ()
#14 0x00007f8046be3540 in ?? ()
#15 0x41da9797a7133373 in ?? ()
#16 0x0000000088d74680 in ?? ()
#17 0x00007f8046be33e0 in ?? ()
#18 0x00007f80543b8228 in ?? ()
#19 0x0000000046be33c0 in ?? ()
#20 0x0000619000059180 in ?? ()
#21 0x0000614000009880 in ?? ()
#22 0x00007f8046be3400 in ?? ()
#23 0x3fb9901d0e000000 in ?? ()
#24 0x413800e6bf445dc2 in ?? ()
#25 0x0000000000000000 in ?? ()
I20260807 22:43:15.147770 8553 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
Thread 10 (LWP 8006):
#0 0x00007f8051dad947 in ?? ()
#1 0x0000614000009680 in ?? ()
#2 0x0000619000058c80 in ?? ()
#3 0x00007f80473e4340 in ?? ()
#4 0x00007f80473e4540 in ?? ()
#5 0x00007f80473e4390 in ?? ()
#6 0x0000614000009680 in ?? ()
#7 0x00007f80473e4400 in ?? ()
#8 0x00007f80543b4bda in ?? ()
#9 0x3f92d816d8000000 in ?? ()
#10 0x0000619000058c80 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058c80 in ?? ()
#13 0x42186f6365000000 in ?? ()
#14 0x00007f80473e4540 in ?? ()
#15 0x41da9797a7133373 in ?? ()
#16 0x0000000088e74880 in ?? ()
#17 0x00007f80473e43e0 in ?? ()
#18 0x00007f80543b8228 in ?? ()
#19 0x00000000473e43c0 in ?? ()
#20 0x0000619000058c80 in ?? ()
#21 0x0000614000009680 in ?? ()
#22 0x00007f80473e4400 in ?? ()
#23 0x3f92d816d8000000 in ?? ()
#24 0x413800e6c16078aa in ?? ()
#25 0x0000000000000000 in ?? ()
Thread 9 (LWP 8005):
#0 0x00007f8051dad947 in ?? ()
#1 0x0000614000009480 in ?? ()
#2 0x0000619000058780 in ?? ()
#3 0x00007f80489ad340 in ?? ()
#4 0x00007f80489ad540 in ?? ()
#5 0x00007f80489ad390 in ?? ()
#6 0x0000614000009480 in ?? ()
I20260807 22:43:15.150611 8551 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 18 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
#7 0x00007f80489ad400 in ?? ()
I20260807 22:43:15.151083 8551 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
#8 0x00007f80543b4bda in ?? ()
#9 0x3fb97566c3000000 in ?? ()
#10 0x0000619000058780 in ?? ()
#11 0x54b249ad2594c37d in ?? ()
#12 0x0000619000058780 in ?? ()
#13 0x42307f497fc00000 in ?? ()
#14 0x00007f80489ad540 in ?? ()
#15 0x41da9797a7133374 in ?? ()
#16 0x000000008912da80 in ?? ()
#17 0x00007f80489ad3e0 in ?? ()
#18 0x00007f80543b8228 in ?? ()
#19 0x0000000000000000 in ?? ()
Thread 8 (LWP 8001):
#0 0x00007f8051da0bb9 in ?? ()
#1 0x00000000000000c8 in ?? ()
#2 0x00007f804a3b77d8 in ?? ()
#3 0x000060200004d5f0 in ?? ()
#4 0x0000000000000002 in ?? ()
#5 0x00000000000000c8 in ?? ()
#6 0x0000000000823a92 in __sanitizer::theDepot ()
#7 0x00007f804a3b7800 in ?? ()
#8 0x0000000000000000 in ?? ()
Thread 7 (LWP 8000):
#0 0x00007f8058e1dfb9 in ?? ()
#1 0x00007f8049bb6140 in ?? ()
#2 0x0000000000000000 in ?? ()
Thread 6 (LWP 7999):
#0 0x00007f8058e219e2 in ?? ()
#1 0x00007f80493b4430 in ?? ()
#2 0x00007f80493b3f20 in ?? ()
W20260807 22:43:15.153030 7770 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.103.3:42621 timed out after 0.068s (SENT) [suppressed 18 similar messages]
#3 0x00007f80493b3650 in ?? ()
#4 0x00007f80493b51c0 in ?? ()
#5 0x00007f80493b3ed0 in ?? ()
#6 0x0000000000000030 in ?? ()
#7 0x00007f80493b3f30 in ?? ()
#8 0x000061100005a2d0 in ?? ()
#9 0x00007f80493b3f20 in ?? ()
#10 0x000000000081de90 in __sanitizer::theDepot ()
#11 0x00007f805e5e5269 in ?? ()
#12 0x00000000009edb7d in __sanitizer::theDepot ()
#13 0x00007f8055b3deaf in ?? ()
#14 0x00007f8058e176db in ?? ()
#15 0x3a73250000000000 in ?? ()
#16 0x642520656e696c20 in ?? ()
#17 0x0000000000000000 in ?? ()
Thread 5 (LWP 7989):
#0 0x00007f8058e1dfb9 in ?? ()
#1 0x00007f804b3b90a0 in ?? ()
#2 0x000000000000001c in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x000061200001ab28 in ?? ()
#5 0x00007f804b3b8cd0 in ?? ()
#6 0x0000000000000038 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 4 (LWP 7988):
#0 0x00007f8058e1dfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 3 (LWP 7986):
#0 0x00007f8058e1dfb9 in ?? ()
#1 0x0000000000000000 in ?? ()
Thread 2 (LWP 7985):
#0 0x00007f8058e1dfb9 in ?? ()
#1 0x0000000000008000 in ?? ()
#2 0x0000000000000002 in ?? ()
#3 0x0000000000000081 in ?? ()
#4 0x0000612000009838 in ?? ()
#5 0x00007f804cbbc450 in ?? ()
#6 0x0000000000000004 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 1 (LWP 7984):
#0 0x00007f8058e21d50 in ?? ()
#1 0x0000000000000000 in ?? ()
I20260807 22:43:15.158702 8113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:15.159351 8112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:15.159950 8110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "753b63233367448f8f28e3a1719be1af" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" is_pre_election: true
I20260807 22:43:15.160197 8109 raft_consensus.cc:3032] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 17 LEADER]: Stepping down as leader of term 17
I20260807 22:43:15.160362 8109 raft_consensus.cc:729] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d6aee0f79fef46c5a7ba0d9714dc0a5e, State: Running, Role: LEADER
I20260807 22:43:15.160631 7770 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 753b63233367448f8f28e3a1719be1af; no voters: 672a15bd3cdb41c29eb4863f63479a24, d6aee0f79fef46c5a7ba0d9714dc0a5e
I20260807 22:43:15.160799 8109 consensus_queue.cc:260] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:15.161181 8109 raft_consensus.cc:3037] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 17 FOLLOWER]: Advancing to term 18
W20260807 22:43:15.162832 8259 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.103.3:42621 timed out after 0.065s (SENT)
I20260807 22:43:15.164773 8111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:43:15.167274 8109 raft_consensus.cc:1261] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 18 FOLLOWER]: Refusing update from remote peer 672a15bd3cdb41c29eb4863f63479a24: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260807 22:43:15.168790 8553 consensus_queue.cc:1035] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260807 22:43:15.172876 8374 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 [term 18 LEADER]: Rejecting Update request from peer d6aee0f79fef46c5a7ba0d9714dc0a5e for earlier term 17. Current term is 18. Ops: []
I20260807 22:43:15.173184 8111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "672a15bd3cdb41c29eb4863f63479a24" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e"
I20260807 22:43:15.174178 8553 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 18 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:15.174480 8553 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:15.175772 7871 raft_consensus.cc:1226] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 18 FOLLOWER]: Rejecting Update request from peer d6aee0f79fef46c5a7ba0d9714dc0a5e for earlier term 17. Current term is 18. Ops: []
I20260807 22:43:15.176653 8551 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P 753b63233367448f8f28e3a1719be1af [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260807 22:43:15.178438 8573 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 18 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:15.178840 8573 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:15.184978 8465 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 18 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:15.185420 8465 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:15.186708 8544 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 672a15bd3cdb41c29eb4863f63479a24. Latest consensus state: current_term: 18 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:15.187122 8544 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
************************* END STACKS ***************************
I20260807 22:43:15.194398 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 7743
I20260807 22:43:15.196086 7927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:15.231803 8552 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 18 leader_uuid: "672a15bd3cdb41c29eb4863f63479a24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } } }
I20260807 22:43:15.232353 8552 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:15.232777 8544 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:43:15.233018 8544 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:15.233865 8544 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
I20260807 22:43:15.234935 8374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
I20260807 22:43:15.348590 7872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:15.348642 7871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "753b63233367448f8f28e3a1719be1af" is_pre_election: true
I20260807 22:43:15.349263 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 7743
I20260807 22:43:15.350931 8005 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af
I20260807 22:43:15.351539 8544 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260807 22:43:15.391424 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 8229
W20260807 22:43:15.393744 8256 consensus_peers.cc:487] T c5fd68d798cc4cc68c2db34d14074bd4 P 672a15bd3cdb41c29eb4863f63479a24 -> Peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Couldn't send request to peer 753b63233367448f8f28e3a1719be1af. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.1:43211: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 22:43:15.409947 8444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:15.433696 8544 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:43:15.434056 8544 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:15.434953 8544 leader_election.cc:288] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723), 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211)
W20260807 22:43:15.437956 8005 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 753b63233367448f8f28e3a1719be1af (127.4.103.1:43211): Network error: Client connection negotiation failed: client connection to 127.4.103.1:43211: connect: Connection refused (error 111)
I20260807 22:43:15.567550 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 8229
I20260807 22:43:15.567459 8374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5fd68d798cc4cc68c2db34d14074bd4" candidate_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "672a15bd3cdb41c29eb4863f63479a24" is_pre_election: true
W20260807 22:43:15.574874 8005 leader_election.cc:334] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 672a15bd3cdb41c29eb4863f63479a24 (127.4.103.2:38723): Timed out: RequestConsensusVote RPC to 127.4.103.2:38723 timed out after 0.069s (SENT)
I20260807 22:43:15.575201 8005 leader_election.cc:302] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6aee0f79fef46c5a7ba0d9714dc0a5e; no voters: 672a15bd3cdb41c29eb4863f63479a24, 753b63233367448f8f28e3a1719be1af
I20260807 22:43:15.575783 8544 raft_consensus.cc:2726] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260807 22:43:15.601506 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 7984
I20260807 22:43:15.610930 8170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:15.749516 8544 raft_consensus.cc:482] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 672a15bd3cdb41c29eb4863f63479a24)
I20260807 22:43:15.749872 8544 raft_consensus.cc:504] T c5fd68d798cc4cc68c2db34d14074bd4 P d6aee0f79fef46c5a7ba0d9714dc0a5e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "672a15bd3cdb41c29eb4863f63479a24" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 38723 } } peers { permanent_uuid: "753b63233367448f8f28e3a1719be1af" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 43211 } } peers { permanent_uuid: "d6aee0f79fef46c5a7ba0d9714dc0a5e" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 42621 } }
I20260807 22:43:15.750602 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 7984
I20260807 22:43:15.796849 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 6294
I20260807 22:43:15.798393 6396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:15.945061 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 6294
2026-08-07T22:43:15Z chronyd exiting
I20260807 22:43:16.007300 4508 test_util.cc:175] -----------------------------------------------
I20260807 22:43:16.007516 4508 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances.1786142554140270-4508-0
[ FAILED ] TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances (26839 ms)
[ RUN ] TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted
2026-08-07T22:43:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T22:43:16Z Disabled control of system clock
I20260807 22:43:16.049523 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.103.62:46193
--webserver_interface=127.4.103.62
--webserver_port=0
--builtin_ntp_servers=127.4.103.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.103.62:46193
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false with env {}
W20260807 22:43:16.470778 8594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:43:16.471206 8594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:43:16.471292 8594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:43:16.479426 8594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 22:43:16.479585 8594 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 22:43:16.479626 8594 flags.cc:425] Enabled experimental flag: --txn_manager_lazily_initialized=false
W20260807 22:43:16.479677 8594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:43:16.479720 8594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 22:43:16.479790 8594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 22:43:16.490448 8594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.103.62:46193
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.103.62:46193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.103.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:43:16.492332 8594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:43:16.494691 8594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:43:16.509617 8603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:16.509644 8604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:16.509675 8606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:43:16.514305 8594 server_base.cc:1034] running on GCE node
I20260807 22:43:16.515482 8594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:43:16.517632 8594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:43:16.518950 8594 hybrid_clock.cc:648] HybridClock initialized: now 1786142596518882 us; error 53 us; skew 500 ppm
I20260807 22:43:16.519366 8594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:43:16.522526 8594 webserver.cc:460] Webserver started at http://127.4.103.62:46163/ using document root <none> and password file <none>
I20260807 22:43:16.523411 8594 fs_manager.cc:359] Metadata directory not provided
I20260807 22:43:16.523541 8594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:43:16.523904 8594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:43:16.527941 8594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/data/instance:
uuid: "d5cb1123849449abbaf882250aa4648f"
format_stamp: "Formatted at 2026-08-07 22:43:16 on dist-test-slave-kvfs"
I20260807 22:43:16.529054 8594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/wal/instance:
uuid: "d5cb1123849449abbaf882250aa4648f"
format_stamp: "Formatted at 2026-08-07 22:43:16 on dist-test-slave-kvfs"
I20260807 22:43:16.536123 8594 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260807 22:43:16.541718 8613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:16.543766 8594 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260807 22:43:16.544130 8594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/wal
uuid: "d5cb1123849449abbaf882250aa4648f"
format_stamp: "Formatted at 2026-08-07 22:43:16 on dist-test-slave-kvfs"
I20260807 22:43:16.544405 8594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:43:16.574692 8594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:43:16.575904 8594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:43:16.576339 8594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:43:16.603608 8594 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.62:46193
I20260807 22:43:16.603632 8703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.62:46193 every 8 connection(s)
I20260807 22:43:16.605525 8594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/data/info.pb
I20260807 22:43:16.610253 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 8594
I20260807 22:43:16.610548 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/master-0/wal/instance
I20260807 22:43:16.610808 8706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:43:16.626789 8706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f: Bootstrap starting.
I20260807 22:43:16.631138 8706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f: Neither blocks nor log segments found. Creating new log.
I20260807 22:43:16.632598 8706 log.cc:824] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f: Log is configured to *not* fsync() on all Append() calls
I20260807 22:43:16.636301 8706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f: No bootstrap required, opened a new log
I20260807 22:43:16.645558 8706 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cb1123849449abbaf882250aa4648f" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 46193 } }
I20260807 22:43:16.646140 8706 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:43:16.646265 8706 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5cb1123849449abbaf882250aa4648f, State: Initialized, Role: FOLLOWER
I20260807 22:43:16.647012 8706 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cb1123849449abbaf882250aa4648f" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 46193 } }
I20260807 22:43:16.647302 8706 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 22:43:16.647476 8706 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 22:43:16.647701 8706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:43:16.650467 8706 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cb1123849449abbaf882250aa4648f" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 46193 } }
I20260807 22:43:16.651139 8706 leader_election.cc:302] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5cb1123849449abbaf882250aa4648f; no voters:
I20260807 22:43:16.652175 8706 leader_election.cc:288] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 22:43:16.652328 8712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:43:16.653630 8712 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [term 1 LEADER]: Becoming Leader. State: Replica: d5cb1123849449abbaf882250aa4648f, State: Running, Role: LEADER
I20260807 22:43:16.654392 8712 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cb1123849449abbaf882250aa4648f" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 46193 } }
I20260807 22:43:16.655186 8706 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 22:43:16.658066 8714 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5cb1123849449abbaf882250aa4648f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cb1123849449abbaf882250aa4648f" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 46193 } } }
I20260807 22:43:16.658479 8714 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [sys.catalog]: This master's current role is: LEADER
I20260807 22:43:16.659265 8715 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5cb1123849449abbaf882250aa4648f. Latest consensus state: current_term: 1 leader_uuid: "d5cb1123849449abbaf882250aa4648f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5cb1123849449abbaf882250aa4648f" member_type: VOTER last_known_addr { host: "127.4.103.62" port: 46193 } } }
I20260807 22:43:16.659660 8715 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f [sys.catalog]: This master's current role is: LEADER
I20260807 22:43:16.661002 8727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 22:43:16.668220 8727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 22:43:16.683308 8727 catalog_manager.cc:1347] Generated new cluster ID: e379e2a7375044bbb6ed1b6864cb886c
I20260807 22:43:16.683495 8727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 22:43:16.721594 8727 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 22:43:16.723436 8727 catalog_manager.cc:1504] Loading token signing keys...
I20260807 22:43:16.731010 8727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5cb1123849449abbaf882250aa4648f: Generated new TSK 0
I20260807 22:43:16.732239 8727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 22:43:16.749145 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.1:0
--local_ip_for_outbound_sockets=127.4.103.1
--webserver_interface=127.4.103.1
--webserver_port=0
--tserver_master_addrs=127.4.103.62:46193
--builtin_ntp_servers=127.4.103.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
I20260807 22:43:16.750528 8640 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43548:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:43:16.765692 8706 master.cc:440] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 3; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20260807 22:43:17.223342 8755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:43:17.223803 8755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:43:17.223992 8755 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:43:17.224153 8755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:43:17.232486 8755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:43:17.232720 8755 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:43:17.232844 8755 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:43:17.232924 8755 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:43:17.233101 8755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.1
I20260807 22:43:17.246601 8755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.103.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:46193
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.1
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:43:17.249229 8755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:43:17.252637 8755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:43:17.270079 8760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:17.270749 8764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:17.270920 8766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:43:17.272423 8755 server_base.cc:1034] running on GCE node
I20260807 22:43:17.273355 8755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:43:17.274672 8755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:43:17.275976 8755 hybrid_clock.cc:648] HybridClock initialized: now 1786142597275898 us; error 48 us; skew 500 ppm
I20260807 22:43:17.276443 8755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:43:17.279500 8755 webserver.cc:460] Webserver started at http://127.4.103.1:34977/ using document root <none> and password file <none>
I20260807 22:43:17.280530 8755 fs_manager.cc:359] Metadata directory not provided
I20260807 22:43:17.280697 8755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:43:17.281098 8755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:43:17.284618 8755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data/instance:
uuid: "fbb7d516f13f438085a3bd7c37805819"
format_stamp: "Formatted at 2026-08-07 22:43:17 on dist-test-slave-kvfs"
I20260807 22:43:17.285579 8755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/wal/instance:
uuid: "fbb7d516f13f438085a3bd7c37805819"
format_stamp: "Formatted at 2026-08-07 22:43:17 on dist-test-slave-kvfs"
I20260807 22:43:17.292212 8755 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260807 22:43:17.296555 8773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:17.298341 8755 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260807 22:43:17.298585 8755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/wal
uuid: "fbb7d516f13f438085a3bd7c37805819"
format_stamp: "Formatted at 2026-08-07 22:43:17 on dist-test-slave-kvfs"
I20260807 22:43:17.298856 8755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:43:17.326829 8755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:43:17.328150 8755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:43:17.328539 8755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:43:17.332707 8755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:43:17.332854 8755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:17.333029 8755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:43:17.333137 8755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:17.392004 8755 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.1:33317
I20260807 22:43:17.392199 8952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.1:33317 every 8 connection(s)
I20260807 22:43:17.393783 8755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
I20260807 22:43:17.396708 8953 heartbeater.cc:347] Connected to a master server at 127.4.103.62:46193
I20260807 22:43:17.397089 8953 heartbeater.cc:464] Registering TS with master...
I20260807 22:43:17.398051 8953 heartbeater.cc:511] Master 127.4.103.62:46193 requested a full tablet report, sending...
I20260807 22:43:17.399387 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 8755
I20260807 22:43:17.399605 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/wal/instance
I20260807 22:43:17.400830 8640 ts_manager.cc:194] Registered new tserver with Master: fbb7d516f13f438085a3bd7c37805819 (127.4.103.1:33317)
I20260807 22:43:17.402782 8640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.1:43351
I20260807 22:43:17.403851 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.2:0
--local_ip_for_outbound_sockets=127.4.103.2
--webserver_interface=127.4.103.2
--webserver_port=0
--tserver_master_addrs=127.4.103.62:46193
--builtin_ntp_servers=127.4.103.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
I20260807 22:43:17.812764 8640 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43552:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:43:17.830168 8955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:43:17.830595 8955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:43:17.830736 8955 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:43:17.830827 8955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:43:17.839201 8955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:43:17.839365 8955 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:43:17.839493 8955 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:43:17.839565 8955 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:43:17.839722 8955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.2
I20260807 22:43:17.850533 8955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.103.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:46193
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.2
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:43:17.852653 8955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:43:17.855425 8955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:43:17.873428 8970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:17.873798 8969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:17.873771 8972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:43:17.875551 8955 server_base.cc:1034] running on GCE node
I20260807 22:43:17.876523 8955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:43:17.877842 8955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:43:17.879110 8955 hybrid_clock.cc:648] HybridClock initialized: now 1786142597879027 us; error 51 us; skew 500 ppm
I20260807 22:43:17.879580 8955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:43:17.882730 8955 webserver.cc:460] Webserver started at http://127.4.103.2:37497/ using document root <none> and password file <none>
I20260807 22:43:17.883698 8955 fs_manager.cc:359] Metadata directory not provided
I20260807 22:43:17.883880 8955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:43:17.884409 8955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:43:17.887800 8955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data/instance:
uuid: "2d63d037cacb4205b689fe6259f9b790"
format_stamp: "Formatted at 2026-08-07 22:43:17 on dist-test-slave-kvfs"
I20260807 22:43:17.888867 8955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/wal/instance:
uuid: "2d63d037cacb4205b689fe6259f9b790"
format_stamp: "Formatted at 2026-08-07 22:43:17 on dist-test-slave-kvfs"
I20260807 22:43:17.895589 8955 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260807 22:43:17.900282 8988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:17.902210 8955 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260807 22:43:17.902473 8955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/wal
uuid: "2d63d037cacb4205b689fe6259f9b790"
format_stamp: "Formatted at 2026-08-07 22:43:17 on dist-test-slave-kvfs"
I20260807 22:43:17.902825 8955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:43:17.927946 8955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:43:17.929235 8955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:43:17.929639 8955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:43:17.933533 8955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:43:17.933713 8955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:17.934026 8955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:43:17.934163 8955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:17.998322 8955 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.2:40639
I20260807 22:43:17.998359 9159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.2:40639 every 8 connection(s)
I20260807 22:43:18.000164 8955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
I20260807 22:43:18.003333 9161 heartbeater.cc:347] Connected to a master server at 127.4.103.62:46193
I20260807 22:43:18.003650 9161 heartbeater.cc:464] Registering TS with master...
I20260807 22:43:18.004750 9161 heartbeater.cc:511] Master 127.4.103.62:46193 requested a full tablet report, sending...
I20260807 22:43:18.006585 8640 ts_manager.cc:194] Registered new tserver with Master: 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639)
I20260807 22:43:18.007401 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 8955
I20260807 22:43:18.007673 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/wal/instance
I20260807 22:43:18.007710 8640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.2:52937
I20260807 22:43:18.011956 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.3:0
--local_ip_for_outbound_sockets=127.4.103.3
--webserver_interface=127.4.103.3
--webserver_port=0
--tserver_master_addrs=127.4.103.62:46193
--builtin_ntp_servers=127.4.103.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
I20260807 22:43:18.407832 8953 heartbeater.cc:503] Master 127.4.103.62:46193 was elected leader, sending a full tablet report...
W20260807 22:43:18.456244 9167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:43:18.456753 9167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:43:18.456933 9167 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:43:18.457063 9167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:43:18.466187 9167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:43:18.466523 9167 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:43:18.466672 9167 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:43:18.466759 9167 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:43:18.466969 9167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.3
I20260807 22:43:18.478696 9167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.103.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:46193
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.3
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:43:18.480688 9167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:43:18.483505 9167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:43:18.502802 9176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:18.502755 9179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:18.502710 9175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:43:18.504125 9167 server_base.cc:1034] running on GCE node
I20260807 22:43:18.505131 9167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:43:18.506516 9167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:43:18.507830 9167 hybrid_clock.cc:648] HybridClock initialized: now 1786142598507742 us; error 61 us; skew 500 ppm
I20260807 22:43:18.508414 9167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:43:18.511626 9167 webserver.cc:460] Webserver started at http://127.4.103.3:34169/ using document root <none> and password file <none>
I20260807 22:43:18.512666 9167 fs_manager.cc:359] Metadata directory not provided
I20260807 22:43:18.512841 9167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:43:18.513262 9167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 22:43:18.516835 9167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data/instance:
uuid: "cdf613248123414e9556f98601d9e402"
format_stamp: "Formatted at 2026-08-07 22:43:18 on dist-test-slave-kvfs"
I20260807 22:43:18.517845 9167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/wal/instance:
uuid: "cdf613248123414e9556f98601d9e402"
format_stamp: "Formatted at 2026-08-07 22:43:18 on dist-test-slave-kvfs"
I20260807 22:43:18.525058 9167 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260807 22:43:18.529769 9188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:18.531602 9167 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260807 22:43:18.531857 9167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/wal
uuid: "cdf613248123414e9556f98601d9e402"
format_stamp: "Formatted at 2026-08-07 22:43:18 on dist-test-slave-kvfs"
I20260807 22:43:18.532240 9167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:43:18.551453 9167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:43:18.552768 9167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:43:18.553220 9167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:43:18.557062 9167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 22:43:18.557256 9167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:18.557435 9167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 22:43:18.557528 9167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:18.618413 9167 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.3:38803
I20260807 22:43:18.618446 9358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.3:38803 every 8 connection(s)
I20260807 22:43:18.620189 9167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
I20260807 22:43:18.622993 9360 heartbeater.cc:347] Connected to a master server at 127.4.103.62:46193
I20260807 22:43:18.623323 9360 heartbeater.cc:464] Registering TS with master...
I20260807 22:43:18.624353 9360 heartbeater.cc:511] Master 127.4.103.62:46193 requested a full tablet report, sending...
I20260807 22:43:18.626240 8640 ts_manager.cc:194] Registered new tserver with Master: cdf613248123414e9556f98601d9e402 (127.4.103.3:38803)
I20260807 22:43:18.626420 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 9167
I20260807 22:43:18.626642 4508 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/wal/instance
I20260807 22:43:18.627664 8640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.3:60983
I20260807 22:43:18.630599 4508 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 22:43:18.829664 8640 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43570:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 22:43:18.830626 8640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 22:43:18.865065 8859 tablet_service.cc:1467] Processing CreateTablet for tablet 857795ece0994bc4b213f19c9151ff5a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0416f5a60d944970aa5f65a60c4df926]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:43:18.865972 9261 tablet_service.cc:1467] Processing CreateTablet for tablet 857795ece0994bc4b213f19c9151ff5a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0416f5a60d944970aa5f65a60c4df926]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:43:18.867319 8859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857795ece0994bc4b213f19c9151ff5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:43:18.868649 9261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857795ece0994bc4b213f19c9151ff5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:43:18.875309 9065 tablet_service.cc:1467] Processing CreateTablet for tablet 857795ece0994bc4b213f19c9151ff5a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0416f5a60d944970aa5f65a60c4df926]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 22:43:18.877686 9065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 857795ece0994bc4b213f19c9151ff5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 22:43:18.882733 9393 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: Bootstrap starting.
I20260807 22:43:18.884914 9395 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: Bootstrap starting.
I20260807 22:43:18.888285 9393 tablet_bootstrap.cc:654] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: Neither blocks nor log segments found. Creating new log.
I20260807 22:43:18.888455 9395 tablet_bootstrap.cc:654] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: Neither blocks nor log segments found. Creating new log.
I20260807 22:43:18.889554 9395 log.cc:824] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: Log is configured to *not* fsync() on all Append() calls
I20260807 22:43:18.889628 9393 log.cc:824] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: Log is configured to *not* fsync() on all Append() calls
I20260807 22:43:18.893180 9395 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: No bootstrap required, opened a new log
I20260807 22:43:18.893539 9393 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: No bootstrap required, opened a new log
I20260807 22:43:18.893525 9395 ts_tablet_manager.cc:1397] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260807 22:43:18.893842 9393 ts_tablet_manager.cc:1397] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260807 22:43:18.895121 9400 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: Bootstrap starting.
I20260807 22:43:18.900436 9400 tablet_bootstrap.cc:654] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: Neither blocks nor log segments found. Creating new log.
I20260807 22:43:18.901927 9400 log.cc:824] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: Log is configured to *not* fsync() on all Append() calls
I20260807 22:43:18.902908 9393 raft_consensus.cc:348] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:18.903575 9393 raft_consensus.cc:374] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:43:18.903780 9393 raft_consensus.cc:729] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d516f13f438085a3bd7c37805819, State: Initialized, Role: FOLLOWER
I20260807 22:43:18.904733 9393 consensus_queue.cc:260] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:18.905453 9395 raft_consensus.cc:348] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:18.905959 9395 raft_consensus.cc:374] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:43:18.906077 9395 raft_consensus.cc:729] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdf613248123414e9556f98601d9e402, State: Initialized, Role: FOLLOWER
I20260807 22:43:18.906764 9395 consensus_queue.cc:260] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:18.907298 9400 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: No bootstrap required, opened a new log
I20260807 22:43:18.907749 9400 ts_tablet_manager.cc:1397] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260807 22:43:18.913023 9402 tablet_replica.cc:439] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:18.914192 9402 tablet_replica.cc:442] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:18.914005 9393 ts_tablet_manager.cc:1428] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20260807 22:43:18.914445 9360 heartbeater.cc:503] Master 127.4.103.62:46193 was elected leader, sending a full tablet report...
I20260807 22:43:18.915062 9403 tablet_replica.cc:439] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:18.915573 9403 tablet_replica.cc:442] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:18.917598 9395 ts_tablet_manager.cc:1428] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: Time spent starting tablet: real 0.024s user 0.012s sys 0.008s
I20260807 22:43:18.921713 9400 raft_consensus.cc:348] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:18.922374 9400 raft_consensus.cc:374] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 22:43:18.922508 9400 raft_consensus.cc:729] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d63d037cacb4205b689fe6259f9b790, State: Initialized, Role: FOLLOWER
I20260807 22:43:18.923364 9400 consensus_queue.cc:260] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:18.924782 9161 heartbeater.cc:503] Master 127.4.103.62:46193 was elected leader, sending a full tablet report...
I20260807 22:43:18.924782 9404 tablet_replica.cc:439] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:18.925335 9404 tablet_replica.cc:442] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:18.925756 9400 ts_tablet_manager.cc:1428] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260807 22:43:18.934162 9403 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:43:18.934567 9403 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:18.936995 9403 leader_election.cc:288] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbb7d516f13f438085a3bd7c37805819 (127.4.103.1:33317), 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639)
I20260807 22:43:18.940660 9404 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:43:18.945709 8886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857795ece0994bc4b213f19c9151ff5a" candidate_uuid: "cdf613248123414e9556f98601d9e402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d516f13f438085a3bd7c37805819" is_pre_election: true
I20260807 22:43:18.946162 9404 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:18.946483 8886 raft_consensus.cc:2454] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdf613248123414e9556f98601d9e402 in term 0.
I20260807 22:43:18.947360 9190 leader_election.cc:302] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdf613248123414e9556f98601d9e402, fbb7d516f13f438085a3bd7c37805819; no voters:
I20260807 22:43:18.947932 9403 raft_consensus.cc:2781] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 22:43:18.948204 9403 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:43:18.948392 9403 raft_consensus.cc:3037] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:43:18.951457 9097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857795ece0994bc4b213f19c9151ff5a" candidate_uuid: "cdf613248123414e9556f98601d9e402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d63d037cacb4205b689fe6259f9b790" is_pre_election: true
I20260807 22:43:18.952013 9097 raft_consensus.cc:2454] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdf613248123414e9556f98601d9e402 in term 0.
I20260807 22:43:18.953613 9403 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:18.954775 9403 leader_election.cc:288] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [CANDIDATE]: Term 1 election: Requested vote from peers fbb7d516f13f438085a3bd7c37805819 (127.4.103.1:33317), 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639)
I20260807 22:43:18.955411 8886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857795ece0994bc4b213f19c9151ff5a" candidate_uuid: "cdf613248123414e9556f98601d9e402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d516f13f438085a3bd7c37805819"
I20260807 22:43:18.955722 8886 raft_consensus.cc:3037] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:43:18.956310 9097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857795ece0994bc4b213f19c9151ff5a" candidate_uuid: "cdf613248123414e9556f98601d9e402" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d63d037cacb4205b689fe6259f9b790"
I20260807 22:43:18.956595 9097 raft_consensus.cc:3037] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 0 FOLLOWER]: Advancing to term 1
I20260807 22:43:18.959733 8886 raft_consensus.cc:2454] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdf613248123414e9556f98601d9e402 in term 1.
I20260807 22:43:18.960351 9190 leader_election.cc:302] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdf613248123414e9556f98601d9e402, fbb7d516f13f438085a3bd7c37805819; no voters:
I20260807 22:43:18.960901 9097 raft_consensus.cc:2454] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdf613248123414e9556f98601d9e402 in term 1.
I20260807 22:43:18.962085 9403 raft_consensus.cc:2781] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 22:43:18.963218 9403 raft_consensus.cc:686] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 1 LEADER]: Becoming Leader. State: Replica: cdf613248123414e9556f98601d9e402, State: Running, Role: LEADER
I20260807 22:43:18.964197 9403 consensus_queue.cc:237] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:18.969357 9411 tablet_replica.cc:439] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: TxnStatusTablet state changed. Reason: New leader cdf613248123414e9556f98601d9e402. Latest consensus state: current_term: 1 leader_uuid: "cdf613248123414e9556f98601d9e402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:18.969815 9411 tablet_replica.cc:442] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:18.974196 8643 catalog_manager.cc:5495] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdf613248123414e9556f98601d9e402 (127.4.103.3). New cstate: current_term: 1 leader_uuid: "cdf613248123414e9556f98601d9e402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } health_report { overall_health: UNKNOWN } } }
I20260807 22:43:18.975610 9404 leader_election.cc:288] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdf613248123414e9556f98601d9e402 (127.4.103.3:38803), fbb7d516f13f438085a3bd7c37805819 (127.4.103.1:33317)
I20260807 22:43:18.983112 9294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857795ece0994bc4b213f19c9151ff5a" candidate_uuid: "2d63d037cacb4205b689fe6259f9b790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdf613248123414e9556f98601d9e402" is_pre_election: true
I20260807 22:43:18.985934 9423 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:43:18.986336 9423 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:43:18.991849 8886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857795ece0994bc4b213f19c9151ff5a" candidate_uuid: "2d63d037cacb4205b689fe6259f9b790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb7d516f13f438085a3bd7c37805819" is_pre_election: true
I20260807 22:43:18.992451 8886 raft_consensus.cc:2379] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2d63d037cacb4205b689fe6259f9b790 in current term 1: Already voted for candidate cdf613248123414e9556f98601d9e402 in this term.
I20260807 22:43:18.993544 8990 leader_election.cc:302] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d63d037cacb4205b689fe6259f9b790; no voters: cdf613248123414e9556f98601d9e402, fbb7d516f13f438085a3bd7c37805819
I20260807 22:43:18.994189 9404 raft_consensus.cc:2726] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260807 22:43:19.003455 9412 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
W20260807 22:43:19.004087 9162 tablet.cc:2369] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 22:43:19.012787 9411 consensus_queue.cc:1035] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:43:19.018627 9404 tablet_replica.cc:439] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: TxnStatusTablet state changed. Reason: New leader cdf613248123414e9556f98601d9e402. Latest consensus state: current_term: 1 leader_uuid: "cdf613248123414e9556f98601d9e402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:19.019057 9404 tablet_replica.cc:442] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:19.023694 9403 tablet_replica.cc:439] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdf613248123414e9556f98601d9e402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:19.024155 9403 tablet_replica.cc:442] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:19.027805 9403 consensus_queue.cc:1035] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 22:43:19.044155 9404 tablet_replica.cc:439] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdf613248123414e9556f98601d9e402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:19.044513 9404 tablet_replica.cc:442] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:19.054268 9402 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cdf613248123414e9556f98601d9e402)
I20260807 22:43:19.054557 9402 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:19.056530 9430 tablet_replica.cc:439] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: TxnStatusTablet state changed. Reason: New leader cdf613248123414e9556f98601d9e402. Latest consensus state: current_term: 1 leader_uuid: "cdf613248123414e9556f98601d9e402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:19.057039 9430 tablet_replica.cc:442] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:19.062049 9402 leader_election.cc:288] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdf613248123414e9556f98601d9e402 (127.4.103.3:38803), 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639)
I20260807 22:43:19.062004 9411 tablet_replica.cc:439] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdf613248123414e9556f98601d9e402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:19.062405 9411 tablet_replica.cc:442] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:19.068950 9436 tablet_replica.cc:439] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdf613248123414e9556f98601d9e402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:19.071921 9436 tablet_replica.cc:442] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:19.091995 9294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857795ece0994bc4b213f19c9151ff5a" candidate_uuid: "fbb7d516f13f438085a3bd7c37805819" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cdf613248123414e9556f98601d9e402" is_pre_election: true
I20260807 22:43:19.094236 9097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857795ece0994bc4b213f19c9151ff5a" candidate_uuid: "fbb7d516f13f438085a3bd7c37805819" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2d63d037cacb4205b689fe6259f9b790" is_pre_election: true
I20260807 22:43:19.098502 8775 leader_election.cc:302] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbb7d516f13f438085a3bd7c37805819; no voters: 2d63d037cacb4205b689fe6259f9b790, cdf613248123414e9556f98601d9e402
I20260807 22:43:19.099920 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 8755
I20260807 22:43:19.100827 9436 raft_consensus.cc:2726] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 22:43:19.101601 8946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:19.213179 9436 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cdf613248123414e9556f98601d9e402)
I20260807 22:43:19.213554 9436 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:19.215082 9436 leader_election.cc:288] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdf613248123414e9556f98601d9e402 (127.4.103.3:38803), 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639)
I20260807 22:43:19.216265 9097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857795ece0994bc4b213f19c9151ff5a" candidate_uuid: "fbb7d516f13f438085a3bd7c37805819" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2d63d037cacb4205b689fe6259f9b790" is_pre_election: true
I20260807 22:43:19.216356 9294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857795ece0994bc4b213f19c9151ff5a" candidate_uuid: "fbb7d516f13f438085a3bd7c37805819" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cdf613248123414e9556f98601d9e402" is_pre_election: true
I20260807 22:43:19.216594 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 8755
W20260807 22:43:19.248855 8991 connection.cc:535] server connection from 127.4.103.1:54601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 22:43:19.249701 9191 connection.cc:535] server connection from 127.4.103.1:40163 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 22:43:19.250705 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 8955
I20260807 22:43:19.252609 9153 generic_service.cc:196] Checking for leaks (request via RPC)
W20260807 22:43:19.258397 9190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.103.1:33317: connect: Connection refused (error 111)
W20260807 22:43:19.260567 9190 consensus_peers.cc:487] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 -> Peer fbb7d516f13f438085a3bd7c37805819 (127.4.103.1:33317): Couldn't send request to peer fbb7d516f13f438085a3bd7c37805819. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.1:33317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 22:43:19.309384 9403 tablet_replica.cc:439] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdf613248123414e9556f98601d9e402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:19.309820 9403 tablet_replica.cc:442] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:19.413367 9404 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cdf613248123414e9556f98601d9e402)
I20260807 22:43:19.413622 9404 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:19.413856 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 8955
W20260807 22:43:19.441402 9191 connection.cc:535] client connection to 127.4.103.2:40639 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 22:43:19.442494 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 9167
W20260807 22:43:19.443749 9191 consensus_peers.cc:487] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 -> Peer 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639): Couldn't send request to peer 2d63d037cacb4205b689fe6259f9b790. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.2:40639: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 22:43:19.444008 9351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:19.592379 9450 tablet_replica.cc:439] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdf613248123414e9556f98601d9e402" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:19.592875 9450 tablet_replica.cc:442] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:19.593159 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 9167
W20260807 22:43:19.626472 9371 connection.cc:535] client connection to 127.4.103.3:38803 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 22:43:19.626885 9371 meta_cache.cc:302] tablet 857795ece0994bc4b213f19c9151ff5a: replica cdf613248123414e9556f98601d9e402 (127.4.103.3:38803) has failed: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 22:43:19.736289 8682 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 4) took 129 ms (client timeout 124 ms). Trace:
W20260807 22:43:19.736563 8682 rpcz_store.cc:269] 0807 22:43:19.606711 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:19.606836 (+ 125us) service_pool.cc:227] Handling call
0807 22:43:19.736176 (+129340us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:19.864352 8681 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 5) took 129 ms (client timeout 124 ms). Trace:
W20260807 22:43:19.864590 8681 rpcz_store.cc:269] 0807 22:43:19.734602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:19.734737 (+ 135us) service_pool.cc:227] Handling call
0807 22:43:19.864252 (+129515us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:19.995196 8682 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 6) took 133 ms (client timeout 124 ms). Trace:
W20260807 22:43:19.995415 8682 rpcz_store.cc:269] 0807 22:43:19.861435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:19.861545 (+ 110us) service_pool.cc:227] Handling call
0807 22:43:19.995091 (+133546us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:20.117612 8681 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 7) took 128 ms (client timeout 124 ms). Trace:
W20260807 22:43:20.117815 8681 rpcz_store.cc:269] 0807 22:43:19.989019 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:19.989154 (+ 135us) service_pool.cc:227] Handling call
0807 22:43:20.117525 (+128371us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:20.256585 8682 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 8) took 140 ms (client timeout 124 ms). Trace:
W20260807 22:43:20.256800 8682 rpcz_store.cc:269] 0807 22:43:20.115665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:20.115758 (+ 93us) service_pool.cc:227] Handling call
0807 22:43:20.256453 (+140695us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:20.376583 8681 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 9) took 133 ms (client timeout 124 ms). Trace:
W20260807 22:43:20.376777 8681 rpcz_store.cc:269] 0807 22:43:20.242966 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:20.243069 (+ 103us) service_pool.cc:227] Handling call
0807 22:43:20.376480 (+133411us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:20.508303 8682 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 10) took 138 ms (client timeout 124 ms). Trace:
W20260807 22:43:20.508517 8682 rpcz_store.cc:269] 0807 22:43:20.370268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:20.370379 (+ 111us) service_pool.cc:227] Handling call
0807 22:43:20.508182 (+137803us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:20.628139 9371 meta_cache.cc:302] tablet 857795ece0994bc4b213f19c9151ff5a: replica cdf613248123414e9556f98601d9e402 (127.4.103.3:38803) has failed: Network error: Client connection negotiation failed: client connection to 127.4.103.3:38803: connect: Connection refused (error 111)
W20260807 22:43:20.629993 8681 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 11) took 133 ms (client timeout 124 ms). Trace:
W20260807 22:43:20.630132 8681 rpcz_store.cc:269] 0807 22:43:20.496935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:20.497039 (+ 104us) service_pool.cc:227] Handling call
0807 22:43:20.629904 (+132865us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:20.749805 9370 meta_cache.cc:1015] Timed out: LookupRpc { table: 'kudu_system.kudu_transactions', partition-key: (RANGE (txn_id): <redacted>), attempt: 1 } failed: LookupRpc timed out after deadline expired: GetTableLocations RPC to 127.4.103.62:46193 timed out after 0.001s (SENT)
W20260807 22:43:20.750293 8682 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 12) took 125 ms (client timeout 124 ms). Trace:
W20260807 22:43:20.750479 8682 rpcz_store.cc:269] 0807 22:43:20.624420 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:20.624567 (+ 147us) service_pool.cc:227] Handling call
0807 22:43:20.750207 (+125640us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:20.880178 8682 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 13) took 128 ms (client timeout 124 ms). Trace:
W20260807 22:43:20.880393 8682 rpcz_store.cc:269] 0807 22:43:20.751396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:20.751482 (+ 86us) service_pool.cc:227] Handling call
0807 22:43:20.880057 (+128575us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:21.013825 8681 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 14) took 134 ms (client timeout 124 ms). Trace:
W20260807 22:43:21.014020 8681 rpcz_store.cc:269] 0807 22:43:20.879320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:20.879439 (+ 119us) service_pool.cc:227] Handling call
0807 22:43:21.013711 (+134272us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:21.127853 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.1:33317
--local_ip_for_outbound_sockets=127.4.103.1
--tserver_master_addrs=127.4.103.62:46193
--webserver_port=34977
--webserver_interface=127.4.103.1
--builtin_ntp_servers=127.4.103.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
W20260807 22:43:21.141373 8682 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 15) took 134 ms (client timeout 124 ms). Trace:
W20260807 22:43:21.141714 8682 rpcz_store.cc:269] 0807 22:43:21.006657 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:21.006798 (+ 141us) service_pool.cc:227] Handling call
0807 22:43:21.141266 (+134468us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:21.273069 8681 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 16) took 133 ms (client timeout 124 ms). Trace:
W20260807 22:43:21.273269 8681 rpcz_store.cc:269] 0807 22:43:21.139352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:21.139503 (+ 151us) service_pool.cc:227] Handling call
0807 22:43:21.272960 (+133457us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:21.393637 8682 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 17) took 127 ms (client timeout 124 ms). Trace:
W20260807 22:43:21.393893 8682 rpcz_store.cc:269] 0807 22:43:21.266252 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:21.266357 (+ 105us) service_pool.cc:227] Handling call
0807 22:43:21.393501 (+127144us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:21.524009 8681 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 18) took 130 ms (client timeout 124 ms). Trace:
W20260807 22:43:21.524247 8681 rpcz_store.cc:269] 0807 22:43:21.393592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:21.393731 (+ 139us) service_pool.cc:227] Handling call
0807 22:43:21.523861 (+130130us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:21.596446 9458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:43:21.596822 9458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:43:21.597003 9458 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:43:21.597105 9458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:43:21.606938 9458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:43:21.607170 9458 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:43:21.607251 9458 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:43:21.607316 9458 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:43:21.607461 9458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.1
I20260807 22:43:21.620819 9458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.1:33317
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.103.1
--webserver_port=34977
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:46193
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.1
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:43:21.622700 9458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:43:21.625594 9458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:43:21.638722 9371 meta_cache.cc:302] tablet 857795ece0994bc4b213f19c9151ff5a: replica 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639) has failed: Network error: Client connection negotiation failed: client connection to 127.4.103.2:40639: connect: Connection refused (error 111)
W20260807 22:43:21.646291 9480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:21.646318 9475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:21.646773 9477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:43:21.647357 9458 server_base.cc:1034] running on GCE node
I20260807 22:43:21.648455 9458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:43:21.649991 9458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:43:21.651297 9458 hybrid_clock.cc:648] HybridClock initialized: now 1786142601651169 us; error 86 us; skew 500 ppm
I20260807 22:43:21.651686 9458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:43:21.654871 9458 webserver.cc:460] Webserver started at http://127.4.103.1:34977/ using document root <none> and password file <none>
W20260807 22:43:21.655012 8682 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 19) took 133 ms (client timeout 124 ms). Trace:
W20260807 22:43:21.655156 8682 rpcz_store.cc:269] 0807 22:43:21.521122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:21.521228 (+ 106us) service_pool.cc:227] Handling call
0807 22:43:21.654902 (+133674us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:21.656168 9458 fs_manager.cc:359] Metadata directory not provided
I20260807 22:43:21.656402 9458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:43:21.663823 9458 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260807 22:43:21.668128 9489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:21.670008 9458 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:43:21.670259 9458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/wal
uuid: "fbb7d516f13f438085a3bd7c37805819"
format_stamp: "Formatted at 2026-08-07 22:43:17 on dist-test-slave-kvfs"
I20260807 22:43:21.671626 9458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:43:21.694408 9458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:43:21.695603 9458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:43:21.695992 9458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:43:21.700894 9500 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 22:43:21.712815 9458 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 22:43:21.713028 9458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.000s
I20260807 22:43:21.713223 9458 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 22:43:21.719564 9458 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 22:43:21.719736 9458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.003s sys 0.000s
I20260807 22:43:21.720376 9500 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: Bootstrap starting.
I20260807 22:43:21.757822 9500 log.cc:824] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: Log is configured to *not* fsync() on all Append() calls
I20260807 22:43:21.772717 9500 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:43:21.773518 9500 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: Bootstrap complete.
I20260807 22:43:21.775035 9500 ts_tablet_manager.cc:1397] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: Time spent bootstrapping tablet: real 0.055s user 0.023s sys 0.025s
I20260807 22:43:21.785804 9500 raft_consensus.cc:348] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:21.787685 9500 raft_consensus.cc:729] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb7d516f13f438085a3bd7c37805819, State: Initialized, Role: FOLLOWER
W20260807 22:43:21.788256 8681 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 20) took 139 ms (client timeout 124 ms). Trace:
W20260807 22:43:21.788411 8681 rpcz_store.cc:269] 0807 22:43:21.648627 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:21.648715 (+ 88us) service_pool.cc:227] Handling call
0807 22:43:21.788145 (+139430us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:21.788822 9500 consensus_queue.cc:260] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:21.790532 9617 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:21.791307 9617 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:21.791323 9500 ts_tablet_manager.cc:1428] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819: Time spent starting tablet: real 0.016s user 0.017s sys 0.001s
I20260807 22:43:21.805904 9458 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.1:33317
I20260807 22:43:21.805979 9668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.1:33317 every 8 connection(s)
I20260807 22:43:21.807706 9458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-0/data/info.pb
I20260807 22:43:21.810678 9669 heartbeater.cc:347] Connected to a master server at 127.4.103.62:46193
I20260807 22:43:21.811017 9669 heartbeater.cc:464] Registering TS with master...
I20260807 22:43:21.811931 9669 heartbeater.cc:511] Master 127.4.103.62:46193 requested a full tablet report, sending...
I20260807 22:43:21.815155 8640 ts_manager.cc:194] Re-registered known tserver with Master: fbb7d516f13f438085a3bd7c37805819 (127.4.103.1:33317)
I20260807 22:43:21.815654 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 9458
I20260807 22:43:21.816704 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.2:40639
--local_ip_for_outbound_sockets=127.4.103.2
--tserver_master_addrs=127.4.103.62:46193
--webserver_port=37497
--webserver_interface=127.4.103.2
--builtin_ntp_servers=127.4.103.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
I20260807 22:43:21.816748 8640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.1:44501
I20260807 22:43:21.857820 9617 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:43:21.858302 9617 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:21.861508 9617 leader_election.cc:288] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdf613248123414e9556f98601d9e402 (127.4.103.3:38803), 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639)
W20260807 22:43:21.862039 9492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.103.3:38803: connect: Connection refused (error 111)
W20260807 22:43:21.864110 9492 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cdf613248123414e9556f98601d9e402 (127.4.103.3:38803): Network error: Client connection negotiation failed: client connection to 127.4.103.3:38803: connect: Connection refused (error 111)
W20260807 22:43:21.864450 9492 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639): Network error: Client connection negotiation failed: client connection to 127.4.103.2:40639: connect: Connection refused (error 111)
I20260807 22:43:21.864576 9492 leader_election.cc:302] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbb7d516f13f438085a3bd7c37805819; no voters: 2d63d037cacb4205b689fe6259f9b790, cdf613248123414e9556f98601d9e402
I20260807 22:43:21.865018 9617 raft_consensus.cc:2726] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 22:43:21.914839 8682 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 21) took 139 ms (client timeout 124 ms). Trace:
W20260807 22:43:21.915102 8682 rpcz_store.cc:269] 0807 22:43:21.775799 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:21.775922 (+ 123us) service_pool.cc:227] Handling call
0807 22:43:21.914707 (+138785us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:21.937686 9617 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:43:21.938006 9617 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:21.938951 9617 leader_election.cc:288] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdf613248123414e9556f98601d9e402 (127.4.103.3:38803), 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639)
W20260807 22:43:21.941485 9492 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cdf613248123414e9556f98601d9e402 (127.4.103.3:38803): Network error: Client connection negotiation failed: client connection to 127.4.103.3:38803: connect: Connection refused (error 111)
W20260807 22:43:21.941737 9492 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639): Network error: Client connection negotiation failed: client connection to 127.4.103.2:40639: connect: Connection refused (error 111)
I20260807 22:43:21.941870 9492 leader_election.cc:302] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbb7d516f13f438085a3bd7c37805819; no voters: 2d63d037cacb4205b689fe6259f9b790, cdf613248123414e9556f98601d9e402
I20260807 22:43:21.942164 9617 raft_consensus.cc:2726] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 22:43:22.043126 8681 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 22) took 140 ms (client timeout 124 ms). Trace:
W20260807 22:43:22.043372 8681 rpcz_store.cc:269] 0807 22:43:21.903066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:21.903191 (+ 125us) service_pool.cc:227] Handling call
0807 22:43:22.043024 (+139833us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:22.052776 9617 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:43:22.053061 9617 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:22.054014 9617 leader_election.cc:288] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdf613248123414e9556f98601d9e402 (127.4.103.3:38803), 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639)
W20260807 22:43:22.056238 9492 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cdf613248123414e9556f98601d9e402 (127.4.103.3:38803): Network error: Client connection negotiation failed: client connection to 127.4.103.3:38803: connect: Connection refused (error 111)
W20260807 22:43:22.056533 9492 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639): Network error: Client connection negotiation failed: client connection to 127.4.103.2:40639: connect: Connection refused (error 111)
I20260807 22:43:22.056677 9492 leader_election.cc:302] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbb7d516f13f438085a3bd7c37805819; no voters: 2d63d037cacb4205b689fe6259f9b790, cdf613248123414e9556f98601d9e402
I20260807 22:43:22.056996 9617 raft_consensus.cc:2726] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 22:43:22.168157 8682 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 23) took 137 ms (client timeout 124 ms). Trace:
W20260807 22:43:22.168386 8682 rpcz_store.cc:269] 0807 22:43:22.030438 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:22.030539 (+ 101us) service_pool.cc:227] Handling call
0807 22:43:22.167996 (+137457us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:22.185356 9617 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:43:22.185626 9617 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:22.186611 9617 leader_election.cc:288] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdf613248123414e9556f98601d9e402 (127.4.103.3:38803), 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639)
W20260807 22:43:22.188794 9492 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cdf613248123414e9556f98601d9e402 (127.4.103.3:38803): Network error: Client connection negotiation failed: client connection to 127.4.103.3:38803: connect: Connection refused (error 111)
W20260807 22:43:22.189051 9492 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639): Network error: Client connection negotiation failed: client connection to 127.4.103.2:40639: connect: Connection refused (error 111)
I20260807 22:43:22.189196 9492 leader_election.cc:302] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbb7d516f13f438085a3bd7c37805819; no voters: 2d63d037cacb4205b689fe6259f9b790, cdf613248123414e9556f98601d9e402
I20260807 22:43:22.189525 9617 raft_consensus.cc:2726] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 22:43:22.274185 9673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:43:22.274603 9673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:43:22.274700 9673 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:43:22.274766 9673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 22:43:22.277729 9617 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:43:22.278039 9617 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:22.279014 9617 leader_election.cc:288] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdf613248123414e9556f98601d9e402 (127.4.103.3:38803), 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639)
W20260807 22:43:22.281282 9492 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cdf613248123414e9556f98601d9e402 (127.4.103.3:38803): Network error: Client connection negotiation failed: client connection to 127.4.103.3:38803: connect: Connection refused (error 111)
W20260807 22:43:22.281563 9492 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639): Network error: Client connection negotiation failed: client connection to 127.4.103.2:40639: connect: Connection refused (error 111)
I20260807 22:43:22.281687 9492 leader_election.cc:302] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbb7d516f13f438085a3bd7c37805819; no voters: 2d63d037cacb4205b689fe6259f9b790, cdf613248123414e9556f98601d9e402
I20260807 22:43:22.282011 9617 raft_consensus.cc:2726] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260807 22:43:22.285022 9673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:43:22.285214 9673 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:43:22.285321 9673 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:43:22.285409 9673 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:43:22.285564 9673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.2
I20260807 22:43:22.297559 9673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.2:40639
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.103.2
--webserver_port=37497
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:46193
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.2
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:43:22.299582 9673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260807 22:43:22.300886 8681 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 24) took 143 ms (client timeout 124 ms). Trace:
W20260807 22:43:22.301218 8681 rpcz_store.cc:269] 0807 22:43:22.157867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:22.157984 (+ 117us) service_pool.cc:227] Handling call
0807 22:43:22.300781 (+142797us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:22.302543 9673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:43:22.321135 9683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:22.321404 9684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:22.321633 9687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:43:22.322966 9673 server_base.cc:1034] running on GCE node
I20260807 22:43:22.323868 9673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:43:22.325300 9673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:43:22.326653 9673 hybrid_clock.cc:648] HybridClock initialized: now 1786142602326550 us; error 57 us; skew 500 ppm
I20260807 22:43:22.327068 9673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:43:22.330394 9673 webserver.cc:460] Webserver started at http://127.4.103.2:37497/ using document root <none> and password file <none>
I20260807 22:43:22.331606 9673 fs_manager.cc:359] Metadata directory not provided
I20260807 22:43:22.331812 9673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:43:22.339310 9673 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 22:43:22.343539 9698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:22.345815 9673 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 22:43:22.346082 9673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/wal
uuid: "2d63d037cacb4205b689fe6259f9b790"
format_stamp: "Formatted at 2026-08-07 22:43:17 on dist-test-slave-kvfs"
I20260807 22:43:22.347411 9673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 22:43:22.368063 9673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:43:22.369316 9673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:43:22.369712 9673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:43:22.374698 9707 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 22:43:22.383327 9673 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 22:43:22.383531 9673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.000s
I20260807 22:43:22.383745 9673 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 22:43:22.389969 9673 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 22:43:22.390141 9673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.005s sys 0.000s
I20260807 22:43:22.390671 9707 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: Bootstrap starting.
I20260807 22:43:22.425649 9707 log.cc:824] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: Log is configured to *not* fsync() on all Append() calls
I20260807 22:43:22.442008 9707 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:43:22.443303 9707 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: Bootstrap complete.
I20260807 22:43:22.445336 9707 ts_tablet_manager.cc:1397] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: Time spent bootstrapping tablet: real 0.055s user 0.024s sys 0.027s
I20260807 22:43:22.461249 9707 raft_consensus.cc:348] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:22.462762 9707 raft_consensus.cc:729] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d63d037cacb4205b689fe6259f9b790, State: Initialized, Role: FOLLOWER
I20260807 22:43:22.463735 9707 consensus_queue.cc:260] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
W20260807 22:43:22.466522 8682 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 25) took 181 ms (client timeout 124 ms). Trace:
W20260807 22:43:22.466751 8682 rpcz_store.cc:269] 0807 22:43:22.284728 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:22.284830 (+ 102us) service_pool.cc:227] Handling call
0807 22:43:22.466381 (+181551us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:22.469508 9843 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:22.470224 9843 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:22.470656 9707 ts_tablet_manager.cc:1428] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790: Time spent starting tablet: real 0.025s user 0.014s sys 0.010s
I20260807 22:43:22.477972 9673 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.2:40639
I20260807 22:43:22.478025 9869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.2:40639 every 8 connection(s)
I20260807 22:43:22.479892 9673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-1/data/info.pb
I20260807 22:43:22.483314 9872 heartbeater.cc:347] Connected to a master server at 127.4.103.62:46193
I20260807 22:43:22.483634 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 9673
I20260807 22:43:22.483685 9872 heartbeater.cc:464] Registering TS with master...
I20260807 22:43:22.484663 4508 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
/tmp/dist-test-taskZnDlfK/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.103.3:38803
--local_ip_for_outbound_sockets=127.4.103.3
--tserver_master_addrs=127.4.103.62:46193
--webserver_port=34169
--webserver_interface=127.4.103.3
--builtin_ntp_servers=127.4.103.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--txn_keepalive_interval_ms=500
--txn_staleness_tracker_interval_ms=100
--txn_staleness_tracker_disabled_interval_ms=100
--raft_heartbeat_interval_ms=50
--leader_failure_max_missed_heartbeat_periods=1.25
--enable_txn_system_client_init=true
--txn_schedule_background_tasks=false with env {}
I20260807 22:43:22.484886 9872 heartbeater.cc:511] Master 127.4.103.62:46193 requested a full tablet report, sending...
I20260807 22:43:22.488607 8640 ts_manager.cc:194] Re-registered known tserver with Master: 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639)
I20260807 22:43:22.490012 8640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.2:45713
I20260807 22:43:22.516571 9843 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:43:22.517005 9843 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:22.519747 9843 leader_election.cc:288] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdf613248123414e9556f98601d9e402 (127.4.103.3:38803), fbb7d516f13f438085a3bd7c37805819 (127.4.103.1:33317)
I20260807 22:43:22.521430 9617 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 22:43:22.521673 9617 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:22.522552 9617 leader_election.cc:288] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdf613248123414e9556f98601d9e402 (127.4.103.3:38803), 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639)
W20260807 22:43:22.522470 9700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.103.3:38803: connect: Connection refused (error 111)
W20260807 22:43:22.524518 9700 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cdf613248123414e9556f98601d9e402 (127.4.103.3:38803): Network error: Client connection negotiation failed: client connection to 127.4.103.3:38803: connect: Connection refused (error 111)
W20260807 22:43:22.526669 9492 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cdf613248123414e9556f98601d9e402 (127.4.103.3:38803): Network error: Client connection negotiation failed: client connection to 127.4.103.3:38803: connect: Connection refused (error 111)
I20260807 22:43:22.533179 9602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857795ece0994bc4b213f19c9151ff5a" candidate_uuid: "2d63d037cacb4205b689fe6259f9b790" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fbb7d516f13f438085a3bd7c37805819" is_pre_election: true
I20260807 22:43:22.533803 9602 raft_consensus.cc:2454] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d63d037cacb4205b689fe6259f9b790 in term 1.
I20260807 22:43:22.534693 9699 leader_election.cc:302] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d63d037cacb4205b689fe6259f9b790, fbb7d516f13f438085a3bd7c37805819; no voters: cdf613248123414e9556f98601d9e402
I20260807 22:43:22.535742 9843 raft_consensus.cc:2781] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 22:43:22.536346 9843 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 22:43:22.536707 9843 raft_consensus.cc:3037] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:43:22.538607 9802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857795ece0994bc4b213f19c9151ff5a" candidate_uuid: "fbb7d516f13f438085a3bd7c37805819" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2d63d037cacb4205b689fe6259f9b790" is_pre_election: true
I20260807 22:43:22.543236 9843 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:22.543920 9802 raft_consensus.cc:2379] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fbb7d516f13f438085a3bd7c37805819 in current term 2: Already voted for candidate 2d63d037cacb4205b689fe6259f9b790 in this term.
I20260807 22:43:22.544328 9843 leader_election.cc:288] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [CANDIDATE]: Term 2 election: Requested vote from peers cdf613248123414e9556f98601d9e402 (127.4.103.3:38803), fbb7d516f13f438085a3bd7c37805819 (127.4.103.1:33317)
W20260807 22:43:22.545010 8681 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 26) took 132 ms (client timeout 124 ms). Trace:
W20260807 22:43:22.545176 8681 rpcz_store.cc:269] 0807 22:43:22.412736 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:22.412865 (+ 129us) service_pool.cc:227] Handling call
0807 22:43:22.544920 (+132055us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260807 22:43:22.545881 9602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "857795ece0994bc4b213f19c9151ff5a" candidate_uuid: "2d63d037cacb4205b689fe6259f9b790" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fbb7d516f13f438085a3bd7c37805819"
I20260807 22:43:22.546207 9602 raft_consensus.cc:3037] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:43:22.546916 9492 leader_election.cc:302] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbb7d516f13f438085a3bd7c37805819; no voters: 2d63d037cacb4205b689fe6259f9b790, cdf613248123414e9556f98601d9e402
W20260807 22:43:22.547092 9700 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cdf613248123414e9556f98601d9e402 (127.4.103.3:38803): Network error: Client connection negotiation failed: client connection to 127.4.103.3:38803: connect: Connection refused (error 111)
I20260807 22:43:22.551316 9602 raft_consensus.cc:2454] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d63d037cacb4205b689fe6259f9b790 in term 2.
I20260807 22:43:22.551542 9617 raft_consensus.cc:2726] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 22:43:22.551980 9699 leader_election.cc:302] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d63d037cacb4205b689fe6259f9b790, fbb7d516f13f438085a3bd7c37805819; no voters: cdf613248123414e9556f98601d9e402
I20260807 22:43:22.552407 9843 raft_consensus.cc:2781] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 22:43:22.553048 9843 raft_consensus.cc:686] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [term 2 LEADER]: Becoming Leader. State: Replica: 2d63d037cacb4205b689fe6259f9b790, State: Running, Role: LEADER
I20260807 22:43:22.553471 9872 heartbeater.cc:503] Master 127.4.103.62:46193 was elected leader, sending a full tablet report...
I20260807 22:43:22.554126 9843 consensus_queue.cc:237] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:22.557543 9886 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 2d63d037cacb4205b689fe6259f9b790. Latest consensus state: current_term: 2 leader_uuid: "2d63d037cacb4205b689fe6259f9b790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:22.558157 9886 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:22.559051 9888 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 22:43:22.559302 9888 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 22:43:22.560415 8640 catalog_manager.cc:5495] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 reported cstate change: term changed from 1 to 2, leader changed from cdf613248123414e9556f98601d9e402 (127.4.103.3) to 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2). New cstate: current_term: 2 leader_uuid: "2d63d037cacb4205b689fe6259f9b790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } health_report { overall_health: HEALTHY } } }
I20260807 22:43:22.610121 9602 raft_consensus.cc:1261] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 2 FOLLOWER]: Refusing update from remote peer 2d63d037cacb4205b689fe6259f9b790: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260807 22:43:22.611752 9886 consensus_queue.cc:1035] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260807 22:43:22.612423 9700 consensus_peers.cc:487] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 -> Peer cdf613248123414e9556f98601d9e402 (127.4.103.3:38803): Couldn't send request to peer cdf613248123414e9556f98601d9e402. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.3:38803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 22:43:22.615437 9669 heartbeater.cc:503] Master 127.4.103.62:46193 was elected leader, sending a full tablet report...
I20260807 22:43:22.629179 9617 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 2d63d037cacb4205b689fe6259f9b790. Latest consensus state: current_term: 2 leader_uuid: "2d63d037cacb4205b689fe6259f9b790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:22.629640 9617 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:22.638980 9617 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2d63d037cacb4205b689fe6259f9b790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:22.639511 9617 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:22.642103 9886 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2d63d037cacb4205b689fe6259f9b790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:22.642535 9886 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
W20260807 22:43:22.665679 9371 meta_cache.cc:302] tablet 857795ece0994bc4b213f19c9151ff5a: replica cdf613248123414e9556f98601d9e402 (127.4.103.3:38803) has failed: Timed out: connection negotiation to 127.4.103.3:38803 for RPC CoordinateTransaction timed out after 0.121s (ON_OUTBOUND_QUEUE)
W20260807 22:43:22.671044 8682 rpcz_store.cc:267] Call kudu.transactions.TxnManagerService.KeepTransactionAlive from 127.0.0.1:43572 (request call id 27) took 130 ms (client timeout 124 ms). Trace:
W20260807 22:43:22.671284 8682 rpcz_store.cc:269] 0807 22:43:22.540390 (+ 0us) service_pool.cc:170] Inserting onto call queue
0807 22:43:22.540505 (+ 115us) service_pool.cc:227] Handling call
0807 22:43:22.670839 (+130334us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260807 22:43:22.857528 9700 consensus_peers.cc:487] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 -> Peer cdf613248123414e9556f98601d9e402 (127.4.103.3:38803): Couldn't send request to peer cdf613248123414e9556f98601d9e402. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.3:38803: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260807 22:43:22.999027 9874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 22:43:22.999490 9874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 22:43:22.999661 9874 flags.cc:425] Enabled unsafe flag: --txn_schedule_background_tasks=false
W20260807 22:43:22.999780 9874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 22:43:23.008249 9874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 22:43:23.008487 9874 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 22:43:23.008586 9874 flags.cc:425] Enabled experimental flag: --txn_staleness_tracker_interval_ms=100
W20260807 22:43:23.008661 9874 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 22:43:23.008811 9874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.103.3
I20260807 22:43:23.020473 9874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.103.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--leader_failure_max_missed_heartbeat_periods=1.25
--raft_heartbeat_interval_ms=50
--fs_data_dirs=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.103.3:38803
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.103.3
--webserver_port=34169
--txn_keepalive_interval_ms=500
--txn_schedule_background_tasks=false
--txn_staleness_tracker_interval_ms=100
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.103.62:46193
--txn_staleness_tracker_disabled_interval_ms=100
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.103.3
--log_dir=/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 07 Aug 2026 21:50:02 UTC on ccycloud.quasar-bxybku.root.comops.site
build id 4573649
ASAN enabled
I20260807 22:43:23.022356 9874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 22:43:23.025416 9874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 22:43:23.044586 9907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:23.044765 9906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 22:43:23.046896 9911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 22:43:23.047156 9874 server_base.cc:1034] running on GCE node
I20260807 22:43:23.048288 9874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 22:43:23.050150 9874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 22:43:23.051592 9874 hybrid_clock.cc:648] HybridClock initialized: now 1786142603051508 us; error 105 us; skew 500 ppm
I20260807 22:43:23.052142 9874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260807 22:43:23.055979 9874 webserver.cc:460] Webserver started at http://127.4.103.3:34169/ using document root <none> and password file <none>
I20260807 22:43:23.057159 9874 fs_manager.cc:359] Metadata directory not provided
I20260807 22:43:23.057382 9874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 22:43:23.065608 9874 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260807 22:43:23.069416 9921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 22:43:23.071224 9874 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260807 22:43:23.071461 9874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/wal
uuid: "cdf613248123414e9556f98601d9e402"
format_stamp: "Formatted at 2026-08-07 22:43:18 on dist-test-slave-kvfs"
I20260807 22:43:23.072851 9874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260807 22:43:23.083818 9700 consensus_peers.cc:487] T 857795ece0994bc4b213f19c9151ff5a P 2d63d037cacb4205b689fe6259f9b790 -> Peer cdf613248123414e9556f98601d9e402 (127.4.103.3:38803): Couldn't send request to peer cdf613248123414e9556f98601d9e402. Status: Network error: Client connection negotiation failed: client connection to 127.4.103.3:38803: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260807 22:43:23.095453 9874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 22:43:23.096755 9874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 22:43:23.097124 9874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 22:43:23.102169 9933 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260807 22:43:23.110100 9874 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 22:43:23.110224 9874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20260807 22:43:23.110400 9874 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 22:43:23.115456 9874 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 22:43:23.115684 9874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20260807 22:43:23.117537 9933 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: Bootstrap starting.
I20260807 22:43:23.155174 9933 log.cc:824] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: Log is configured to *not* fsync() on all Append() calls
I20260807 22:43:23.170378 9933 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 22:43:23.171428 9933 tablet_bootstrap.cc:492] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: Bootstrap complete.
I20260807 22:43:23.173741 9933 ts_tablet_manager.cc:1397] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: Time spent bootstrapping tablet: real 0.057s user 0.032s sys 0.019s
I20260807 22:43:23.183413 9874 rpc_server.cc:307] RPC server started. Bound to: 127.4.103.3:38803
I20260807 22:43:23.183576 10111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.103.3:38803 every 8 connection(s)
I20260807 22:43:23.185525 9874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZnDlfK/test-tmp/txn_status_manager-itest.0.TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted.1786142554140270-4508-0/minicluster-data/ts-2/data/info.pb
I20260807 22:43:23.188587 10112 heartbeater.cc:347] Connected to a master server at 127.4.103.62:46193
I20260807 22:43:23.188895 10112 heartbeater.cc:464] Registering TS with master...
I20260807 22:43:23.188378 9933 raft_consensus.cc:348] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:23.190109 10112 heartbeater.cc:511] Master 127.4.103.62:46193 requested a full tablet report, sending...
I20260807 22:43:23.190217 9933 raft_consensus.cc:729] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdf613248123414e9556f98601d9e402, State: Initialized, Role: FOLLOWER
I20260807 22:43:23.191126 9933 consensus_queue.cc:260] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:23.192551 10115 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:23.193069 10115 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:23.193526 9933 ts_tablet_manager.cc:1428] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402: Time spent starting tablet: real 0.020s user 0.018s sys 0.001s
I20260807 22:43:23.195305 4508 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu as pid 9874
I20260807 22:43:23.195880 8640 ts_manager.cc:194] Re-registered known tserver with Master: cdf613248123414e9556f98601d9e402 (127.4.103.3:38803)
I20260807 22:43:23.197418 8640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.103.3:36105
I20260807 22:43:23.200963 10112 heartbeater.cc:503] Master 127.4.103.62:46193 was elected leader, sending a full tablet report...
I20260807 22:43:23.202380 10042 raft_consensus.cc:3037] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 1 FOLLOWER]: Advancing to term 2
I20260807 22:43:23.211061 10119 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2d63d037cacb4205b689fe6259f9b790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:23.211688 10119 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:23.211608 10115 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: New leader 2d63d037cacb4205b689fe6259f9b790. Latest consensus state: current_term: 2 leader_uuid: "2d63d037cacb4205b689fe6259f9b790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:23.211907 10115 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 22:43:23.227815 9899 tablet_replica.cc:439] TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2d63d037cacb4205b689fe6259f9b790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } } }
I20260807 22:43:23.228305 9899 tablet_replica.cc:442] This TxnStatusTablet replica's current role is: LEADER
I20260807 22:43:25.717420 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 9458
I20260807 22:43:25.726707 9664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:25.834087 10141 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2d63d037cacb4205b689fe6259f9b790)
I20260807 22:43:25.834528 10141 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P fbb7d516f13f438085a3bd7c37805819 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:25.834847 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 9458
I20260807 22:43:25.864259 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 9673
I20260807 22:43:25.874541 9865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:25.926293 10148 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2d63d037cacb4205b689fe6259f9b790)
I20260807 22:43:25.926759 10148 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:25.929015 10148 leader_election.cc:288] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fbb7d516f13f438085a3bd7c37805819 (127.4.103.1:33317), 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639)
W20260807 22:43:25.930598 9922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.103.1:33317: connect: Connection refused (error 111)
W20260807 22:43:25.932950 9922 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fbb7d516f13f438085a3bd7c37805819 (127.4.103.1:33317): Network error: Client connection negotiation failed: client connection to 127.4.103.1:33317: connect: Connection refused (error 111)
I20260807 22:43:26.017112 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 9673
I20260807 22:43:26.054798 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 9874
W20260807 22:43:26.056008 10150 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 22:43:25.929509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.103.2:40639 (local address 127.4.103.3:52233)
0807 22:43:25.930476 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0807 22:43:25.930498 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 22:43:25.930719 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 22:43:26.015554 (+ 84835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 22:43:26.015560 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 22:43:26.015607 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 22:43:26.016048 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 22:43:26.016065 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 22:43:26.054507 (+ 38442us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.103.2:40639: BlockingRecv error: recv got EOF from 127.4.103.2:40639 (error 108)
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":1672,"threads_started":1}
W20260807 22:43:26.056344 9923 leader_election.cc:334] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2d63d037cacb4205b689fe6259f9b790 (127.4.103.2:40639): Network error: Client connection negotiation failed: client connection to 127.4.103.2:40639: BlockingRecv error: recv got EOF from 127.4.103.2:40639 (error 108)
I20260807 22:43:26.056728 9923 leader_election.cc:302] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cdf613248123414e9556f98601d9e402; no voters: 2d63d037cacb4205b689fe6259f9b790, fbb7d516f13f438085a3bd7c37805819
I20260807 22:43:26.057265 10148 raft_consensus.cc:2726] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260807 22:43:26.063546 10105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:26.166474 10148 raft_consensus.cc:482] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2d63d037cacb4205b689fe6259f9b790)
I20260807 22:43:26.166781 10148 raft_consensus.cc:504] T 857795ece0994bc4b213f19c9151ff5a P cdf613248123414e9556f98601d9e402 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdf613248123414e9556f98601d9e402" member_type: VOTER last_known_addr { host: "127.4.103.3" port: 38803 } } peers { permanent_uuid: "fbb7d516f13f438085a3bd7c37805819" member_type: VOTER last_known_addr { host: "127.4.103.1" port: 33317 } } peers { permanent_uuid: "2d63d037cacb4205b689fe6259f9b790" member_type: VOTER last_known_addr { host: "127.4.103.2" port: 40639 } }
I20260807 22:43:26.167299 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 9874
I20260807 22:43:26.203555 4508 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu pid 8594
I20260807 22:43:26.205202 8696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260807 22:43:26.346107 4508 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZnDlfK/build/asan/bin/kudu with pid 8594
2026-08-07T22:43:26Z chronyd exiting
[ OK ] TxnStatusManagerITest.TxnKeptAliveByClientIfStatusManagerRestarted (10399 ms)
[----------] 4 tests from TxnStatusManagerITest (52146 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (52147 ms total)
[ PASSED ] 3 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] TxnStatusManagerITest.TxnKeepAliveMultiTxnStatusManagerInstances
1 FAILED TEST